cryptography - Reverse engineering a file encryption (most likely XOR) -

im trying reverse engineer file format encrypted. uses xor encryption. can create encrypted files known plaintext, analyzed:

enc     71 8d 7e 84 29 20 b8 cb 6c ed bb 8a 62 a1  dec     74 68 69 73 20 69 73 20 61 20 74 65 73 74  xor     05 e5 17 f7 09 49 cb eb 0d cd cf ef 11 d5  txt     t  h   s      s         t  e  s  t    enc     61 ad 84 29 20 b8 cb 6c ed bb 8a 62 a1  dec     64 68 69 73 20 69 73 20 61 20 74 65 73 74  xor     05 c5 d7 f7 09 49 cb eb 0d cd cf ef 11 d5  txt     d  h   s      s         t  e  s  t   enc     62 a5 ae a4 e9 a0 b8 cb 6c ed bb 8a 62 a1  dec     67 68 69 73 20 69 73 20 61 20 74 65 73 74  xor     05 cd c7 d7 c9 c9 cb eb 0d cd cf ef 11 d5 txt     g  h   s      s         t  e  s  t   

it obvious original text part of encryption. first byte of key 05. second byte of key can calculated this:

(enc1 + dec1) or xor1

the rather low entropy of key implies similar rule other key-bytes.

any ideas?

you got it!

the key's byte @ m position given :

km = [(en + dn) ^ kn] | secret 

where :

en previous encrypted byte dn previous plain text byte kn previous key byte (k0 = 5) secret arbitrary number starting @ 5 , incremented 2 every 2 turns ^ xor operator | or operator 

a simple c# key generator :

namespace sample.customencrypt {     using system.collections.generic;     using system.text;      class program {         static void main() {             var key1 = generatekey("this test");             var key2 = generatekey("dhis test");             var key3 = generatekey("ghis test");         }          public static byte[] generatekey(string input) {             var plain = encoding.utf8.getbytes(input);             var secret = 5;             var key = new list<byte> {                 0x05             };              (var = 0; < plain.length - 1; i++) {                 var dn = plain[i];                 var kn = key[i];                 var en = (byte)(dn ^ kn);                 var km = (byte)(((dn + en) ^ kn) | secret);                  key.add(km);                  if (i % 2 == 0) {                     secret += 2;                 }             }              return key.toarray();         }     } } 

ps: pointed out eugene should post on reverse engineering or cryptography next time.