Dynamic ad-hoc formats failing from fuzzer

[magnumripper]

First one from the dynamic fuzzer that fails:

$ ../run/john -test -form:'dynamic=md4(MD4($p).$s.md4($p)),debug'
push
push
app_p
f4H
.
app_sh
.
push
app_p
f4h
f4h

crc32 = 07023FDB
pExpr=md4(MD4($p).$s.md4($p))
extraParams=,debug
signature=@dynamic=md4(MD4($p).$s.md4($p)),debug@
line0=@dynamic=md4(MD4($p).$s.md4($p))@ee42213df9cba9153d42c226f0daf9c2$719a288e
line1=@dynamic=md4(MD4($p).$s.md4($p))@7fb0bf5fe7ba047508b6fc03361f23d1$3d85c237
line2=@dynamic=md4(MD4($p).$s.md4($p))@d691bcd654a62b3db1d9363726d85f60$7184f1d5
line3=@dynamic=md4(MD4($p).$s.md4($p))@5b306463764d0e4cb43d8ef7d6879348$85b68792
line4=@dynamic=md4(MD4($p).$s.md4($p))@b6c89de455aef85fcc552dafdd91e8a1$83213d35
##############################################################
#  Dynamic script for expression md4(MD4($p).$s.md4($p)),debug
##############################################################
Expression=dynamic=md4(MD4($p).$s.md4($p))
#  Flags for this format
Flag=MGF_FLAT_BUFFERS
Flag=MGF_SALTED
#  Lengths used in this format
SaltLen=-32
MaxInputLenX86=110
MaxInputLen=110
#  The functions in the script
Func=DynamicFunc__clean_input_kwik
Func=DynamicFunc__LargeHash_OUTMode_base16u
Func=DynamicFunc__append_keys
Func=DynamicFunc__MD4_crypt_input1_overwrite_input2
Func=DynamicFunc__LargeHash_OUTMode_base16
Func=DynamicFunc__append_keys2
Func=DynamicFunc__MD4_crypt_input2_overwrite_input2
Func=DynamicFunc__clean_input_kwik
Func=DynamicFunc__append_salt
Func=DynamicFunc__append_input_from_input2
Func=DynamicFunc__MD4_crypt_input1_to_output1_FINAL
#  The test hashes that validate this script
Test=@dynamic=md4(MD4($p).$s.md4($p))@ee42213df9cba9153d42c226f0daf9c2$719a288e:abc
Test=@dynamic=md4(MD4($p).$s.md4($p))@7fb0bf5fe7ba047508b6fc03361f23d1$3d85c237:john
Test=@dynamic=md4(MD4($p).$s.md4($p))@d691bcd654a62b3db1d9363726d85f60$7184f1d5:passweird
Test=@dynamic=md4(MD4($p).$s.md4($p))@5b306463764d0e4cb43d8ef7d6879348$85b68792:ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyzABCDEF
Test=@dynamic=md4(MD4($p).$s.md4($p))@b6c89de455aef85fcc552dafdd91e8a1$83213d35:

$ ../run/john -test -form:'dynamic=md4(MD4($p).$s.md4($p))'
Benchmarking: dynamic=md4(MD4($p).$s.md4($p)) [128/128 AVX 4x3]... FAILED (cmp_all(1))

I'm not 100% sure the problem is in the new test vectors or an actual bug, but it looks like the latter.

[jfoug]

No, this is a known issue that I am working on, but have not completed yet. It is partly due to trying to go directly from a recursive descent parsed program and 'converting' it into a flat iterated program.

we do this MD4($p) putting that into buf 2.
then we append keys and do md4() overwriting the buf 2.
we then put salt into buf 1, append results of buf 2 and then do a final MD4

Well you can see that this is not what matches the expression. This type problem was found with the generic-dynamic-fuzzer, but I have not 'fixed' it yet.

This has nothing to do with upcase, low case. or length of buffers. It failed before any changes you made. simply the dyna-compiler not properly emitting code.

[kholia]

The following dynamic expression is also failing,

$ ../run/john -format=dynamic='md5(sha1($p).md5($p).sha1($p))' hash
...
Self test failed (cmp_all(1))

$ cat hash
c756b56aed8d6748ee63e1e270c71a3f

Password for this hash is password. This hashing scheme is used in SunShop Shopping Cart.

[kholia]

$ ../run/john -format=dynamic='md5(sha1($p).md5($p).sha1($p)),debug' hash
...
Expression=dynamic=md5(sha1($p).md5($p).sha1($p))
#  Flags for this format
Flag=MGF_FLAT_BUFFERS
#  Lengths used in this format
MaxInputLenX86=110
MaxInputLen=110
#  The functions in the script
Func=DynamicFunc__clean_input_kwik
Func=DynamicFunc__append_keys
Func=DynamicFunc__SHA1_crypt_input1_overwrite_input2
Func=DynamicFunc__append_keys2
Func=DynamicFunc__MD5_crypt_input2_overwrite_input2
Func=DynamicFunc__clean_input_kwik
Func=DynamicFunc__append_keys
Func=DynamicFunc__SHA1_crypt_input1_overwrite_input2
Func=DynamicFunc__clean_input_kwik
Func=DynamicFunc__append_input_from_input2
Func=DynamicFunc__append_input_from_input2
Func=DynamicFunc__MD5_crypt_input1_to_output1_FINAL

It seems that the input2 buffer is being overwritten.

[jfoug]

Please test using the PR: #3568

[jfoug]

@kholia:

After PR:

$ ../run/john -format=dynamic='md5(sha1($p).md5($p).sha1($p))' hash
cmp_one() failed. This format will FAIL and needs the Slower dyna-compiler format
This expression will use the RDP dynamic compiler format.
Using default input encoding: UTF-8
Loaded 1 password hash (dynamic=md5(sha1($p).md5($p).sha1($p)) [Dynamic RDP])
Warning: no OpenMP support for this hash type, consider --fork=2
Proceeding with single, rules:Wordlist
Press 'q' or Ctrl-C to abort, almost any other key for status
Almost done: Processing the remaining buffered candidate passwords, if any
Proceeding with wordlist:../run/password.lst, rules:Wordlist
password         (?)
1g 0:00:00:00 DONE 2/3 (2018-12-27 17:32) 18.86g/s 63396p/s 63396c/s 63396C/s 123456..Geronimo
Use the "--show --format=dynamic=md5(sha1($p).md5($p).sha1($p))" options to display all of the cracked passwords reliably
Session completed

$ cat hash
c756b56aed8d6748ee63e1e270c71a3f

$ cat ../run/john.pot
@dynamic=md5(sha1($p).md5($p).sha1($p))@c756b56aed8d6748ee63e1e270c71a3f:password

$ ../run/john -format=dynamic='md5(sha1($p).md5($p).sha1($p))' -show in
cmp_one() failed. This format will FAIL and needs the Slower dyna-compiler format
This expression will use the RDP dynamic compiler format.
?:password

1 password hash cracked, 0 left

Since this was 'supposed' RW data, so I wanted to test all aspects of it, including the .pot file.