DYNAMIC "compiler" mode UTF-16 bug

[RomanEmelyanov]

What I do wrong?

$ ../run/john -form=dynamic='md5(utf16($p).$s)' --test
NOTE: This is a debug build, speed will be lower than normal
Benchmarking: dynamic=md5(utf16($p).$s) [128/128 SSE2 4x3]... FAILED (cmp_all(1))

$ ../run/john -form=dynamic='sha1(utf16($p).$s)' --test
NOTE: This is a debug build, speed will be lower than normal
Benchmarking: dynamic=sha1(utf16($p).$s) [128/128 SSE2 4x1]... FAILED (cmp_all(1))

$ ../run/john -form=dynamic='sha256(utf16($p).$s)' --test
NOTE: This is a debug build, speed will be lower than normal
Benchmarking: dynamic=sha256(utf16($p).$s) [128/128 SSE2 4x]... FAILED (cmp_all(1))

$ ../run/john --list=build-info
Version: 1.8.0.13-jumbo-1-bleeding-1e3f3b1 2018-08-21 17:03:18 +0200
Build: linux-gnu 64-bit x86_64 SSE2 AC OMP debug
SIMD: SSE2, interleaving: MD4:3 MD5:3 SHA1:1 SHA256:1 SHA512:1
$JOHN is ../run/
Format interface version: 14
Max. number of reported tunable costs: 4
Rec file version: REC4
Charset file version: CHR3
CHARSET_MIN: 1 (0x01)
CHARSET_MAX: 255 (0xff)
CHARSET_LENGTH: 24
SALT_HASH_SIZE: 1048576
Max. Markov mode level: 400
Max. Markov mode password length: 30
gcc version: 5.4.0
GNU libc version: 2.23 (loaded: 2.23)
Crypto library: OpenSSL
OpenSSL library version: 01000207f
OpenSSL 1.0.2g 1 Mar 2016
File locking: fcntl()
fseek(): fseek
ftell(): ftell
fopen(): fopen
memmem(): System's
Built with these debugging options
'#define DEBUG' set

[magnumripper]

This is a bug, I can reproduce it. (running x86-64 macOS, normal build)

[RomanEmelyanov]

Is any workaround for brute by wordlist or mask?
May be "iconv" can help?
I'm found just idea for hashcat - https://pastebin.com/raw/L3WPDnUq

[magnumripper]

The only workaround here is to write old-style dynamic formats. The upside of that is they can often be faster. See doc/DYNAMIC and related files.

[magnumripper]

I could now spot the bug in the generated script. Here's the debug output of that format:

$ ../run/john -test -form:'dynamic=md5(utf16($p).$s),debug'
(...)
##############################################################
#  Dynamic script for expression md5(utf16($p).$s),debug
##############################################################
Expression=dynamic=md5(utf16($p).$s)
#  Flags for this format
Flag=MGF_FLAT_BUFFERS
Flag=MGF_SALTED
Flag=MGF_UTF8
#  Lengths used in this format
SaltLen=-32
MaxInputLenX86=110
MaxInputLen=110
#  The functions in the script
Func=DynamicFunc__clean_input_kwik
Func=DynamicFunc__setmode_unicode
Func=DynamicFunc__append_keys
Func=DynamicFunc__append_salt
Func=DynamicFunc__MD5_crypt_input1_to_output1_FINAL
#  The test hashes that validate this script
Test=@dynamic=md5(utf16($p).$s)@e0022de67f9edac2c463431f091f8e28$76931fac:abc
Test=@dynamic=md5(utf16($p).$s)@8451c87041c98bb990c4870d51306e66$9dab2b36:john
Test=@dynamic=md5(utf16($p).$s)@62845cee79c254bfe32c573dd278afff$c248b87d:passweird
Test=@dynamic=md5(utf16($p).$s)@9428447fb09222678305414113bc22dd$6ae33f9a:ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyzABCDEF
Test=@dynamic=md5(utf16($p).$s)@02f89bc032c1e27ed96e097e225037f5$62d7183a:

Looking at some UTF-16 formats in dynamic.conf, the missing thing is this line:

  Func=DynamicFunc__clean_input_kwik
  Func=DynamicFunc__setmode_unicode
  Func=DynamicFunc__append_keys
+ Func=DynamicFunc__setmode_normal
  Func=DynamicFunc__append_salt
  Func=DynamicFunc__MD5_crypt_input1_to_output1_FINAL

So a work-around until @jfoug fixes this is to put a fixed script in dynamic.conf as an old-style numbered dynamic format. Let's say we name it dynamic_1609:

diff --git a/run/dynamic.conf b/run/dynamic.conf
index 76376abd8..cee449b91 100644
--- a/run/dynamic.conf
+++ b/run/dynamic.conf
@@ -1270,6 +1270,32 @@ Test=$dynamic_1608$f2a778f1a6ed3d5bc59a5d79104c598f3f07093f240ca4e91333fb09ed4f3
 Test=$dynamic_1608$8b12147de49a2832aca47a5bf6fbca12689420ac14c2547ab90f6d495f21f6dc:ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyzABCDEF
 Test=$dynamic_1608$2a1a9918abe22f14d737462301e0c17b125a5f9ba11dc1e872b5320180437d12:openwall

+##############################################################
+#  md5(utf16($p).$s)
+##############################################################
+[List.Generic:dynamic_1609]
+Expression=dynamic=md5(utf16($p).$s)
+#  Flags for this format
+Flag=MGF_FLAT_BUFFERS
+Flag=MGF_SALTED
+Flag=MGF_UTF8
+#  Lengths used in this format
+SaltLen=-32
+MaxInputLenX86=110
+MaxInputLen=110
+#  The functions in the script
+Func=DynamicFunc__clean_input_kwik
+Func=DynamicFunc__setmode_unicode
+Func=DynamicFunc__append_keys
+Func=DynamicFunc__setmode_normal
+Func=DynamicFunc__append_salt
+Func=DynamicFunc__MD5_crypt_input1_to_output1_FINAL
+#  The test hashes that validate this script
+Test=$dynamic_1609$e0022de67f9edac2c463431f091f8e28$76931fac:abc
+Test=$dynamic_1609$8451c87041c98bb990c4870d51306e66$9dab2b36:john
+Test=$dynamic_1609$62845cee79c254bfe32c573dd278afff$c248b87d:passweird
+Test=$dynamic_1609$9428447fb09222678305414113bc22dd$6ae33f9a:ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyzABCDEF
+Test=$dynamic_1609$02f89bc032c1e27ed96e097e225037f5$62d7183a:

 # this should be last line of the file.  Put other formats before this.  The formats in
 # the following included file are replacement formats for the MD4/5 formats which use

$ ../run/john -test -form:dynamic_1609
Benchmarking: dynamic_1609 [dynamic=md5(utf16($p).$s) 256/256 AVX2 8x3]... DONE
Many salts:     11558K c/s real, 11558K c/s virtual
Only one salt:  9061K c/s real, 9061K c/s virtual

[magnumripper]

Here's an experimental patch that makes the dynamic compiler mode do the Right Thing[tm] for this case:

diff --git a/src/dynamic_compiler.c b/src/dynamic_compiler.c
index ddc6832bc..797435e67 100644
--- a/src/dynamic_compiler.c
+++ b/src/dynamic_compiler.c
@@ -1786,9 +1786,10 @@ static int parse_expression(DC_struct *p) {
 					}
 				} else {
 					// if final hash, then dont clear the mode to normal
-					if ( in_unicode && !(!pCode[i+1] || !pCode[i+1][0]))
+					if ( in_unicode && !(!pCode[i+1] || !pCode[i+1][0])) {
 						comp_add_script_line("Func=DynamicFunc__setmode_normal\n");
-					in_unicode = 0;
+						in_unicode = 0;
+					}
 					if ( (out_raw||out_64||out_64c||out_16u) && !(!pCode[i+1] || !pCode[i+1][0]))
 						comp_add_script_line("Func=DynamicFunc__LargeHash_OUTMode_base16\n");
 					out_raw = out_64 = out_64c = out_16u = 0;
@@ -1804,12 +1805,24 @@ static int parse_expression(DC_struct *p) {
 							if (!strncmp(pCode[x], "app_p", 5)) {
 								comp_add_script_line("Func=DynamicFunc__append_keys%s\n", use_inp1?"":"2"); use_inp1 ? ++inp_cnt : ++inp_cnt2; }
 							else if (!strcmp(pCode[x], "app_s")) {
+								if (in_unicode) {
+									comp_add_script_line("Func=DynamicFunc__setmode_normal\n");
+									in_unicode = 0;
+								}
 								comp_add_script_line("Func=DynamicFunc__append_salt%s\n", use_inp1?"":"2"); use_inp1 ? ++salt_cnt : ++salt_cnt2; }
 							else if (!strncmp(pCode[x], "app_u", 5)) {
 								comp_add_script_line("Func=DynamicFunc__append_userid%s\n", use_inp1?"":"2"); use_inp1 ? ++ex_cnt : ++ex_cnt2; }
 							else if (!strcmp(pCode[x], "app_s2")) {
+								if (in_unicode) {
+									comp_add_script_line("Func=DynamicFunc__setmode_normal\n");
+									in_unicode = 0;
+								}
 								comp_add_script_line("Func=DynamicFunc__append_2nd_salt%s\n", use_inp1?"":"2"); use_inp1 ? ++ex_cnt : ++ex_cnt2; }
 							else if (!strcmp(pCode[x], "app_sh")) {
+								if (in_unicode) {
+									comp_add_script_line("Func=DynamicFunc__setmode_normal\n");
+									in_unicode = 0;
+								}
 								comp_add_script_line("Func=DynamicFunc__append_salt%s\n", use_inp1?"":"2"); use_inp1 ? ++salt_cnt : ++salt_cnt2; }
 							else if (!strcmp(pCode[x], "app_1")) {
 								comp_add_script_line("Func=DynamicFunc__append_input%s_from_CONST1\n", use_inp1?"1":"2"); use_inp1 ? ++ex_cnt : ++ex_cnt2; }

A problem is that I'm not quite sure it's 100% correct (for other cases). Another problem is that when trying the other way round, md5($s.utf16($p)) we reveal other bugs (input is erroneously wiped after appending keys) with or without this patch. Perhaps we need to drop the dynamic compiler mode (as in mark as experimental so it won't be compiled in by default) until @jfoug comes back? That would be a sad thing.

[jfoug]

Please test using the PR: #3568

[magnumripper]

Well it does work

$ ../run/john -test -form:'dynamic=md5(utf16($p).$s)'
This expression will use the RDP dynamic compiler format.
Benchmarking: dynamic=md5(utf16($p).$s) [Dynamic RDP]... DONE
Many salts:	1555K c/s real, 1555K c/s virtual
Only one salt:	1530K c/s real, 1545K c/s virtual

Why is it using RDP and OpenSSL though? This is a simple format, should be SIMD compiled.

[jfoug]

On 12/29/2018 2:51 AM, magnum wrote: Why is it using RDP and OpenSSL though? This is a simple format, should be SIMD compiled.

Here is the script: $ ../run/john -test -form:'dynamic=md5(utf16($p).$s),debug' push push app_p futf16 . app_sh f5h crc32 = 79F7E411 pExpr=md5(utf16($p).$s) extraParams=,debug signature=@dynamic=md5(utf16($p).$s)@ line0=@dynamic=md5(utf16($p).$s)@640a69fc5f37aa010090e4ab3f6b0bdb$df694488 line1=@dynamic=md5(utf16($p).$s)@bdbe6ffdbaab568a3d17071f9de21dec$87ffb1c9 line2=@dynamic=md5(utf16($p).$s)@da0f8e24711a55a3cd58126d9c5ba9cc$a69c5744 line3=@dynamic=md5(utf16($p).$s)@23631bd0e3e54cb39c305c58f9b0885d$4f58497b line4=@dynamic=md5(utf16($p).$s)@fde281b0df722ebeafc558c3af37ea9b$e0b88e64 ############################################################## #  Dynamic script for expression md5(utf16($p).$s),debug ############################################################## Expression=dynamic=md5(utf16($p).$s) #  Flags for this format Flag=MGF_FLAT_BUFFERS Flag=MGF_SALTED Flag=MGF_UTF8 #  Lengths used in this format SaltLen=-32 MaxInputLenX86=110 MaxInputLen=110 #  The functions in the script Func=DynamicFunc__clean_input_kwik Func=DynamicFunc__setmode_unicode Func=DynamicFunc__append_keys Func=DynamicFunc__append_salt Func=DynamicFunc__MD5_crypt_input1_to_output1_FINAL #  The test hashes that validate this script Test=@dynamic=md5(utf16($p).$s)@640a69fc5f37aa010090e4ab3f6b0bdb$df694488:abc Test=@dynamic=md5(utf16($p).$s)@bdbe6ffdbaab568a3d17071f9de21dec$87ffb1c9:john Test=@dynamic=md5(utf16($p).$s)@da0f8e24711a55a3cd58126d9c5ba9cc$a69c5744:passweird Test=@dynamic=md5(utf16($p).$s)@23631bd0e3e54cb39c305c58f9b0885d$4f58497b:ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyzABCDEF Test=@dynamic=md5(utf16($p).$s)@fde281b0df722ebeafc558c3af37ea9b$e0b88e64: The problem, is that the parser is treating utf16 as a function. It is not a function, but a 'setter' for all data inside of it's params. Thus it needs a start and a stop.  The proper script is: Func=DynamicFunc__clean_input_kwik Func=DynamicFunc__setmode_unicode Func=DynamicFunc__append_keys Func=DynamicFunc__setmode_normal   #this line was missing. Func=DynamicFunc__append_salt Func=DynamicFunc__MD5_crypt_input1_to_output1_FINAL So, yes, this is a prime example of one of the tasks I have listed in #3568 (fix more cases which should work).