DYNAMIC "compiler" mode UTF-16 bug #3389

RomanEmelyanov · 2018-08-25T12:34:29Z

What I do wrong?

$ ../run/john -form=dynamic='md5(utf16($p).$s)' --test
NOTE: This is a debug build, speed will be lower than normal
Benchmarking: dynamic=md5(utf16($p).$s) [128/128 SSE2 4x3]... FAILED (cmp_all(1))

$ ../run/john -form=dynamic='sha1(utf16($p).$s)' --test
NOTE: This is a debug build, speed will be lower than normal
Benchmarking: dynamic=sha1(utf16($p).$s) [128/128 SSE2 4x1]... FAILED (cmp_all(1))

$ ../run/john -form=dynamic='sha256(utf16($p).$s)' --test
NOTE: This is a debug build, speed will be lower than normal
Benchmarking: dynamic=sha256(utf16($p).$s) [128/128 SSE2 4x]... FAILED (cmp_all(1))

$ ../run/john --list=build-info
Version: 1.8.0.13-jumbo-1-bleeding-1e3f3b1 2018-08-21 17:03:18 +0200
Build: linux-gnu 64-bit x86_64 SSE2 AC OMP debug
SIMD: SSE2, interleaving: MD4:3 MD5:3 SHA1:1 SHA256:1 SHA512:1
$JOHN is ../run/
Format interface version: 14
Max. number of reported tunable costs: 4
Rec file version: REC4
Charset file version: CHR3
CHARSET_MIN: 1 (0x01)
CHARSET_MAX: 255 (0xff)
CHARSET_LENGTH: 24
SALT_HASH_SIZE: 1048576
Max. Markov mode level: 400
Max. Markov mode password length: 30
gcc version: 5.4.0
GNU libc version: 2.23 (loaded: 2.23)
Crypto library: OpenSSL
OpenSSL library version: 01000207f
OpenSSL 1.0.2g 1 Mar 2016
File locking: fcntl()
fseek(): fseek
ftell(): ftell
fopen(): fopen
memmem(): System's
Built with these debugging options
'#define DEBUG' set

magnumripper · 2018-08-25T15:51:47Z

This is a bug, I can reproduce it. (running x86-64 macOS, normal build)

RomanEmelyanov · 2018-08-25T16:55:06Z

Is any workaround for brute by wordlist or mask?
May be "iconv" can help?
I'm found just idea for hashcat - https://pastebin.com/raw/L3WPDnUq

magnumripper · 2018-08-25T18:36:22Z

The only workaround here is to write old-style dynamic formats. The upside of that is they can often be faster. See doc/DYNAMIC and related files.

magnumripper · 2018-09-03T16:26:31Z

I could now spot the bug in the generated script. Here's the debug output of that format:

$ ../run/john -test -form:'dynamic=md5(utf16($p).$s),debug'
(...)
##############################################################
#  Dynamic script for expression md5(utf16($p).$s),debug
##############################################################
Expression=dynamic=md5(utf16($p).$s)
#  Flags for this format
Flag=MGF_FLAT_BUFFERS
Flag=MGF_SALTED
Flag=MGF_UTF8
#  Lengths used in this format
SaltLen=-32
MaxInputLenX86=110
MaxInputLen=110
#  The functions in the script
Func=DynamicFunc__clean_input_kwik
Func=DynamicFunc__setmode_unicode
Func=DynamicFunc__append_keys
Func=DynamicFunc__append_salt
Func=DynamicFunc__MD5_crypt_input1_to_output1_FINAL
#  The test hashes that validate this script
Test=@dynamic=md5(utf16($p).$s)@e0022de67f9edac2c463431f091f8e28$76931fac:abc
Test=@dynamic=md5(utf16($p).$s)@8451c87041c98bb990c4870d51306e66$9dab2b36:john
Test=@dynamic=md5(utf16($p).$s)@62845cee79c254bfe32c573dd278afff$c248b87d:passweird
Test=@dynamic=md5(utf16($p).$s)@9428447fb09222678305414113bc22dd$6ae33f9a:ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyzABCDEF
Test=@dynamic=md5(utf16($p).$s)@02f89bc032c1e27ed96e097e225037f5$62d7183a:

Looking at some UTF-16 formats in dynamic.conf, the missing thing is this line:

  Func=DynamicFunc__clean_input_kwik
  Func=DynamicFunc__setmode_unicode
  Func=DynamicFunc__append_keys
+ Func=DynamicFunc__setmode_normal
  Func=DynamicFunc__append_salt
  Func=DynamicFunc__MD5_crypt_input1_to_output1_FINAL

So a work-around until @jfoug fixes this is to put a fixed script in dynamic.conf as an old-style numbered dynamic format. Let's say we name it dynamic_1609:

diff --git a/run/dynamic.conf b/run/dynamic.conf
index 76376abd8..cee449b91 100644
--- a/run/dynamic.conf
+++ b/run/dynamic.conf
@@ -1270,6 +1270,32 @@ Test=$dynamic_1608$f2a778f1a6ed3d5bc59a5d79104c598f3f07093f240ca4e91333fb09ed4f3
 Test=$dynamic_1608$8b12147de49a2832aca47a5bf6fbca12689420ac14c2547ab90f6d495f21f6dc:ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyzABCDEF
 Test=$dynamic_1608$2a1a9918abe22f14d737462301e0c17b125a5f9ba11dc1e872b5320180437d12:openwall

+##############################################################
+#  md5(utf16($p).$s)
+##############################################################
+[List.Generic:dynamic_1609]
+Expression=dynamic=md5(utf16($p).$s)
+#  Flags for this format
+Flag=MGF_FLAT_BUFFERS
+Flag=MGF_SALTED
+Flag=MGF_UTF8
+#  Lengths used in this format
+SaltLen=-32
+MaxInputLenX86=110
+MaxInputLen=110
+#  The functions in the script
+Func=DynamicFunc__clean_input_kwik
+Func=DynamicFunc__setmode_unicode
+Func=DynamicFunc__append_keys
+Func=DynamicFunc__setmode_normal
+Func=DynamicFunc__append_salt
+Func=DynamicFunc__MD5_crypt_input1_to_output1_FINAL
+#  The test hashes that validate this script
+Test=$dynamic_1609$e0022de67f9edac2c463431f091f8e28$76931fac:abc
+Test=$dynamic_1609$8451c87041c98bb990c4870d51306e66$9dab2b36:john
+Test=$dynamic_1609$62845cee79c254bfe32c573dd278afff$c248b87d:passweird
+Test=$dynamic_1609$9428447fb09222678305414113bc22dd$6ae33f9a:ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyzABCDEF
+Test=$dynamic_1609$02f89bc032c1e27ed96e097e225037f5$62d7183a:

 # this should be last line of the file.  Put other formats before this.  The formats in
 # the following included file are replacement formats for the MD4/5 formats which use

$ ../run/john -test -form:dynamic_1609
Benchmarking: dynamic_1609 [dynamic=md5(utf16($p).$s) 256/256 AVX2 8x3]... DONE
Many salts:     11558K c/s real, 11558K c/s virtual
Only one salt:  9061K c/s real, 9061K c/s virtual

magnumripper · 2018-09-03T16:28:55Z

Here's an experimental patch that makes the dynamic compiler mode do the Right Thing[tm] for this case:

diff --git a/src/dynamic_compiler.c b/src/dynamic_compiler.c
index ddc6832bc..797435e67 100644
--- a/src/dynamic_compiler.c
+++ b/src/dynamic_compiler.c
@@ -1786,9 +1786,10 @@ static int parse_expression(DC_struct *p) {
 					}
 				} else {
 					// if final hash, then dont clear the mode to normal
-					if ( in_unicode && !(!pCode[i+1] || !pCode[i+1][0]))
+					if ( in_unicode && !(!pCode[i+1] || !pCode[i+1][0])) {
 						comp_add_script_line("Func=DynamicFunc__setmode_normal\n");
-					in_unicode = 0;
+						in_unicode = 0;
+					}
 					if ( (out_raw||out_64||out_64c||out_16u) && !(!pCode[i+1] || !pCode[i+1][0]))
 						comp_add_script_line("Func=DynamicFunc__LargeHash_OUTMode_base16\n");
 					out_raw = out_64 = out_64c = out_16u = 0;
@@ -1804,12 +1805,24 @@ static int parse_expression(DC_struct *p) {
 							if (!strncmp(pCode[x], "app_p", 5)) {
 								comp_add_script_line("Func=DynamicFunc__append_keys%s\n", use_inp1?"":"2"); use_inp1 ? ++inp_cnt : ++inp_cnt2; }
 							else if (!strcmp(pCode[x], "app_s")) {
+								if (in_unicode) {
+									comp_add_script_line("Func=DynamicFunc__setmode_normal\n");
+									in_unicode = 0;
+								}
 								comp_add_script_line("Func=DynamicFunc__append_salt%s\n", use_inp1?"":"2"); use_inp1 ? ++salt_cnt : ++salt_cnt2; }
 							else if (!strncmp(pCode[x], "app_u", 5)) {
 								comp_add_script_line("Func=DynamicFunc__append_userid%s\n", use_inp1?"":"2"); use_inp1 ? ++ex_cnt : ++ex_cnt2; }
 							else if (!strcmp(pCode[x], "app_s2")) {
+								if (in_unicode) {
+									comp_add_script_line("Func=DynamicFunc__setmode_normal\n");
+									in_unicode = 0;
+								}
 								comp_add_script_line("Func=DynamicFunc__append_2nd_salt%s\n", use_inp1?"":"2"); use_inp1 ? ++ex_cnt : ++ex_cnt2; }
 							else if (!strcmp(pCode[x], "app_sh")) {
+								if (in_unicode) {
+									comp_add_script_line("Func=DynamicFunc__setmode_normal\n");
+									in_unicode = 0;
+								}
 								comp_add_script_line("Func=DynamicFunc__append_salt%s\n", use_inp1?"":"2"); use_inp1 ? ++salt_cnt : ++salt_cnt2; }
 							else if (!strcmp(pCode[x], "app_1")) {
 								comp_add_script_line("Func=DynamicFunc__append_input%s_from_CONST1\n", use_inp1?"1":"2"); use_inp1 ? ++ex_cnt : ++ex_cnt2; }

A problem is that I'm not quite sure it's 100% correct (for other cases). Another problem is that when trying the other way round, md5($s.utf16($p)) we reveal other bugs (input is erroneously wiped after appending keys) with or without this patch. Perhaps we need to drop the dynamic compiler mode (as in mark as experimental so it won't be compiled in by default) until @jfoug comes back? That would be a sad thing.

jfoug · 2018-12-27T23:28:11Z

Please test using the PR: #3568

magnumripper · 2018-12-29T08:51:45Z

Well it does work

$ ../run/john -test -form:'dynamic=md5(utf16($p).$s)'
This expression will use the RDP dynamic compiler format.
Benchmarking: dynamic=md5(utf16($p).$s) [Dynamic RDP]... DONE
Many salts:	1555K c/s real, 1555K c/s virtual
Only one salt:	1530K c/s real, 1545K c/s virtual

Why is it using RDP and OpenSSL though? This is a simple format, should be SIMD compiled.

jfoug · 2018-12-29T13:34:30Z

On 12/29/2018 2:51 AM, magnum wrote: Why is it using RDP and OpenSSL though? This is a simple format, should be SIMD compiled.

Here is the script: $ ../run/john -test -form:'dynamic=md5(utf16($p).$s),debug' push push app_p futf16 . app_sh f5h crc32 = 79F7E411 pExpr=md5(utf16($p).$s) extraParams=,debug signature=@dynamic=md5(utf16($p).$s)@ line0=@dynamic=md5(utf16($p).$s)@640a69fc5f37aa010090e4ab3f6b0bdb$df694488 line1=@dynamic=md5(utf16($p).$s)@bdbe6ffdbaab568a3d17071f9de21dec$87ffb1c9 line2=@dynamic=md5(utf16($p).$s)@da0f8e24711a55a3cd58126d9c5ba9cc$a69c5744 line3=@dynamic=md5(utf16($p).$s)@23631bd0e3e54cb39c305c58f9b0885d$4f58497b line4=@dynamic=md5(utf16($p).$s)@fde281b0df722ebeafc558c3af37ea9b$e0b88e64 ############################################################## # Dynamic script for expression md5(utf16($p).$s),debug ############################################################## Expression=dynamic=md5(utf16($p).$s) # Flags for this format Flag=MGF_FLAT_BUFFERS Flag=MGF_SALTED Flag=MGF_UTF8 # Lengths used in this format SaltLen=-32 MaxInputLenX86=110 MaxInputLen=110 # The functions in the script Func=DynamicFunc__clean_input_kwik Func=DynamicFunc__setmode_unicode Func=DynamicFunc__append_keys Func=DynamicFunc__append_salt Func=DynamicFunc__MD5_crypt_input1_to_output1_FINAL # The test hashes that validate this script Test=@dynamic=md5(utf16($p).$s)@640a69fc5f37aa010090e4ab3f6b0bdb$df694488:abc Test=@dynamic=md5(utf16($p).$s)@bdbe6ffdbaab568a3d17071f9de21dec$87ffb1c9:john Test=@dynamic=md5(utf16($p).$s)@da0f8e24711a55a3cd58126d9c5ba9cc$a69c5744:passweird Test=@dynamic=md5(utf16($p).$s)@23631bd0e3e54cb39c305c58f9b0885d$4f58497b:ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyzABCDEF Test=@dynamic=md5(utf16($p).$s)@fde281b0df722ebeafc558c3af37ea9b$e0b88e64: The problem, is that the parser is treating utf16 as a function. It is not a function, but a 'setter' for all data inside of it's params. Thus it needs a start and a stop. The proper script is: Func=DynamicFunc__clean_input_kwik Func=DynamicFunc__setmode_unicode Func=DynamicFunc__append_keys Func=DynamicFunc__setmode_normal #this line was missing. Func=DynamicFunc__append_salt Func=DynamicFunc__MD5_crypt_input1_to_output1_FINAL So, yes, this is a prime example of one of the tasks I have listed in #3568 (fix more cases which should work).

jfoug · 2018-12-29T15:07:09Z

On 12/29/2018 2:51 AM, magnum wrote: Why is it using RDP and OpenSSL though? This is a simple format, should be SIMD compiled.

I got this working for SIMD code, but want to do a lot more testing. With the code emitter, I was not removing the unicode, if this was the last function. This 'is' the last function (other than the outside md5(),) however, there was the $s to append. I simply removed the check for last function, and things work. BUT I think I need to re-think the last function. It should honor the last function IF there is no additional work to do. SO md5($s.utf16($p)) would not 'clear' the unicode setting. BUT md5(utf16($p).$s) WOULD clear it. The later is still last function, BUT there is additional work to be done, and that must be done without the utf16 mode being set. ``` $ ../run/john -test -form:'dynamic=md5(utf16($p).$s)' Benchmarking: dynamic=md5(utf16($p).$s) [256/256 AVX2 8x3]... DONE Many salts: 3110K c/s real, 4001K c/s virtual Only one salt: 2453K c/s real, 3233K c/s virtual $ ../run/john -test -form:'dynamic=md5(utf16($p).$s),rdp' This expression will use the RDP dynamic compiler format. Benchmarking: dynamic=md5(utf16($p).$s) [Dynamic RDP]... DONE Warning: "Many salts" test limited: 127/256 Many salts: 417534 c/s real, 487680 c/s virtual Only one salt: 380320 c/s real, 470993 c/s virtual ``` This shows the with and without using the RDP. The RDP code did get this working, but as you point out, this is a format which 'should' work properly in original dyna-script mode of the compiler. The RDP is not doing 'too' bad. We are 6.3x slower here, but expect that, since we do not get the 8x SIMD boost. I do not believe there is much to gain optimizing the RDP, other than there may be improvements doing things like common expression removal , such as md5(md5($p.$s).$s.md5($p.$s)) being reduced to tmp1=md5($p.$s) result_binary=md5($tmp1.$s.$tmp1). That should be able to be added to the RDP (will have to reparse/lexi to do it, but still easy. I would need to first find the common expressions, remove and replace them with temps. So first would be expr_t0 = md5($p.$s) then the expression is md5($t0.$s.$t0) then parse each temp expression, then parse the final expression. Then in the RDP, it would first run all temp variable expressions (in this case 1). Then run the real expression. The real expression code would simply be clean_input1, append_t0, append_s, append_t0, md5, convert_result_to_binary But this optimization is in no way a stopper for Jumbo2. It will be an enhancement. As it stands, getting the dyna script to handle more cases is also not a show stopper, now that RDP will fill in the problem hashes.

* added dynamic-compiler RDP format when script builds incorrectly * fixed compiler errors about signed/unsigned chars * better algo label. Fixed bug when MD5_X2 set * the failed message (i.e. using RDP) needed moved inside an if clause * dyna-compiler. added ,rdp switch to force RDP format. common_init needed for dyna-lib. expanded buffers in the compiler. * fixed john.pot output of things like -form=dynamic=md5($p) * Handle extra params for compiler lib formats. Fixed WS and speeling errors, per reviews * exe bit set * VC at 2015 is C99 compliant for vsnprintf. Handle RDP where base format stored keys in input buffers Closes #1746 and #2800, see also #3389, #3125

magnumripper · 2020-04-01T17:26:51Z

Closing bc wo

magnumripper added the bug label Aug 25, 2018

magnumripper assigned jfoug Aug 25, 2018

magnumripper changed the title ~~DYNAMIC utf16 test error - FAILED (cmp_all(1))~~ DYNAMIC "compiler" mode UTF-16 bug Sep 3, 2018

magnumripper added this to the 1.8.0-jumbo-2 milestone Dec 19, 2018

jfoug mentioned this issue Dec 24, 2018

Dynamic 'compiler' work (main work task) #3562

Open

5 tasks

claudioandre-br added the fixed - pending verify label Dec 27, 2018

magnumripper removed the fixed - pending verify label Dec 29, 2018

jfoug added the fixed - pending verify label Dec 30, 2018

jfoug mentioned this issue Mar 1, 2019

Unprintable characters in pot file after cracking a dynamic_25 hash #2437

Closed

solardiz modified the milestones: 1.9.0-jumbo-1, 1.8.0-post-jumbo-2 Mar 29, 2019

magnumripper closed this as completed Apr 1, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

DYNAMIC "compiler" mode UTF-16 bug #3389

DYNAMIC "compiler" mode UTF-16 bug #3389

RomanEmelyanov commented Aug 25, 2018 •

edited

Loading

magnumripper commented Aug 25, 2018 •

edited

Loading

RomanEmelyanov commented Aug 25, 2018

magnumripper commented Aug 25, 2018

magnumripper commented Sep 3, 2018

magnumripper commented Sep 3, 2018

jfoug commented Dec 27, 2018

magnumripper commented Dec 29, 2018

jfoug commented Dec 29, 2018 via email

jfoug commented Dec 29, 2018 via email

magnumripper commented Apr 1, 2020

DYNAMIC "compiler" mode UTF-16 bug #3389

DYNAMIC "compiler" mode UTF-16 bug #3389

Comments

RomanEmelyanov commented Aug 25, 2018 • edited Loading

magnumripper commented Aug 25, 2018 • edited Loading

RomanEmelyanov commented Aug 25, 2018

magnumripper commented Aug 25, 2018

magnumripper commented Sep 3, 2018

magnumripper commented Sep 3, 2018

jfoug commented Dec 27, 2018

magnumripper commented Dec 29, 2018

jfoug commented Dec 29, 2018 via email

jfoug commented Dec 29, 2018 via email

magnumripper commented Apr 1, 2020

RomanEmelyanov commented Aug 25, 2018 •

edited

Loading

magnumripper commented Aug 25, 2018 •

edited

Loading