Allow --dupe-suppression in batch mode #5524
Comments
|
This was tricky to figure out, but looks easy if we accept the side-effects as desirable: commit cd8d511847dca14d0cbadabc2d6587ae99cf5352 (HEAD -> fixes-20240923)
Author: Solar <solar@openwall.com>
Date: Mon Sep 23 02:50:30 2024 +0200
Options: Allow --rules and/or --dupe-suppression in batch mode
Fixes #5524
diff --git a/src/options.h b/src/options.h
index d0a71b01e..09d82605e 100644
--- a/src/options.h
+++ b/src/options.h
@@ -69,7 +69,7 @@
(FLG_EXTERNAL_CHK | FLG_ACTION | FLG_CRACKING_SUP | FLG_PWD_SUP)
/* Batch cracker */
#define FLG_BATCH_CHK 0x0000000000004000ULL
-#define FLG_BATCH_SET (FLG_BATCH_CHK | FLG_CRACKING_SET)
+#define FLG_BATCH_SET (FLG_BATCH_CHK | FLG_CRACKING_SET | FLG_RULES_ALLOW)
/* Stdout mode */
#define FLG_STDOUT 0x0000000000008000ULL
/* Restoring an interrupted session */This allows not only However, specifying only |
|
Are you saying it somehow doesn't behave like |
Yes. |
|
Something is b0rken for sure. john.conf: Note how it says "9 rules (3 x 3)" here, which is correct. The above looks normal. Apparently the stacked rules are actually applied differently in single mode: First the whole mode runs through stacked rule 1, then everything rewinds and we go through stacked rule 2, and so on. My memory of implementing it is very faint but it was likely easiest to add that way, or perhaps just least intrusive. Now let's see what happens next: Here's the first sign of some change that may have caused the regression. The total should be printed later and say "9 (3 x 3)" just like in single mode. This problem isn't just with batch mode, it happens with ...and then it seems to get stuck in an indefinite loop. Note that the SAME happens using Not sure what caused this, we should be able to bisect it. |
|
Hmm maybe there wasn't an indefinite loop after all. words.lst I had my default verbosity bumped (for seeing opencl build warnings 🙄) and that emitted the "stacked rule #..." for every word. |
|
With that confusion cleared, here's batch mode, using the same short word list: Note though that the stacked rules are applied to incremental mode as well (as designed). Perhaps the "3 (3 x 1) rules" is merely cosmetic, from some change that didn't cause other trouble. Note that running with verbosity 4 or higher can hit performance drastically due to logging alone! Perhaps that was/is the problem? |
That is not it. Running batch mode with |
|
Re-opening so that we don't forget to look into the new (or remaining) weirdness further, although we may instead want to open a separate issue for that (please feel free to do that @magnumripper). |
|
I think I've just fixed the Anyway, if the current fix works, then let's close this issue. @magnumripper Can you please check using your test cases? Thank you! |
|
FWIW I still get the out of order stats: Should be: I'd like to find what broke it. |
Is this related to my allowing for stacked rules in batch mode or is it a separate/older issue? |
Way older. I tried to bisect but failed (once back at 2020 or so, today's compilers refuse to build for all sorts of odd reasons - I'd need to set up some older version of Linux). I will try to figure it out without bisecting. |
|
OK this appears to be a purely cosmetical problem now, and 'git blame' indicates it might have been this way ever since stacked rules were merged. Here's a fix for it: diff --git a/src/wordlist.c b/src/wordlist.c
index e2c827ff8..cbd5cd745 100644
--- a/src/wordlist.c
+++ b/src/wordlist.c
@@ -1037,16 +1037,6 @@ REDO_AFTER_LMLOOP:
rules_init(db, length);
rule_count = rules_count(&ctx, -1);
- if (do_lmloop || !db->plaintexts->head) {
- if (rules_stacked_after)
- log_event("- Total %u (%d x %u) preprocessed word mangling rules",
- rule_count * crk_stacked_rule_count,
- rule_count, crk_stacked_rule_count);
- else
- log_event("- %d preprocessed word mangling rules", rule_count);
- }
-
-
apply = rules_apply;
} else {
rule_ctx = NULL;
@@ -1077,6 +1067,15 @@ REDO_AFTER_LMLOOP:
crk_init(db, fix_state, NULL);
}
+ if (rules && (do_lmloop || !db->plaintexts->head)) {
+ if (rules_stacked_after)
+ log_event("- Total %u (%d x %u) preprocessed word mangling rules",
+ rule_count * crk_stacked_rule_count,
+ rule_count, crk_stacked_rule_count);
+ else
+ log_event("- %d preprocessed word mangling rules", rule_count);
+ }
+
if (dupeCheck || rules) {
int force = (dupeCheck || (options.flags & FLG_STDOUT)) && options.suppressor_size;
suppressor_init(SUPPRESSOR_UPDATE | (force ? SUPPRESSOR_FORCE : 0));This moves the logging to after |
|
@magnumripper Since the remaining issue is much older, I suggest we close this issue and you may open a separate issue/PR for that older issue. |
|
Yes... or maybe I just make a PR with that diff above? I really can't see any dragons hiding there. |
Sure, that's what I meant by "open a separate issue/PR for that older issue". |
It wasn't reporting stacked rules correcly. See openwall#5524
It wasn't reporting stacked rules correcly. See #5524
Dupe suppression is enabled by default in batch mode (because it includes wordlist+rules), but the command-line option to control it is not allowed (technically because no other option specifies the use of rules). Ideally, we'd allow this option to make it easier to explicitly disable the dupe suppressor or adjust its memory usage. Right now, the only ways to do that are editing the configuration file or overriding the configuration file name.
This may be a bit tricky to implement (which is why it hasn't been done right away).
The text was updated successfully, but these errors were encountered: