Prince mode silently skips passwords if password length over current `--max-length`

[klensy]

Checklist

• 🥇 I've read and understood these instrucions(<---- btw, here a typo);
  • This is not a support forum, it's a bug tracker. For questions and support, review postings on the john-users mailing list.
• 👍 I've tested using latest bleeding version from this repository.
  • Be clear about your environment and what you are doing. Share a sample hash or file that can be used to reproduce.
• 😕 I'm confused and I need guidance.
  • Please, read the instructions at https://www.openwall.com/john/#lists, then join the list before posting.

$ ./john --list=build-info
Version: 1.9.0-jumbo-1+bleeding-2e6eba4 2021-07-07 17:16:06 +0200
Build: cygwin 64-bit x86_64 AVX2 AC OMP
SIMD: AVX2, interleaving: MD4:3 MD5:3 SHA1:1 SHA256:1 SHA512:1
CPU tests: AVX2
CPU fallback binary: john-xop
OMP fallback binary: john-avx2-non-omp
$JOHN is /run/
Format interface version: 14
Max. number of reported tunable costs: 4
Rec file version: REC4
Charset file version: CHR3
CHARSET_MIN: 1 (0x01)
CHARSET_MAX: 255 (0xff)
CHARSET_LENGTH: 24
SALT_HASH_SIZE: 1048576
SINGLE_IDX_MAX: 2147483648
SINGLE_BUF_MAX: 4294967295
Effective limit: Number of salts vs. SingleMaxBufferSize
Max. Markov mode level: 400
Max. Markov mode password length: 30
gcc version: 10.2.0
OpenCL headers version: 2.2
Crypto library: OpenSSL
OpenSSL library version: 01010106f
OpenSSL 1.1.1f  31 Mar 2020
GMP library version: 6.2.1
File locking: fcntl()
fseek(): fseek
ftell(): ftell
fopen(): fopen
memmem(): System's
times(2) sysconf(_SC_CLK_TCK) is 1000
Using times(2) for timers, resolution 1 ms
HR timer: QueryPerformanceCounter(), resolution 256 ns
Total physical host memory: 16324 MiB
Available physical host memory: 10033 MiB
Terminal locale string: ru_RU.UTF-8
Parsed terminal locale: UTF-8

command:

$ ./john --format=NT hashlist.txt  --progress-every=5  --no-log --prince=wordlist.txt --prince-elem-cnt-max=8  --prince-elem-cnt-min=1

wordlist:

aaaaaa
bb
cc
dddddddd

I've expecting that if combined password length (given current wordlist and prince-elem-cnt-max) can be over current --max-length, john should warn about that, instead of skipping that passwords.

It can be not specific to prince mode, but i didn't checked that.

It can be not specific to prince mode

It is not (using the same wordlist).

john --progress-every=5  --no-log --wordlist=wordlist.txt --max-len=5 --stdout
Using default input encoding: UTF-8
bb
cc
2p 0:00:00:00 100.00% (2021-08-04 19:37) 5.000p/s cc

I've expecting that if combined password length [...] john should warn about that, instead of skipping that passwords.

Well, as I provide the --max-length and the wordlist, I don't miss the warning. It does not appear to be of general interest and may degrade performance. So please be patient to get an answer from Magnum or Solar.

[klensy]

Isn't this should be cheap?

if max_word_length_from_wordlist * prince-elem-cnt-max > max_length: print warn

For prince? Lets talk about prince + mask or prince + rules or both?

john --progress-every=5  --no-log --prince=wordlist.txt --max-len=5 --mask=?w?d --rules --stdout
640p 0:00:00:00 100.00% (2021-08-04 20:07) 6400p/s 8ccc7

$ john --progress-every=5  --no-log --prince=wordlist.txt --max-len=5 --mask=?w?d --rules=jumbo --stdout
70280p 0:00:00:00 100.00% (2021-08-04 20:08) 159727p/s cc©7

For wordlist mode, think about RockYou: it is a list containing billions of items.

[klensy]

First password from rockyou.txt(123456 in mine) already be over max-len==5.
Actually, it can be not max_word_length_from_wordlist but any_word_length_from_wordlist.

[magnumripper]

IMHO this is not an issue, the option works as designed and as documented.

[solardiz]

I think this is the intended behavior, and one I would expect. I don't know whether/what we should/can reasonably do to address other expectations. Doing this for PRINCE mode only would possibly only make things worse, by creating expectations that if one mode detects this, all others would too. Also, pre-scanning the wordlist isn't cheap, and again only doing it when the wordlist is tiny or only scanning e.g. first 1000 lines would create wrong expectations. So let's just close this?

@klensy Makes sense to you?

[AlekseyCherepanov]

pre-scanning the wordlist isn't cheap

Yet pre-scanning is done for wordlists under some circumstances (at least all wordlist+rules cases). words variable in wordlist.c is the index for candidates. And according to logs, it is certain that wordlist is read fully before cracking to count lines and calculate size of index. (I found it due to #5126.)

[AlekseyCherepanov]

But prince+rules case is not suitable to warn about bad lengths before cracking. Some analysis on rules might be applied: it might be possible to identify rules that definitely do not affect length, definitely increase length, sometimes increase, same for "decrease", and warn respectively. But then there is external mode that is harder to analyze automatically.

It seems more realistic to handle it as a part of #4032.

[solardiz]

pre-scanning is done for wordlists under some circumstances (at least all wordlist+rules cases).

FWIW, the "all" in "at least all wordlist+rules cases" is wrong - for huge wordlists, we do not currently pre-scan/preload them.

[AlekseyCherepanov]

FWIW, the "all" in "at least all wordlist+rules cases" is wrong - for huge wordlists, we do not currently pre-scan/preload them.

I rechecked. You're right.

$ for k in 10 100 1000 10000 100000 1000000 10000000 100000000 1000000000; do echo; echo $k; seq $k > t.lst; ls -l t.lst; ./john/run/john --log-stderr nt.pw --wordlist=t.lst --rules=':(a' 2>&1 | grep -e index -e mem; done
[...]
10000000
-rw-r--r-- 1 user user 78888897 May 10 15:48 t.lst
0:00:00:00 - memory mapping wordlist (78888897 bytes)
0:00:00:00 - loading wordfile t.lst into memory (78888897 bytes, max_size=150000000)
0:00:00:00 - wordfile had 10000000 lines and required 80000000 bytes for index.

100000000
-rw-r--r-- 1 user user 888888898 May 10 15:48 t.lst
0:00:00:00 - memory mapping wordlist (888888898 bytes)

1000000000
-rw-r--r-- 1 user user 9888888899 May 10 15:48 t.lst