Add a FMT_FALSE companying FMT_NOT_EXACT

[magnumripper]

There are two types of formats that can emit several candidates for one same hash: One example is CRC32, where each crack is correct so they're not false positives - any of them is correct. An example of the other type is DMG (at least in older code) which could emit actual false candidates - they will not work for opening that DMG file!

Current code has only the false positive formats setting FMT_NOT_EXACT (after a while of scope-creep from its original idea, because the two situations really should be separated). If you want more candidates from CRC32, just add --keep-guessing option.

We could want to add a FMT_FALSE flag that allows us to separate them, and describe this better. We'd replace all current FMT_NOT_EXACT with FMT_FALSE, and then use FMT_NOT_EXACT for CRC32 and the likes.

FMT_NOT_EXACT currently prints warnings like "Note: This format may emit false positives, so it will keep trying even after finding a possible candidate." or (using --no-keep-guessing) "Note: This format may emit false positives, but we will stop after finding first".

Having both flags, those existing messages should be for FMT_FALSE instead, while the new FMT_NOT_EXACT would be more like "Note: This format has more than one correct password for each hash, (...)"

[magnumripper]

After implementing this, we could have FMT_FALSE formats (like old DMG) default to keep trying (like they do now) but print a notice about --no-keep-trying.
For FMT_NOT_EXACT (like CRC32) we should probably default to not keep trying (like we do now) but conversely print a hint about --keep-trying.

[magnumripper]

We might (then) also want to refactor FMT_NOT_EXACT to something less confusing, such as FMT_HAS_COLLISIONS

[magnumripper]

Formats with actual collsions, off the top of my head: CRC-32, oldoffice (RC4 key truncated to 40 bits), perhaps BFEgg (some weirdness with repeated patterns). I'm pretty sure there are more (but we could want to agree upon [and add a comment in formats.h] at what threshold we'd set the flag - perhaps <= 48 bits or so?).

[magnumripper]

Also, Hashcat has a "collider mode" for PDF versions 1.1-1.3, not sure if we support them or whether it applies to this context.

Edit: It seems we do, $pdf$1*2*40*(...) is also 40-bit RC4

[solardiz]

I'm not convinced we need to change anything, but if we introduce an FMT_FALSE like you describe (or maybe FMT_HAS_FALSE_POSITIVES?), then the other flag should in fact be called FMT_HAS_COLLISIONS.

Speaking of collisions, there are also related issues with 7-bitness and truncation. We already have separate flags for those, but I imagine there could be weird cases where one would want to see all words in a list that match a hash despite of 7-bitness and/or truncation getting in the way. For example, when having related sets of passwords hashed as descrypt and bcrypt, first crack the descrypt and get all matches from RockYou, then test those against the bcrypt. So e.g. if both "nightwing" and "nightwind" match a descrypt hash, then actually detect and record both because the difference between them will matter when cracking the bcrypt.

Edit: Related, should these flags prevent exclusion of previously cracked hashes found in john.pot? In other words, do they take effect only within one john invocation or also across several invocations?

I think in contests, team john-users occasionally inadvertently had cracks with 8-bit characters against descrypt, which later prevented the simpler 7-bit-only cracks against those and possibly resulted in fewer cracks of other hash types (some base words not automatically found because of the inadvertent 8-bit chars). However, a feature in john will not fully solve this: it'd only help get the hash cracked another way within the same john invocation or at best several, but if a hash gets recorded as cracked in the team's shared files, then it'll also be excluded from further attacks by all team members unless those scripts are adjusted too.

[magnumripper]

I think in contests, team john-users occasionally inadvertently had cracks with 8-bit characters against descrypt, which later prevented the simpler 7-bit-only cracks against those and possibly resulted in fewer cracks of other hash types (some base words not automatically found because of the inadvertent 8-bit chars). However, a feature in john will not fully solve this: it'd only help get the hash cracked another way within the same john invocation or at best several, but if a hash gets recorded as cracked in the team's shared files, then it'll also be excluded from further attacks by all team members unless those scripts are adjusted too.

For that very issue, the canonical solution is definitely to store 7-bit passwords in the pot file for 7-bit formats no matter what cracked it. Specifically, it could show like

böse          (user5)

while cracking, even though the pot entry would be bvse (assuming 8859-1).

I recall you opted against this at some point in time. And I agree it would be confusing once some clueless user runs --show and don't recognize what's in there compared to what was shown. The latter issue could be solved eg. by printing it as

böse          (user5)	- Note: stored as "bvse" due to being 7-bit hash

or

bvse          (user5)	- Note: from candidate "böse" but hash is 7-bit

...or something like that.

Another alternative is to simply store both. Not sure how current --show would handle that, I recall we have some open issue for such things.

[magnumripper]

Another issue for contests (but only contests, me thinks) might be that we always store an 8-bit NT password as UTF-8, no matter how we cracked it. At least with current defaults IIRC.

This applies to any format having FMT_UNICODE flag set.