SunMD5 format crashes in some arches

SunMD5 format crashes in all non X86 arches if SIMD and OMP is enabled.

• it fails in clang and gcc;
• on Linux and macOS.

Below is macOS on ARM (see also #5296 (comment) and #5296 (comment)):

E,g.:

Testing: SunMD5 [MD5 128/128 ASIMD 4x2]... (4xOMP) run_tests.sh: line 62: 13013 Abort trap: 6
     $JTR_BIN -test-full=0 --format=cpu

Exit status: 134

The best clue we have so far is something around line 791.

john/src/sunmd5_fmt_plug.c

Line 791 in c8daf44

SIMDmd5body(input_buf_big[group_idx][j], out_buf[group_idx], out_buf[group_idx], SSEi_RELOAD|SSEi_MIXED_IN);

It's more about documenting, but maybe someone can reproduce the issue and find a misuse of 'some' API by the format.

[magnumripper]

Probably something unaligned.

[magnumripper]

I managed to reproduce now. The trick is to use eg. OMP_NUM_THREADS=3 but it still doesn't happen on Linux, only the M1. My current fixes in #5299 doesn't help.

[magnumripper]

Without lldb working, my only clue is this:

Will run 3 OpenMP threads
Testing: SunMD5 [MD5 128/128 ASIMD 4x2]... (3xOMP) Bus error: 10

I think I'll try installing a real gcc and gdb and see what happens. Edit: Apparently I already had gcc-12 and that's what built john. But Homebrew doesn't supply gdb for arm64. I'll try building with native clang "gcc" and see if anything changes.

[magnumripper]

Unfortunately a build with native clang "gcc" never segfaults, so I can't reproduce. I even tried --format=cpu in case some other format would be interfering.

[magnumripper]

Lots of comments in #5299 but here's a better place.

I no longer think this is about alignment at all. Putting that debug print with correct brackets results in a never segfault nor fail (and never shows misalignment).

And to recap, when compiled with GNU gcc it never crashes and we've also ruled out OpenSSL as --without-openssl doesn't make a difference.

Oh, and also my fixes for the aliasing violations doesn't make a difference - so it's not the compiler acting up on UB.