log4j2: coverage builds are failing

[vy]

log4j2¹ coverage builds have been failing for quite a while, the last successful coverage build was on 2023-10-06. The coverage failures in the last couple of months have been in the following form:

Starting Step #5
Step #5: Already have image (with digest): gcr.io/oss-fuzz-base/base-runner
Step #5: warning [/corpus/log4j-core-fuzz-test-PatternLayoutFuzzer.zip]:  zipfile is empty
Step #5: Failed to unpack the corpus for log4j-core-fuzz-test-PatternLayoutFuzzer. This usually means that corpus backup for a particular fuzz target does not exist. If a fuzz target was added in the last 24 hours, please wait one more day. Otherwise, something is wrong with the fuzz target or the infrastructure, and corpus pruning task does not finish successfully.
...
Step #5: ********************************************************************************
Step #5: Code coverage report generation failed.
Step #5: To reproduce, run:
Step #5: python infra/helper.py build_image log4j2
Step #5: python infra/helper.py build_fuzzers --sanitizer coverage log4j2
Step #5: python infra/helper.py coverage log4j2
Step #5: ********************************************************************************
Finished Step #5

I am able to locally trigger a coverage failure, but a slightly different one?

$ python infra/helper.py coverage $PROJECT_NAME
INFO:__main__:Downloading corpora for log4j2 project to /home/vy/Projects/google-oss-fuzz~master/build/corpus/log4j2.
warning [/home/vy/Projects/google-oss-fuzz~master/build/corpus/log4j2/log4j-layout-template-json-fuzz-test-JsonTemplateLayoutFuzzer.zip]:  zipfile is empty
ERROR:__main__:Corpus download for log4j-layout-template-json-fuzz-test-JsonTemplateLayoutFuzzer failed: Command '['unzip', '-q', '-o', '/home/vy/Projects/google-oss-fuzz~master/build/corpus/log4j2/log4j-layout-template-json-fuzz-test-JsonTemplateLayoutFuzzer.zip', '-d', '/home/vy/Projects/google-oss-fuzz~master/build/corpus/log4j2/log4j-layout-template-json-fuzz-test-JsonTemplateLayoutFuzzer']' returned non-zero exit status 1..
...

I guess log4j2 corpus is for some reason not getting stored. An infrastructural issue? In #12417, @jonathanmetzman pointed me to google/clusterfuzz#4238. Maybe related?

¹ log4j2 fuzzers were recently revamped in #12304.

[nwellnhof]

cmark coverage build also started to fail with a similar error. This happened suddenly without changes to the cmark repo.

Starting Step #5
Step #5: Already have image (with digest): gcr.io/oss-fuzz-base/base-runner
Step #5: [/corpus/cmark_fuzzer.zip]
Step #5:   End-of-central-directory signature not found.  Either this file is not
Step #5:   a zipfile, or it constitutes one disk of a multi-part archive.  In the
Step #5:   latter case the central directory and zipfile comment will be found on
Step #5:   the last disk(s) of this archive.
Step #5: unzip:  cannot find zipfile directory in one of /corpus/cmark_fuzzer.zip or
Step #5:         /corpus/cmark_fuzzer.zip.zip, and cannot find /corpus/cmark_fuzzer.zip.ZIP, period.
Step #5: Failed to unpack the corpus for cmark_fuzzer. This usually means that corpus backup for a particular fuzz target does not exist. If a fuzz target was added in the last 24 hours, please wait one more day. Otherwise, something is wrong with the fuzz target or the infrastructure, and corpus pruning task does not finish successfully.
Step #5: ********************************************************************************
Step #5: Code coverage report generation failed.
Step #5: To reproduce, run:
Step #5: python infra/helper.py build_image cmark
Step #5: python infra/helper.py build_fuzzers --sanitizer coverage cmark
Step #5: python infra/helper.py coverage cmark
Step #5: ********************************************************************************
Finished Step #5
ERROR
ERROR: build step 5 "gcr.io/oss-fuzz-base/base-runner" failed: step exited with non-zero status: 1

[evverx]

One of the systemd fuzz targets along with the code it covers is missing from the systemd coverage report.

https://oss-fuzz-build-logs.storage.googleapis.com/log-81df5e7e-741d-41d0-a06f-da7f8c36f853.txt says

Step #5: Already have image (with digest): gcr.io/oss-fuzz-base/base-runner
Step #5: [/corpus/fuzz-journal-remote.zip]
Step #5:   End-of-central-directory signature not found.  Either this file is not
Step #5:   a zipfile, or it constitutes one disk of a multi-part archive.  In the
Step #5:   latter case the central directory and zipfile comment will be found on
Step #5:   the last disk(s) of this archive.
Step #5: unzip:  cannot find zipfile directory in one of /corpus/fuzz-journal-remote.zip or
Step #5:         /corpus/fuzz-journal-remote.zip.zip, and cannot find /corpus/fuzz-journal-remote.zip.ZIP, period.
...
Step #5: Running fuzz-journal-remote
Step #5: Error occured while running fuzz-journal-remote:
Step #5: WARNING: Failed to find function "__sanitizer_acquire_crash_state".
Step #5: WARNING: Failed to find function "__sanitizer_print_stack_trace".
Step #5: WARNING: Failed to find function "__sanitizer_set_death_callback".
Step #5: INFO: Running with entropic power schedule (0xFF, 100).
Step #5: INFO: Seed: 1287985286
Step #5: No such file or directory: /corpus/fuzz-journal-remote; exiting

The latest coverage report where it's present was built on November 22: https://storage.googleapis.com/oss-fuzz-coverage/systemd/reports/20241122/linux/src/systemd/src/journal-remote/report.html

[jonathanmetzman]

Ouch, there's definitely a bug in the cmark and systemd cases. Looking into this.

[jonathanmetzman]

Ouch, there's definitely a bug in the cmark and systemd cases. Looking into this.

Actually, cmark seems to be working.
Systemd should be working but I haven't been able to verify as I need to wait for the last failing pruning job to finish.
Also, we are fuzzing, it's just the coverage reporting mechanism is broken

[evverx]

To judge from https://storage.googleapis.com/oss-fuzz-coverage/systemd/reports/20241223/linux/src/systemd/src/journal-remote/report.html it seems to be working. Thanks!

[korovindenis]

The issue has not been resolved for #12878