maintain periodically updated corpora for popular data formats

[kcc]

Already discussed it in a few places, but I don't think we have a bug for this.
Since now we have corpus cross-pollination between different targets it's worth having
a corpus that is not specific to any target and periodically merge it into every target.

Such corpus may consist of a list of URLs that we should periodically download to get the new samples.

(The ones that we recently discussed are about fonts, e.g.
https://github.com/unicode-org/text-rendering-tests or https://github.com/google/fonts or https://github.com/adobe-fonts/adobe-variable-font-prototype)

[inferno-chromium]

I think a global cloud storage bucket where we can push files from time to time. But need to remember to clean this up, maintenance headache. But we do get this usecase from time to time.

@Dor1s @oliverchang @tanin47 - lets brainstorm and implement this. i am even thinking this could be a new version of our data bundle management on ClusterFuzz, we need to migrate away from blobstore anyway.

[Dor1s]

We may have a bucket with some auto-deletion period, (e.g. 30 days), let's call it corpus-candidates. So the workflow would be:

1. grab new files from different sources (URLs as suggested above, hopefully something else)
2. put them into corpus-canditates
3. during next 30 days these files will be merged into all fuzzers
4. files get deleted from corpus-candidates - this is fine, since they already made an impact to corpora of our fuzzers

I imagine this like a Furnace, where we can drop some firewood from time to time :)

[MarkusTeufelberger]

Similar to @Dor1s' suggestion, I'd suggest dropping all found/generated/collected test cases globally in a "furnace" and then having all individual fuzzers run a corpus minimization ("distillation"?) on that, which is the new seed corpus for the day/hour/next run of that fuzzer. If a file is actively used, it will be refreshed, since it will be resubmitted once a worker finished (after a local corpus minimization step before submitting to the furnace).

Since files in there expire anyways if they are not used in any corpus, it might even be possible to allow for external user submissions to that pool.

[alex]

@Dor1s do you want to delete the them? Say we have a bunch of jpg parsers today, and in two months we add a new jpg parser fuzzer, it'd be good to seed it with the existing shared corpus.

[Dor1s]

@alex yeah, I though the idea is to delete them after some time, otherwise we'll have to many files which are useless for existing fuzzers.
I agree with your point about new fuzzers and our possible desire to feed them with corpora that we've already deleted. In this case, we can store snapshots of our Furnace.
For example, take a snapshot every 29 days. Then, once we've got a new fuzzer, we'll run it on all those snapshots for the first time -- a kind of warmup to get more interesting corpus from the early beginning of fuzzer's life.

[inferno-chromium]

This is done, please upload stuff to
gs://global-corpus.clusterfuzz-external.appspot.com

Put in a sub-directory for easy understanding. If you plan to put a ton of stuff, please give heads up. This is synced to bot local, so lets keep this in limits like < 10 gb overall or something.