Optimize how files are stored or processed

[schnuerle]

Once the RDS is complete in Phase 1, we'd like to optimize the data storage and processing process if possible. If you have ideas on how to do this, you can make a pull request to add support for it with examples to this repo so everyone can benefit.

One idea is to store each JSON file in a zip, thus reducing storage costs.

Use this issue as a place to discuss and collaborate.

[Riebart]

Having done this before, LZ4 is pure magic, compressing fast, and decompressing really fast. We were able to lift the throughput out of S3 by almost an order of magnitude during batch processing by using LZ4 compressed CSVs in S3.