LitData Benchmarks

This directory contains simple, ready-to-use scripts for benchmarking data loading and optimization with LitData and FFCV.

• Use the litdata/ folder for LitData-based optimization and streaming.
• Use the ffcv/ folder for FFCV-based dataset conversion and streaming.

You can compare both approaches for your own datasets and training pipelines.

How to use

• See the README in each subfolder for step-by-step instructions.
• All scripts are CLI-based and easy to run.

Why benchmarks?

Benchmarks help you:

• Measure data loading speed and efficiency
• Compare different formats and pipelines
• Choose the best setup for your training

---

For more details, check the README in each subfolder.