A collection of from-scratch, zero-dependency C implementations of data engineering pipelines, lakehouse architectures, and analytical processing engines. Each module models real data infrastructure — from ETL/ELT pipelines to OLAP engines, data lakes, lakehouse table formats, stream processing, and data quality frameworks. Modules map to industry standards and reference architectures.
| Module | Topics | Key References |
|---|---|---|
| mini-data-lake | Object storage (S3-like), partition scheme, file formats (Parquet/ORC sim), data catalog | AWS S3, Apache Hadoop |
| mini-data-warehouse | Star/snowflake schema, fact/dimension tables, SCD types, aggregate navigation, query federation | Kimball DWH Toolkit |
| mini-lakehouse | Delta Lake, Apache Iceberg, Apache Hudi, ACID transactions on data lake, time travel, schema evolution | Delta Lake, Iceberg Spec |
| mini-etl-elt | Extract-Transform-Load vs ELT, pipeline DAG, incremental load (CDC), merge/upsert, data validation | Apache Airflow, dbt |
| mini-batch-processing | MapReduce model, shuffle/sort, partition, combiner, DAG execution, task scheduling | MapReduce paper, Spark |
| mini-stream-processing | Event time vs processing time, windowing (tumbling/sliding/session), watermark, exactly-once, state store | Flink, Kafka Streams, Beam |
| mini-olap-engine | Columnar storage, vectorized execution, SIMD filtering, late materialization, zone maps, data skipping | ClickHouse, DuckDB, Vertica |
| mini-data-modeling | Dimensional modeling (Kimball), Data Vault 2.0, Inmon CIF, entity-relationship, normal forms | Kimball, Data Vault |
| mini-data-quality | Data profiling, schema validation, anomaly detection, completeness/consistency checks, SLO monitoring | Great Expectations, Deequ |
| mini-data-lineage | Column-level lineage graph, transformation tracking, impact analysis, provenance model | OpenLineage, Marquez |
| mini-meta-management | Data catalog, schema registry, glossary, tag system, search/discovery, access control | Apache Atlas, DataHub |
| mini-bi-analytics | Semantic layer (metrics/dimensions), query builder, OLAP cube, dashboard model, caching | Looker, Superset, Metabase |
| mini-feature-store | Feature registry, online/offline serving, point-in-time correct joins, feature versioning, transformation | Feast, Tecton |
| mini-rec-risk-pipeline | Recommendation pipeline (recall/rank), risk scoring pipeline, feature engineering, model serving, feedback loop | Netflix/TikTok arch |
- Zero external dependencies — pure C (C99/C11), only
libcandlibm - Self-contained modules — each directory has its own
Makefile,include/,src/,examples/,demos/,tests/ - Pipeline simulation — educational models of data engineering patterns
- Theory-to-practice mapping — every module includes
docs/with reference-alignment notes - Practical demos — MapReduce executor, lakehouse time travel, stream windowing engine, and more
cd mini-etl-elt
make all # build everything
make test # run testsRequires GCC and GNU Make.
mini-data-engine-lakehouse/
├── mini-data-lake/ # Data Lake
├── mini-data-warehouse/ # Data Warehouse
├── mini-lakehouse/ # Lakehouse (Delta/Iceberg/Hudi)
├── mini-etl-elt/ # ETL & ELT Pipelines
├── mini-batch-processing/ # Batch Processing
├── mini-stream-processing/ # Stream Processing
├── mini-olap-engine/ # OLAP Engine
├── mini-data-modeling/ # Data Modeling
├── mini-data-quality/ # Data Quality
├── mini-data-lineage/ # Data Lineage
├── mini-meta-management/ # Metadata Management
├── mini-bi-analytics/ # BI & Analytics
├── mini-feature-store/ # Feature Store
└── mini-rec-risk-pipeline/ # Rec & Risk Pipeline
MIT