Bulk Stash is a docker rclone service to sync, or copy, files between different storage services. For example, you can copy files either to or from a remote storage services like Amazon S3 to Google Cloud Storage, or locally from your laptop to a remote storage.

A fully incremental model, that transforms raw web event data generated by the Snowplow JavaScript tracker into a series of derived tables of varying levels of aggregation.

Learn the basics of Apache Kafka® from leaders in the Kafka community with these video courses covering the Kafka ecosystem and hands-on exercises.

Open Source Data Platform on Kubernetes

Finnhub data streaming pipeline for real-time Bitcoin trades analysis.

Built Apache Airflow DAGs to automate Yahoo Finance stock data ingestion, storage, and querying, then extended with a Python log analyzer to monitor execution errors. Demonstrates orchestration, scheduling, operator use, and pipeline monitoring.

A sample data pipeline for transforming invoice images and CSV files into beautiful numbers

🧬 A Streamlit-based web app for immune gene variant annotation with AWS integration (EC2, S3, DynamoDB, DVC)

Real-time MySQL Change Data Capture (CDC) to AWS S3 using Debezium + Kafka + Docker

Near real-time data replication pipeline from MySQL to PostgreSQL for analytics, reporting, and downstream systems. Supports inserts, updates, deletes, and initial snapshot replication.

Secure, self-hosted ETL automation pipeline. Built with n8n, PostgreSQL, and Docker Compose under Zero-Trust principles.

To use dbt as an orchestration tool to process a static file and join two data sources together. This repository can be used as a template example of creating a dbt pipeline with testing. See the two simple sets below to using the dbt pipeline to generate tables in BigQuery (GCP).

End-to-end data engineering pipeline for analyzing heart attack prediction in Indonesia. Automates data ingestion, transformation, and visualization using GCP (Terraform, BigQuery, Cloud Storage), Apache Airflow, dbt, and Python scripts. Provides actionable insights via Power BI dashboards.

GCP-based Regulatory Reporting Lakehouse — Tier-1 Swiss Bank (Simulated Case Study):- Documentation-only repo illustrating a cloud-native data lakehouse architecture for regulatory reporting on Google Cloud Platform (GCS + BigQuery + Dataflow + Composer). Includes ADRs, runbooks, and compliance data contracts.

A modular, contract-driven HPC pipeline for converting SRA archives to FASTQ using a pinned SRA Toolkit, featuring strict preflight validation, an explicit SLURM execution ABI, parallel array processing, and restart-safe per-accession isolation.

Apache NiFi configuration and workflow templates

Clinical cohort construction and metric derivation using DuckDB SQL. Explores healthcare encounter data, profiles keys and dates, identifies duplication patterns, and prepares a reproducible pipeline for building an analysis-ready cohort dataset.

Enterprise airline attendance big data pipeline with Hadoop MapReduce, Docker, synthetic dataset generation, and validation.

A modular, contract-driven HPC pipeline for deterministic alignment of paired-end FASTQ data using BWA and samtools, producing sorted and indexed BAM files with strict SLURM execution, explicit ABI contracts, and comprehensive preflight validation.

A contract‑driven HPC pipeline for FASTQ trimming using BBDUK or Trimmomatic, featuring strict preflight validation, explicit SLURM ABI propagation, deterministic tool installation, and restart‑safe per-sample execution.