大数据知识仓库涉及到数据仓库建模、实时计算、大数据、数据中台、系统设计、Java、算法等。

Example end to end data engineering project.

Replicate data from MySQL, Postgres and MongoDB to ClickHouse®

Nyc_Taxi_Data_Pipeline - DE Project

Ecommerce Realtime Data Pipeline (Data Modeling, Workflow Orchestration, Change Data Capture, Analytical Database and Dashboarding)

Enables Python developers to leverage Debezium's CDC capabilities with custom event handlers and seamless integration.

This project shows how to capture changes from postgres database and stream them into kafka

Repo for CDC with debezium blog post

Built a real-time streaming pipeline to extract stock data, using Apache Nifi, Debezium, Kafka, and Spark Streaming. Loaded the transformed data into Glue database and created real-time dashboards using Power BI and Tableau with Athena. The pipeline is orchestrated using Airflow.

A Smart Traffic Management System for Ho Chi Minh City, Vietnam leveraging batch and real-time data processing, intuitive dashboards, and monitoring tools to optimize traffic flow, enhance safety, and support sustainable urban mobility through advanced analytics and user-friendly applications.

Data Streaming with Debezium, Kafka, Spark Streaming, Delta Lake, and MinIO

Stream CDC into an Amazon S3 data lake in Apache Iceberg format with AWS Glue Streaming using Amazon MSK Serverless and MSK Connect (Debezium)

Stream CDC into an Amazon S3 data lake in Apache Iceberg format with AWS Glue Streaming using Amazon MSK and MSK Connect (Debezium)

Data Pipeline for CDC data from MySQL DB to Amazon S3 through Amazon MSK Serverless using Amazon MSK Connect (Debezium).

Guardian for your Kafka Connect connectors. It check status of connectors and tasks and restart if they are failed

Multi-cloud e-commerce data platform playground with synthetic generators, CDC pipelines, and Terraform-driven infra.

Demos showing how to build with Feldera

Outbox pattern using Debezium and Protobuf serialization

Пример создания CDC через Debezium