This project demonstrates an end-to-end Data Engineering pipeline built on Databricks. It showcases modern ETL/ELT practices using Delta Lake, Delta Live Tables (DLT), Auto Loader, and Unity Catalog. The solution follows the Medallion Architecture (Bronze → Silver → Gold) to ensure scalable, reliable, and high-quality data processing.
python data-science etl azure realtime pyspark secu adf databricks dataengineering blob-storage delta-lake adlsgen2 medallion-architecture unitycatalog deltaoptimization
-
Updated
Mar 28, 2026 - Jupyter Notebook