This project demonstrates a full-scale end-to-end data engineering pipeline built on Microsoft Azure.
It covers data ingestion, storage, transformation, analytics, and visualization using industry-standard tools.
The pipeline ingests data from an on-premises SQL Server, processes it using a Medallion Architecture (Bronze → Silver → Gold), and delivers insights through an interactive dashboard.
- Source: On-Prem SQL Server (AdventureWorks)
- Ingestion: Azure Data Factory
- Storage: Azure Data Lake Gen2
- Processing: Azure Databricks (Bronze → Silver → Gold)
- Analytics: Azure Synapse Analytics
- Visualization: Power BI
- Security & Governance: Azure Active Directory, Azure Key Vault
- Lookup Activity
- Dynamically retrieves all table names from the source database
- ForEach Activity
- Iterates over tables and copies data to Data Lake (Bronze Layer)
- Databricks Notebook – Bronze to Silver
- Data cleansing, schema enforcement, standardization
- Databricks Notebook – Silver to Gold
- Aggregations, business logic, analytics-ready datasets
- Scheduled Trigger runs daily
- Successful executions monitored via ADF Trigger Runs
- Enables automated and reliable data refresh
- Azure Data Factory
- Azure Databricks
- Azure Data Lake Storage Gen2
- Azure Synapse Analytics
- Azure Key Vault
All resources are deployed within a single resource group for centralized management.
- Database: AdventureWorks
- Tables ingested:
- SalesLT.Customer
- SalesLT.Product
- SalesLT.SalesOrderHeader
- SalesLT.SalesOrderDetail
- Product, Category, Address tables
- Total Sales
- Total Orders
- Number of Products
- Top 10 Best-Selling Products
- Sales by Category
- Price vs Sales Analysis
- Interactive slicers for category and price range
Built using Power BI on top of curated Gold Layer data.
| Layer | Description |
|---|---|
| Bronze | Raw ingested data from SQL Server |
| Silver | Cleaned, standardized, validated data |
| Gold | Aggregated, analytics-ready datasets |
- Azure Active Directory for authentication
- Azure Key Vault for secrets management
- Secure access between ADF, Databricks, and Storage
- Azure Data Factory
- Azure Databricks
- Azure Data Lake Gen2
- Azure Synapse Analytics
- Azure SQL Server
- Power BI
- Python (PySpark)
- SQL
- Building scalable cloud data pipelines
- Implementing Medallion Architecture
- Automating ingestion using ADF
- Data transformations with Databricks
- Enterprise-grade security in Azure
- Delivering insights through dashboards
- Incremental data loads (CDC)
- CI/CD with Azure DevOps
- Data quality checks
- Cost optimization strategies
- Streaming ingestion with Event Hub
⭐ If you found this project useful, feel free to star the repository!