This project showcases my experience with various Azure services for data engineering. I have utilized Azure Data Factory, Azure Data Lake Storage, Azure App Services, Azure Databricks, and Azure Synapse Analytics to build an end-to-end data pipeline and perform analytics on the data.
-
Azure Data Factory: Used for orchestrating and automating data movement and transformation. I have used the HTTP request method to extract data and stored it in Azure Data Lake Storage.
-
Azure Data Lake Storage: Used as a scalable data lake for storing raw and processed data.
-
Azure App Services: Hosted the web application for data visualization and interaction.
-
Azure Databricks: Utilized for data transformation and running machine learning models.
-
Azure Synapse Analytics: Used for running SQL queries and performing analytics on the data.
- ETL Pipeline: Created using Azure Data Factory to extract data, transform it using Databricks, and load it into Data Lake Storage.
- Web Application: Hosted on Azure App Services, providing interactive dashboards for data visualization.
- Analytics: Leveraged Azure Synapse Analytics for running SQL queries and generating insights from the data.
This project has deepened my understanding of Azure services for data engineering, especially in building scalable and efficient data pipelines, performing data transformations at scale, and leveraging analytics for data-driven insights.