Data Integration Pipelines for NYC Payroll Data Analytics with Azure Data Factory (udacity Final Project)
The City of New York would like to develop a Data Analytics platform on Azure Synapse Analytics to accomplish two primary objectives:
Analyze how the City's financial resources are allocated and how much of the City's budget is being devoted to overtime. Make the data available to the interested public to show how the City’s budget is being spent on salary and overtime pay for all municipal employees. You have been hired as a Data Engineer to create high-quality data pipelines that are dynamic, can be automated, and monitored for efficient operation. The project team also includes the city’s quality assurance experts who will test the pipelines to find any errors and improve overall data quality.
The source data resides in Azure Data Lake and needs to be processed in a NYC data warehouse in Azure Synapse Analytics. The source datasets consist of CSV files with Employee master data and monthly payroll data entered by various City agencies.
For this project, you'll do your work in the Azure Portal, using several Azure resources including:
Azure Data Lake Gen2 (Storage account with Hierarchical Namespaces checkbox checked when creating) Azure SQL DB Azure Data Factory Azure Synapse Analytics
For the project data, here is the direct link to Udacity
Otherwise, You can try with data found online.