A stand-alone test framework that allows to write unit tests for Data Factory pipelines on Microsoft Fabric, Azure Data Factory and Azure Synapse Analytics.
An end-to-end data engineering pipeline that fetches data from Wikipedia, cleans and transforms it with Apache Airflow and saves it on Azure Data Lake. Other processing takes place on Azure Data Factory, Azure Synapse and Tableau.
A python application that leverages Microsoft Azure to build a fully functional trading system.
End-to-end ETL pipeline in the Microsoft Azure cloud - (Jun '24 - Jul '24)
Formula 1 race data engineering project which utilises azure services and databricks to ingest and analyse the data.
Data engineering project using Azure Data Factory and Apache Airflow
A data pipeline project build on databricks and azure to demostrate lifecycle of a cloud data project.
Python script that cleans the JSON document used by an Azure Data Factory or Azure Synapse Copy Activity to specify the source-sink mapping.
A demand forecasting pipeline deployed on Azure and AWS
Building a next-generation hybrid data pipeline architecture that combines the power of Microsoft Fabric, Azure Cloud, and Power BI. This pipeline is engineered to tackle the challenges of real-time data ingestion, multi-layered processing, and analytics, delivering business-critical insights.
In this project, I've created an end-to-end ETL pipeline and subsequently developed a machine learning model to predict the price of Amazon products based on several product-related features.
A repository to continue my education on Azure Data Services.
ETL pipeline tailored for Olympics data
A B2B solution and architecture. This includes code for Azure Function App, SQL Server. Integration with azure synapse and event hubs as well.
Data ingestion and transformation pipeline for AirBnB using Azure Data Factory, Synapse, CosmosDB, and ADLS for automated CDC and analytical reporting.
Projeto utilizando a base de CNPJ da Receita Federal
Azure Data Factory v2 datapipeline sample code snippets...
This repository contains code for an end-to-end IoT data pipeline using Azure services. It ingests, processes, and stores IoT device data from AWS S3 to Azure Data Lake Storage and Azure SQL Database, leveraging Azure Data Factory and Azure Functions for seamless integration and automation.
Using LLMS to extract medical Data for Doctors and Hospitals
Treinamento Azure Data Factory + Azure Databricks + Azure Analysis Services + Power BI
Add a description, image, and links to the azure-data-factory topic page so that developers can more easily learn about it.
To associate your repository with the azure-data-factory topic, visit your repo's landing page and select "manage topics."