Skip to content

Commit

Permalink
Archive synapse samples, update readme to remove "Modern Data Warehou…
Browse files Browse the repository at this point in the history
…se" terminology, replace with "Medallion" (#757)

* Archive synapse samples, update readme to remove Modern Data Warehouse

* fix linting issues

* add utilities folder, update docs

* add medallion doc link

* image fixes

* add iamge

* add exception to markdown lint for br

* png fix

* Apply suggestions from code review

Co-authored-by: Anuj Parashar <promisinganuj@gmail.com>

* fix: pr

---------

Co-authored-by: Anuj Parashar <promisinganuj@gmail.com>
  • Loading branch information
devlace and promisinganuj authored Nov 6, 2024
1 parent 3ba42b4 commit 548d504
Show file tree
Hide file tree
Showing 220 changed files with 164 additions and 34 deletions.
3 changes: 3 additions & 0 deletions .markdownlint.yml
Original file line number Diff line number Diff line change
@@ -1,3 +1,6 @@
MD013: false
MD024: false
MD037: false
MD033:
allowed_elements:
- "br"
44 changes: 18 additions & 26 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,49 +13,41 @@ products:
- azure-databricks
- azure-stream-analytics
- azure-synapse-analytics
description: "Code samples showcasing how to apply DevOps concepts to the modern data warehouse architecture leveraging different Azure data technologies."
description: "Code samples showcasing how to apply DevOps concepts to common data engineering patterns and architectures leveraging different Microsoft data platform technologies."
---

# DataOps for the Modern Data Warehouse
# DataOps

This repository contains numerous code samples and artifacts on how to apply DevOps principles to data pipelines built according to the [Modern Data Warehouse (MDW)](https://learn.microsoft.com/en-au/azure/architecture/solution-ideas/articles/enterprise-data-warehouse) architectural pattern on Microsoft Azure.
This repository contains numerous code samples and artifacts on how to apply DevOps principles to common data engineering patterns and architectures utilizing Microsoft data platform technologies.

The samples are either focused on a single azure service (**Single Tech Samples**) or showcases an end to end data pipeline solution as a reference implementation (**End to End Samples**). Each sample contains code and artifacts relating one or more of the following
The samples are either focused on a single microsoft service ([Single-Technology Samples](#single-technology-samples)) or showcases an end-to-end data pipeline solution as a reference implementation ([End-to-End Samples](#end-to-end-samples)). Each sample contains code and artifacts related to one or more of the following capabilities:

- Infrastructure as Code (IaC)
- Build and Release Pipelines (CI/CD)
- Testing
- Observability / Monitoring

## Single Technology Samples
## Single-Technology Samples

- [Microsoft Fabric](./single_tech_samples/fabric/README.md)
- [CI/CD - Microsoft Fabric](./single_tech_samples/fabric/fabric_ci_cd/README.md)
- [Feature engineering on Microsoft Fabric](./single_tech_samples/fabric/feature_engineering_on_fabric/README.md)
- [Azure SQL database](./single_tech_samples/azuresql/README.md)
- [CI/CD - Azure SQL database](./single_tech_samples/azuresql/azuresql_ci_cd/README.md)
- [Azure Databricks](single_tech_samples/databricks/)
- [IaC - Basic deployment](single_tech_samples/databricks/databricks_ci_cd/README.md)
- [Azure Data Factory](./single_tech_samples/datafactory/README.md)
- [CI/CD - Auto publish](./single_tech_samples/datafactory/adf_cicd_auto_publish/README.md)
- [Data pre-processing using Azure Batch](./single_tech_samples/datafactory/adf_data_pre_processing_with_azure_batch/README.md)
- [Azure Synapse Analytics](./single_tech_samples/synapseanalytics/README.md)
- [Serverless best practices](./single_tech_samples/synapseanalytics/synapse_serverless/README.md)
- [Azure Stream Analytics](./single_tech_samples/streamanalytics/README.md)
- [CI/CD - Azure Stream Analytics](./single_tech_samples/streamanalytics/streamanalytics_ci_cd/README.md)
| Technology | Samples |
| ---------- | ------- |
| [Microsoft Fabric](./single_tech_samples/fabric/README.md) | ▪️ [CI/CD for Microsoft Fabric](./single_tech_samples/fabric/fabric_ci_cd/README.md)<br> ▪️ [Feature engineering on Microsoft Fabric](./single_tech_samples/fabric/feature_engineering_on_fabric/README.md) |
| [Azure SQL database](./single_tech_samples/azuresql/README.md) | ▪️ [CI/CD for Azure SQL database](./single_tech_samples/azuresql/azuresql_ci_cd/README.md) |
| [Azure Databricks](single_tech_samples/databricks/) | ▪️ [CI/CD for Azure Databricks](single_tech_samples/databricks/databricks_ci_cd/README.md) |
| [Azure Data Factory](./single_tech_samples/datafactory/README.md) | ▪️ [CI/CD for ADF with Auto publish](./single_tech_samples/datafactory/adf_cicd_auto_publish/README.md)<br> ▪️ [Data pre-processing using Azure Batch](./single_tech_samples/datafactory/adf_data_pre_processing_with_azure_batch/README.md) |
| [Azure Stream Analytics](./single_tech_samples/streamanalytics/README.md) | ▪️ [CI/CD for Azure Stream Analytics](./single_tech_samples/streamanalytics/streamanalytics_ci_cd/README.md) |

## End to End samples
## End-to-End Samples

### Parking Sensor Solution
### DataOps for Medallion with Azure Data Factory and Azure Databricks

This demonstrates batch, end-to-end data pipeline following the MDW architecture, along with a corresponding CI/CD process.
This [sample](e2e_samples/parking_sensors/) demonstrates batch, end-to-end data pipeline utilizing Azure Data Factory and Azure Databricks built according to the [medallion architecture](https://learn.microsoft.com/en-us/azure/databricks/lakehouse/medallion), along with a corresponding CI/CD process, observability and automated testing.

![Architecture](docs/images/CI_CD_process_simplified.png?raw=true "Architecture")
[![Medallion with Azure Data Factory and Azure Databricks](docs/images/CI_CD_process_simplified.png "Architecture")](e2e_samples/parking_sensors/)

This has two version of the solution:
### DataOps for Medallion with Microsoft Fabric

- [Azure Data Factory and Azure Databricks Version](e2e_samples/parking_sensors/)
- [Azure Synapse Version](e2e_samples/parking_sensors_synapse/)
- Coming soon.

## Contributing

Expand Down
1 change: 1 addition & 0 deletions archive/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,3 +25,4 @@ Here is a list of the archived samples:
- [Azure Storage lifecycle management](./../archive/single_tech_samples/storage_lifecycle_management/README.md)
- [Metadata driven module loading in Azure Synapse](./../archive/single_tech_samples/synapse_loading_dynamic_modules/README.md)
- [Azure Synapse integration testing](./../archive/single_tech_samples/synapse_integration_testing/README.md)
- [Serverless pool best practices](./../archive/single_tech_samples/synapse_serverless/README.md)
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
Binary file removed docs/images/CI_CD_process_simplified.PNG
Diff not rendered.
Loading

0 comments on commit 548d504

Please sign in to comment.