[Anuj] Adding Fabric CI/CD sample

.markdownlinkcheck.json

@@ -18,6 +18,9 @@
         {
             "pattern": "^https://azure.microsoft.com/en-us/products/"
         },
+        {
+            "pattern": "^https://learn.microsoft.com/azure/"
+        },
         {
             "pattern": "^https://dev.azure.com"
         }

README.md

@@ -30,6 +30,7 @@ The samples are either focused on a single azure service (**Single Tech Samples*
 ## Single Technology Samples
 
 - [Microsoft Fabric](./single_tech_samples/fabric/README.md)
+  - [CI/CD - Microsoft Fabric](./single_tech_samples/fabric/fabric_ci_cd/README.md)
   - [Feature engineering on Microsoft Fabric](./single_tech_samples/fabric/feature_engineering_on_fabric/README.md)
 - [Azure SQL database](./single_tech_samples/azuresql/README.md)
   - [CI/CD - Azure SQL database](./single_tech_samples/azuresql/azuresql_ci_cd/README.md)

archive/e2e_samples/mdw_governance/README.md

@@ -21,7 +21,7 @@ This sample demonstrates how to provision end-to-end modern data warehouse solut
 
 ## Solution Overview
 
-This solution sets up an Azure Data Lake storage account, with two containers: Datalake and Dropzone. The folder structure in datalake is structured to enable data tiering (Bronze, Silver, Gold), hold shared data (Reference) and shared libraries (Sys). Azure Data Factory instance with linked services connecting to the Azure Data Lake, Azure Key Vault, Azure Databricks and Azure Purview. Application Insights is used for event logging and Office365 API Connection in combination with Logic Apps is used for sending notification emails. A Virtual Network and Private Endpoints are deployed for Data Lake and Key Vault, however the firewall on the services is left open in the initial deployment. Additionally the Databricks virtual network is not peered to the solution's and the databricks public IP is enabled. To fully secure the solution, the Virtual Network should be peered to an internal corporate network, and firewall closed on those services, and follow the [Databricks single tech sample](https://github.com/Azure-Samples/modern-data-warehouse-dataops/tree/main/single_tech_samples/databricks/sample2_enterprise_azure_databricks_environment) for more information on locking down a databricks environment.
+This solution sets up an Azure Data Lake storage account, with two containers: Datalake and Dropzone. The folder structure in datalake is structured to enable data tiering (Bronze, Silver, Gold), hold shared data (Reference) and shared libraries (Sys). Azure Data Factory instance with linked services connecting to the Azure Data Lake, Azure Key Vault, Azure Databricks and Azure Purview. Application Insights is used for event logging and Office365 API Connection in combination with Logic Apps is used for sending notification emails. A Virtual Network and Private Endpoints are deployed for Data Lake and Key Vault, however the firewall on the services is left open in the initial deployment. Additionally the Databricks virtual network is not peered to the solution's and the databricks public IP is enabled. To fully secure the solution, the Virtual Network should be peered to an internal corporate network, and firewall closed on those services, and follow the [Databricks single tech sample](./../../single_tech_samples/databricks_enterprise_azure_databricks_environment/README.md) for more information on locking down a databricks environment.
 
 The Azure Data Factory contains an ADF Pipeline that is stored in a git repository, that is taking data from the Dropzone and ingesting it into the bronze folder, after anonymizing its content using [Presidio](https://github.com/microsoft/presidio). The data files for running the pipeline can be found in [Data](./data) folder of this repository.
 

archive/single_tech_samples/databricks_enterprise_azure_databricks_environment/README.md

@@ -31,8 +31,6 @@ In this sample we focus on hardening the security around the Azure Databricks en
 
 3.[Azure Private Links](https://docs.microsoft.com/en-us/azure/private-link/private-link-overview) - To secure connectivity with dependent PaaS services.
 
-This sample is also aligned to the implementation pattern published by Databricks around [Data Exfiltration Protection with Azure Databricks](https://databricks.com/blog/2020/03/27/data-exfiltration-protection-with-azure-databricks.html).
-
 The sample implements automating the provisioning of the required services and configurations using the [Infrastructure as Code](https://docs.microsoft.com/en-us/dotnet/architecture/cloud-native/infrastructure-as-code) pattern.
 
 ### 1.1. Scope

e2e_samples/parking_sensors/README.md

@@ -134,7 +134,7 @@ The following summarizes key learnings and best practices demonstrated by this s
 ### 7. Monitor infrastructure, pipelines and data
 
 - A proper monitoring solution should be in-place to ensure failures are identified, diagnosed and addressed in a timely manner. Aside from the base infrastructure and pipeline runs, data quality should also be monitored. A common area that should have data monitoring is the malformed record store.
-- As an example this repository showcases how to use open source framework [Great Expectations](https://docs.greatexpectations.io/docs/) to define, measure and report data quality metrics at different stages of the data pipeline. Captured Data Quality metrics are reported to Azure Monitor for further visualizing and alerting. Take a look at sample [Data Quality report](docs/images/data_quality_report.png) generated with Azure Monitor workbook. Great Expectations can be configured to generate HTML reports and host directly as static site on Azure Blob Storage. Read more on [How to host and share Data Docs on Azure Blob Storage](https://legacy.docs.greatexpectations.io/en/latest/guides/how_to_guides/configuring_data_docs/how_to_host_and_share_data_docs_on_azure_blob_storage.html).
+- As an example this repository showcases how to use open source framework [Great Expectations](https://docs.greatexpectations.io/docs/) to define, measure and report data quality metrics at different stages of the data pipeline. Captured Data Quality metrics are reported to Azure Monitor for further visualizing and alerting. Take a look at sample [Data Quality report](docs/images/data_quality_report.png) generated with Azure Monitor workbook. Great Expectations can be configured to generate HTML reports and host directly as static site on Azure Blob Storage. Read more on [How to host and share Data Docs on Azure Blob Storage](https://docs.greatexpectations.io/docs/oss/guides/setup/configuring_data_docs/host_and_share_data_docs/).
 
 ## Key Concepts
 
@@ -199,13 +199,13 @@ More resources:
 
 #### Databricks
 
-- [Monitoring Azure Databricks with Azure Monitor](https://docs.microsoft.com/en-us/azure/architecture/databricks-monitoring/)
+- [Monitoring Azure Databricks with Azure Monitor](https://learn.microsoft.com/azure/architecture/databricks-monitoring/)
 - [Monitoring Azure Databricks Jobs with Application Insights](https://msdn.microsoft.com/en-us/magazine/mt846727.aspx)
 
 #### Data Factory
 
-- [Monitor Azure Data Factory with Azure Monitor](https://docs.microsoft.com/en-us/azure/data-factory/monitor-using-azure-monitor)
-- [Alerting in Azure Data Factory](https://azure.microsoft.com/en-us/blog/create-alerts-to-proactively-monitor-your-data-factory-pipelines/)
+- [Monitor Azure Data Factory with Azure Monitor](https://learn.microsoft.com/azure/data-factory/monitor-data-factory)
+- [Alerting in Azure Data Factory](https://azure.microsoft.com/blog/create-alerts-to-proactively-monitor-your-data-factory-pipelines/)
 
 ## How to use the sample
 

single_tech_samples/fabric/README.md

@@ -1,7 +1,9 @@
 # Microsoft Fabric
 
-[Microsoft Fabric](https://learn.microsoft.com/en-us/fabric/get-started/microsoft-fabric-overview) is an all-in-one analytics solution for enterprises that covers everything from data movement to data science, Real-Time Analytics, and business intelligence. It offers a comprehensive suite of services, including data lake, data engineering, and data integration, all in one place.
+[Microsoft Fabric](https://learn.microsoft.com/fabric/get-started/microsoft-fabric-overview) is an all-in-one analytics solution for enterprises that covers everything from data movement to data science, Real-Time Analytics, and business intelligence. It offers a comprehensive suite of services, including data lake, data engineering, and data integration, all in one place.
 
 ## Samples
 
 - [Feature Engineering on Microsoft Fabric](./feature_engineering_on_fabric/README.md) - This sample demonstrates how to use Azure ML managed feature store and Microsoft Fabric to build a feature engineering system. It also shows how to track and monitor the data lineage of the features and the model training process using Microsoft Purview. The provided sample also encompasses data validation and exploratory data analysis (EDA) within Fabric notebooks.
+
+- [CI/CD - Microsoft Fabric](./fabric_ci_cd/README.md) - This sample demonstrates how to implement a CI/CD process for Microsoft Fabric using Azure DevOps and Fabric Deployment Pipelines.

single_tech_samples/fabric/fabric_ci_cd/.envtemplate

@@ -0,0 +1,30 @@
+# The Azure subscription Id.
+AZURE_SUBSCRIPTION_ID=""
+# The location where the Azure resources will be created.
+AZURE_LOCATION=""
+# The email address of the Fabric capacity admin. This should be from the same tenant where capacity is being created.
+CAPACITY_ADMIN_EMAIL=""
+# The name of the existing Fabric capacity. This name is required if you want to use an existing capacity instead of creating a new one.
+EXISTING_CAPACITY_NAME=""
+# The name of the Fabric project. This name is used for naming the Fabric resources.
+FABRIC_PROJECT_NAME=""
+# The name of the Fabric domain. It can be an existing domain.
+FABRIC_DOMAIN_NAME=""
+# The name of the Fabric subdomain. It can be an existing subdomain.
+FABRIC_SUBDOMAIN_NAME=""
+# The bearer token for calling the Fabric APIs.
+FABRIC_BEARER_TOKEN=""
+# The Azure DevOps organization name.
+ORGANIZATION_NAME=""
+# The Azure DevOps project name.
+PROJECT_NAME=""
+# The Azure DevOps repository name.
+REPOSITORY_NAME=""
+# Azure DevOps branch name. This branch should already exist in the repository.
+BRANCH_NAME=""
+# The directory used by Fabric to sync the workspace code. It can be "/" or any other sub-directory. If specifying a sub-directory, it must exist in the repository.
+DIRECTORY_NAME=""
+# UserPrincipalName (UPN) list of the workspace admins. These users will have admin access to the Fabric workspaces. The values are separated by space.
+WORKSPACE_ADMIN_UPNS=("user1@contoso.com" "user2@contoso.com" "...")
+# UserPrincipalName (UPN) list of the pipeline admins. These users will have admin access to the Fabric deployment pipeline. The values are separated by space.
+PIPELINE_ADMIN_UPNS=("user1@contoso.com" "user2@contoso.com" "...")