Brsherwi/829

e2e_samples/parking_sensors/README.md

@@ -61,7 +61,7 @@ The sample demonstrate how DevOps principles can be applied end to end Data Pipe
 
 ## Solution Overview
 
-The solution pulls near realtime [Melbourne Parking Sensor data](https://www.melbourne.vic.gov.au/about-council/governance-transparency/open-data/Pages/on-street-parking-data.aspx) from a publicly available REST api endpoint and saves this to [Azure Data Lake Gen2](https://docs.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-introduction). It then validates, cleanses, and transforms the data to a known schema using [Azure Databricks](https://azure.microsoft.com/en-us/products/databricks/). A second Azure Databricks job then transforms these into a [Star Schema](https://en.wikipedia.org/wiki/Star_schema) which are then loaded into [Azure Synapse Analytics (formerly SQLDW)](https://azure.microsoft.com/en-us/products/synapse-analytics/) using [Polybase](https://docs.microsoft.com/en-us/sql/relational-databases/polybase/polybase-guide?view=sql-server-ver15). The entire pipeline is orchestrated with [Azure Data Factory](https://azure.microsoft.com/en-us/products/data-factory/).
+The solution pulls fictional near realtime parking sensor data from a data generator that was installed during the deploy and saves this to [Azure Data Lake Gen2](https://docs.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-introduction). It then validates, cleanses, and transforms the data to a known schema using [Azure Databricks](https://azure.microsoft.com/en-us/products/databricks/). A second Azure Databricks job then transforms these into a [Star Schema](https://en.wikipedia.org/wiki/Star_schema) which are then loaded into [Azure Synapse Analytics (formerly SQLDW)](https://azure.microsoft.com/en-us/products/synapse-analytics/) using [Polybase](https://docs.microsoft.com/en-us/sql/relational-databases/polybase/polybase-guide?view=sql-server-ver15). The entire pipeline is orchestrated with [Azure Data Factory](https://azure.microsoft.com/en-us/products/data-factory/).
 
 ### Architecture
 
@@ -339,7 +339,7 @@ Set up the environment variables as specified, fork the GitHub repository, and l
 
 4. **Trigger an initial Release**
 
-   - In the **DEV** Data Factory portal, navigate to Pipelines and open the "P_Ingest_MelbParkingData" pipeline.
+   - In the **DEV** Data Factory portal, navigate to Pipelines and open the "P_Ingest_ParkingData" pipeline.
    - In the top left corner, open the git drop down and create a Dev branch by clicking in "New Branch".
    - Once the Dev branch is created, select the branch from the drop-down list and make a change in the Description fields from one of the pipeline tasks.
    - Save the pipeline.
@@ -352,7 +352,7 @@ Set up the environment variables as specified, fork the GitHub repository, and l
       - You may need to authorize the Pipelines initially to use the Service Connection and deploy the target environments for the first time.
       ![Release Pipeline](docs/images/ReleasePipeline.png "Release Pipelines")
    - **Optional**. Trigger the Data Factory Pipelines per environment.
-      - In the Data Factory portal of each environment, navigate to "Author", then select the `P_Ingest_MelbParkingData`.
+      - In the Data Factory portal of each environment, navigate to "Author", then select the `P_Ingest_ParkingData`.
       - Select "Trigger > Trigger Now".
       - To monitor the run, go to "Monitor > Pipeline runs".
       ![Data Factory Run](docs/images/ADFRun.png "Data Factory Run]")

e2e_samples/parking_sensors/adf/dataset/Ds_AdlsGen2_MelbParkingData.json → e2e_samples/parking_sensors/adf/dataset/Ds_AdlsGen2_ParkingData.json

@@ -1,5 +1,5 @@
 {
-	"name": "Ds_AdlsGen2_MelbParkingData",
+	"name": "Ds_AdlsGen2_ParkingData",
 	"properties": {
 		"linkedServiceName": {
 			"referenceName": "Ls_AdlsGen2_01",

e2e_samples/parking_sensors/adf/dataset/Ds_Http_Parking_Bay.json → e2e_samples/parking_sensors/adf/dataset/Ds_Http_Parking_Locations.json

@@ -1,8 +1,8 @@
 {
-    "name": "Ds_Http_Parking_Bay",
+    "name": "Ds_Http_Parking_Locations",
     "properties": {
         "linkedServiceName": {
-            "referenceName": "Ls_Http_Parking_Bay_01",
+            "referenceName": "Ls_Http_DataSimulator",
             "type": "LinkedServiceReference"
         },
         "annotations": [],
@@ -11,7 +11,7 @@
             "location": {
                 "type": "HttpServerLocation",
                 "relativeUrl": {
-                    "value": "on-street-parking-bays/exports/json",
+                    "value": "/locations",
                     "type": "Expression"
                 }
             }

e2e_samples/parking_sensors/adf/dataset/Ds_Http_Parking_Bay_Sensors.json → e2e_samples/parking_sensors/adf/dataset/Ds_Http_Parking_Sensors.json

@@ -1,16 +1,16 @@
 {
-    "name": "Ds_Http_Parking_Bay_Sensors",
+    "name": "Ds_Http_Parking_Sensors",
     "properties": {
         "linkedServiceName": {
-            "referenceName": "Ls_Http_Parking_Bay_01",
+            "referenceName": "Ls_Http_DataSimulator",
             "type": "LinkedServiceReference"
         },
         "annotations": [],
         "type": "Json",
         "typeProperties": {
             "location": {
                 "type": "HttpServerLocation",
-                "relativeUrl": "on-street-parking-bay-sensors/exports/json"
+                "relativeUrl": "/sensors"
             }
         },
         "schema": {}

e2e_samples/parking_sensors/adf/dataset/Ds_REST_MelbParkingData.json → e2e_samples/parking_sensors/adf/dataset/Ds_REST_ParkingData.json

@@ -1,8 +1,8 @@
 {
-	"name": "Ds_REST_MelbParkingData",
+	"name": "Ds_REST_ParkingData",
 	"properties": {
 		"linkedServiceName": {
-			"referenceName": "Ls_Rest_MelParkSensors_01",
+			"referenceName": "Ls_Rest_ParkSensors_01",
 			"type": "LinkedServiceReference"
 		},
 		"parameters": {

e2e_samples/parking_sensors/adf/linkedService/Ls_Http_Parking_Bay_01.json → e2e_samples/parking_sensors/adf/linkedService/Ls_Http_DataSimulator.json.template

@@ -1,10 +1,10 @@
 {
-    "name": "Ls_Http_Parking_Bay_01",
+    "name": "Ls_Http_DataSimulator",
     "properties": {
         "annotations": [],
         "type": "HttpServer",
         "typeProperties": {
-            "url": "https://data.melbourne.vic.gov.au/api/explore/v2.1/catalog/datasets/",
+            "url": "https://<apiBaseUrl>",
             "enableServerCertificateValidation": true,
             "authenticationType": "Anonymous"
         }

e2e_samples/parking_sensors/adf/linkedService/Ls_Rest_MelParkSensors_01.json → e2e_samples/parking_sensors/adf/linkedService/Ls_Rest_ParkSensors_01.json.template

@@ -1,11 +1,11 @@
 {
-	"name": "Ls_Rest_MelParkSensors_01",
+	"name": "Ls_Rest_ParkSensors_01",
 	"type": "Microsoft.DataFactory/factories/linkedservices",
 	"properties": {
 		"annotations": [],
 		"type": "RestService",
 		"typeProperties": {
-			"url": "https://data.melbourne.vic.gov.au/resource/",
+			"url": "https://<apiBaseUrl>",
 			"enableServerCertificateValidation": true,
 			"authenticationType": "Anonymous"
 		}