-
Notifications
You must be signed in to change notification settings - Fork 18
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
added examples and fixed column-parsing bug
- Loading branch information
1 parent
16913b9
commit 0f65ab5
Showing
15 changed files
with
313 additions
and
56 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
76 changes: 76 additions & 0 deletions
76
examples/data_product_pipeline/Finance/Finwire Archive/data_product.json
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,76 @@ | ||
| { | ||
| "template_id": "data_product", | ||
| "project_id": "data-mesh-344315", | ||
| "region": "us-central1", | ||
| "fields": [ | ||
| { | ||
| "field_id": "data_domain", | ||
| "field_value": "Finance" | ||
| }, | ||
| { | ||
| "field_id": "data_subdomain", | ||
| "field_value": "Financial_Insights" | ||
| }, | ||
| { | ||
| "field_id": "data_product_name", | ||
| "field_value": "Finwire Archive" | ||
| }, | ||
| { | ||
| "field_id": "data_product_description", | ||
| "field_value": "This dataset contains financial information about companies and securities obtained from a financial newswire service that has been archived over an extended period of time." | ||
| }, | ||
| { | ||
| "field_id": "data_confidentiality", | ||
| "field_value": "Public" | ||
| }, | ||
| { | ||
| "field_id": "business_criticality", | ||
| "field_value": "Medium" | ||
| }, | ||
| { | ||
| "field_id": "business_owner", | ||
| "field_value": "John Williams" | ||
| }, | ||
| { | ||
| "field_id": "technical_owner", | ||
| "field_value": "Emily Doe" | ||
| }, | ||
| { | ||
| "field_id": "number_data_resources", | ||
| "field_value": "409" | ||
| }, | ||
| { | ||
| "field_id": "storage_location", | ||
| "field_value": "us-central1" | ||
| }, | ||
| { | ||
| "field_id": "data_retention_period", | ||
| "field_value": "7_years" | ||
| }, | ||
| { | ||
| "field_id": "data_latency_slo", | ||
| "field_value": "quarterly" | ||
| }, | ||
| { | ||
| "field_id": "documentation_link", | ||
| "field_value": "https://www.tpc.org/tpc_documents_current_versions/pdf/tpc-di_v1.1.0.pdf " | ||
| }, | ||
| { | ||
| "field_id": "access_request_link", | ||
| "field_value": "go/sphinx/finwire-archive" | ||
| }, | ||
| { | ||
| "field_id": "data_product_status", | ||
| "field_value": "DRAFT" | ||
| }, | ||
| { | ||
| "field_id": "last_modified_date", | ||
| "field_value": "2022-08-28" | ||
| } | ||
| ], | ||
| "included_uris": "bigquery/project/data-mesh-343422/dataset/finance", | ||
| "excluded_uris": "", | ||
| "refresh_mode": "ON_DEMAND", | ||
| "tag_history": true, | ||
| "tag_stream": false | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
17 changes: 17 additions & 0 deletions
17
examples/data_product_pipeline/Finance/Finwire Archive/data_standardization.json
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,17 @@ | ||
| { | ||
| "template_id": "data_standardization", | ||
| "project_id": "data-mesh-344315", | ||
| "region": "us-central1", | ||
| "fields": [ | ||
| { | ||
| "field_id": "degree", | ||
| "query_expression": "with finwire_unmatches as (select count(*) umatched_counts from $project.$dataset.$table f1 where $column not in (select in_id from reference.Industry)), finwire_total as (select count(*) as total_counts from $project.$dataset.$table) select round(safe_divide((total_counts - umatched_counts), total_counts) * 100) as degree from finwire_unmatches, finwire_total" | ||
| } | ||
| ], | ||
| "included_uris": "bigquery/project/data-mesh-343422/dataset/finance/FINWIRE*_CMP/industryID", | ||
| "excluded_uris": "", | ||
| "refresh_mode": "ON_DEMAND", | ||
| "tag_history": true, | ||
| "tag_stream": false | ||
| } | ||
|
|
76 changes: 76 additions & 0 deletions
76
examples/data_product_pipeline/Finance/Trade Transactions/data_product.json
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,76 @@ | ||
| { | ||
| "template_id": "data_product", | ||
| "project_id": "data-mesh-344315", | ||
| "region": "us-central1", | ||
| "fields": [ | ||
| { | ||
| "field_id": "data_domain", | ||
| "field_value": "Finance" | ||
| }, | ||
| { | ||
| "field_id": "data_subdomain", | ||
| "field_value": "Trade_and_Banking" | ||
| }, | ||
| { | ||
| "field_id": "data_product_name", | ||
| "field_value": "Trade Transactions" | ||
| }, | ||
| { | ||
| "field_id": "data_product_description", | ||
| "field_value": "Data on customers, accounts, brokers, securities, trade details, account balances, and market information from the system of record." | ||
| }, | ||
| { | ||
| "field_id": "data_confidentiality", | ||
| "field_value": "Confidential" | ||
| }, | ||
| { | ||
| "field_id": "business_criticality", | ||
| "field_value": "High" | ||
| }, | ||
| { | ||
| "field_id": "business_owner", | ||
| "field_value": "Sunil Kumar" | ||
| }, | ||
| { | ||
| "field_id": "technical_owner", | ||
| "field_value": "Sudipta Nelson" | ||
| }, | ||
| { | ||
| "field_id": "number_data_resources", | ||
| "field_value": "13" | ||
| }, | ||
| { | ||
| "field_id": "storage_location", | ||
| "field_value": "us-central1" | ||
| }, | ||
| { | ||
| "field_id": "data_retention_period", | ||
| "field_value": "2_years" | ||
| }, | ||
| { | ||
| "field_id": "data_latency_slo", | ||
| "field_value": "daily" | ||
| }, | ||
| { | ||
| "field_id": "documentation_link", | ||
| "field_value": "https://www.tpc.org/tpc_documents_current_versions/pdf/tpc-di_v1.1.0.pdf " | ||
| }, | ||
| { | ||
| "field_id": "access_request_link", | ||
| "field_value": "go/sphinx/trade-tx" | ||
| }, | ||
| { | ||
| "field_id": "data_product_status", | ||
| "field_value": "DRAFT" | ||
| }, | ||
| { | ||
| "field_id": "last_modified_date", | ||
| "field_value": "2022-08-30" | ||
| } | ||
| ], | ||
| "included_uris": "bigquery/project/data-mesh-343422/dataset/oltp", | ||
| "excluded_uris": "", | ||
| "refresh_mode": "ON_DEMAND", | ||
| "tag_history": true, | ||
| "tag_stream": false | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
17 changes: 17 additions & 0 deletions
17
examples/data_product_pipeline/Finance/Trade Transactions/data_standardization.json
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,17 @@ | ||
| { | ||
| "template_id": "data_standardization", | ||
| "project_id": "data-mesh-344315", | ||
| "region": "us-central1", | ||
| "fields": [ | ||
| { | ||
| "field_id": "degree", | ||
| "query_expression": "with status_unmatches as (select count(*) umatched_counts from $project.$dataset.$table where $column not in (select st_id from $project.reference.StatusType)), records_total as (select count(*) as total_counts from $project.$dataset.$table) select round(safe_divide((total_counts - umatched_counts), total_counts) * 100) as degree from status_unmatches, records_total" | ||
| } | ||
| ], | ||
| "included_uris": "bigquery/project/data-mesh-343422/dataset/oltp/Account/ca_st_id", | ||
| "excluded_uris": "", | ||
| "refresh_mode": "ON_DEMAND", | ||
| "tag_history": true, | ||
| "tag_stream": false | ||
| } | ||
|
|
Oops, something went wrong.