Bug 1697602 Remove AET schemas #664

jklukas · 2021-03-15T19:48:39Z

See https://bugzilla.mozilla.org/show_bug.cgi?id=1697602

This will probably be in draft for a while. Merging this will require
coordination with Data SRE to remove the underlying tables.

Checklist for reviewer:

Commits should reference a bug or github issue, if relevant (if a bug is referenced, the pull request should include the bug number in the title)
If adding a new field, the field should have a description (see Bug 1652842 - Add a blocklist field so the environment schema works with pre- and post-bug-1647225 data #576 for an example)
Scan the PR and verify that no changes (particularly to .circleci/config.yml) will cause environment variables (particularly credentials) to be exposed in test logs
If the PR comes from a fork, trigger the integration CI test by pushing this revision as discussed in the README and review the report posted in the comments.

For glean changes:

Update templates/include/glean/CHANGELOG.md

See https://bugzilla.mozilla.org/show_bug.cgi?id=1697602 This will probably be in draft for a while. Merging this will require coordination with Data SRE to remove the underlying tables.

whd · 2021-03-17T20:01:09Z

Looks like some tests need to be updated/removed as well.

dataops-ci-bot · 2021-03-18T20:12:29Z

Integration report for "Update test"

`bq_schema_5f213002-7172ea19.diff`

Click to expand!

diff --new-file --exclude '*.txt' /app/integration/5f213002/firefox-accounts.account-ecosystem.1.bq /app/integration/7172ea19/firefox-accounts.account-ecosystem.1.bq
1,42d0
< [
<   {
<     "description": "Country name as extracted in the FxA pipeline; contains logical values like 'United States' and 'India' rather than 2-letter country codes",
<     "mode": "NULLABLE",
<     "name": "country",
<     "type": "STRING"
<   },
<   {
<     "mode": "NULLABLE",
<     "name": "ecosystem_client_id",
<     "type": "STRING"
<   },
<   {
<     "description": "Account Ecosystem Telemetry user identifier; this value is not present in the original payload sent by clients, but is instead inserted by the pipeline after decrypting and removing ecosystem_anon_id",
<     "mode": "NULLABLE",
<     "name": "ecosystem_user_id",
<     "type": "STRING"
<   },
<   {
<     "description": "Value of the 'event' field in the source LogEntry payload",
<     "mode": "NULLABLE",
<     "name": "event",
<     "type": "STRING"
<   },
<   {
<     "description": "Value of the 'clientId' field in the source LogEntry payload, indicating the OAuth client associated with the event",
<     "mode": "NULLABLE",
<     "name": "oauth_client_id",
<     "type": "STRING"
<   },
<   {
<     "mode": "REPEATED",
<     "name": "previous_ecosystem_user_ids",
<     "type": "STRING"
<   },
<   {
<     "description": "Region name as extracted in the FxA pipeline; contains logical values like 'Texas' rather than abbreviations like 'TX'",
<     "mode": "NULLABLE",
<     "name": "region",
<     "type": "STRING"
<   }
< ]
diff --new-file --exclude '*.txt' /app/integration/5f213002/metadata.pioneer-decoded.1.bq /app/integration/7172ea19/metadata.pioneer-decoded.1.bq
228,233d227
<     "description": "Bug 1677567 - Rally id reported by the core addon",
<     "mode": "NULLABLE",
<     "name": "rally_id",
<     "type": "STRING"
<   },
<   {
diff --new-file --exclude '*.txt' /app/integration/5f213002/metadata.pioneer-error.1.bq /app/integration/7172ea19/metadata.pioneer-error.1.bq
137,142d136
<     "description": "Bug 1677567 - Rally id reported by the core addon",
<     "mode": "NULLABLE",
<     "name": "rally_id",
<     "type": "STRING"
<   },
<   {
diff --new-file --exclude '*.txt' /app/integration/5f213002/metadata.pioneer-ingestion.1.bq /app/integration/7172ea19/metadata.pioneer-ingestion.1.bq
181,186d180
<     "description": "Bug 1677567 - Rally id reported by the core addon",
<     "mode": "NULLABLE",
<     "name": "rally_id",
<     "type": "STRING"
<   },
<   {
diff --new-file --exclude '*.txt' /app/integration/5f213002/telemetry.account-ecosystem.4.bq /app/integration/7172ea19/telemetry.account-ecosystem.4.bq
1,284d0
< [
<   {
<     "fields": [
<       {
<         "mode": "NULLABLE",
<         "name": "architecture",
<         "type": "STRING"
<       },
<       {
<         "mode": "NULLABLE",
<         "name": "build_id",
<         "type": "STRING"
<       },
<       {
<         "mode": "NULLABLE",
<         "name": "channel",
<         "type": "STRING"
<       },
<       {
<         "mode": "NULLABLE",
<         "name": "display_version",
<         "type": "STRING"
<       },
<       {
<         "mode": "NULLABLE",
<         "name": "name",
<         "type": "STRING"
<       },
<       {
<         "mode": "NULLABLE",
<         "name": "platform_version",
<         "type": "STRING"
<       },
<       {
<         "mode": "NULLABLE",
<         "name": "vendor",
<         "type": "STRING"
<       },
<       {
<         "mode": "NULLABLE",
<         "name": "version",
<         "type": "STRING"
<       },
<       {
<         "mode": "NULLABLE",
<         "name": "xpcom_abi",
<         "type": "STRING"
<       }
<     ],
<     "mode": "NULLABLE",
<     "name": "application",
<     "type": "RECORD"
<   },
<   {
<     "mode": "NULLABLE",
<     "name": "creation_date",
<     "type": "STRING"
<   },
<   {
<     "fields": [
<       {
<         "fields": [
<           {
<             "mode": "NULLABLE",
<             "name": "creation_date",
<             "type": "FLOAT64"
<           },
<           {
<             "mode": "NULLABLE",
<             "name": "first_use_date",
<             "type": "FLOAT64"
<           }
<         ],
<         "mode": "NULLABLE",
<         "name": "profile",
<         "type": "RECORD"
<       },
<       {
<         "fields": [
<           {
<             "mode": "NULLABLE",
<             "name": "locale",
<             "type": "STRING"
<           }
<         ],
<         "mode": "NULLABLE",
<         "name": "settings",
<         "type": "RECORD"
<       },
<       {
<         "fields": [
<           {
<             "fields": [
<               {
<                 "mode": "NULLABLE",
<                 "name": "speed_m_hz",
<                 "type": "FLOAT64"
<               }
<             ],
<             "mode": "NULLABLE",
<             "name": "cpu",
<             "type": "RECORD"
<           },
<           {
<             "mode": "NULLABLE",
<             "name": "memory_mb",
<             "type": "FLOAT64"
<           },
<           {
<             "fields": [
<               {
<                 "mode": "NULLABLE",
<                 "name": "locale",
<                 "type": "STRING"
<               },
<               {
<                 "mode": "NULLABLE",
<                 "name": "name",
<                 "type": "STRING"
<               },
<               {
<                 "mode": "NULLABLE",
<                 "name": "version",
<                 "type": "STRING"
<               }
<             ],
<             "mode": "NULLABLE",
<             "name": "os",
<             "type": "RECORD"
<           }
<         ],
<         "mode": "NULLABLE",
<         "name": "system",
<         "type": "RECORD"
<       }
<     ],
<     "mode": "NULLABLE",
<     "name": "environment",
<     "type": "RECORD"
<   },
<   {
<     "mode": "NULLABLE",
<     "name": "id",
<     "type": "STRING"
<   },
<   {
<     "fields": [
<       {
<         "mode": "NULLABLE",
<         "name": "duration",
<         "type": "INT64"
<       },
<       {
<         "mode": "NULLABLE",
<         "name": "ecosystem_client_id",
<         "type": "STRING"
<       },
<       {
<         "description": "Account Ecosystem Telemetry user identifier; this value is not present in the original payload sent by clients, but is instead inserted by the pipeline after decrypting and removing ecosystemAnonId",
<         "mode": "NULLABLE",
<         "name": "ecosystem_user_id",
<         "type": "STRING"
<       },
<       {
<         "fields": [
<           {
<             "fields": [
<               {
<                 "mode": "NULLABLE",
<                 "name": "key",
<                 "type": "STRING"
<               },
<               {
<                 "fields": [
<                   {
<                     "mode": "NULLABLE",
<                     "name": "key",
<                     "type": "STRING"
<                   },
<                   {
<                     "fields": [
<                       {
<                         "mode": "NULLABLE",
<                         "name": "bucket_count",
<                         "type": "INT64"
<                       },
<                       {
<                         "mode": "NULLABLE",
<                         "name": "histogram_type",
<                         "type": "INT64"
<                       },
<                       {
<                         "mode": "NULLABLE",
<                         "name": "log_sum",
<                         "type": "FLOAT64"
<                       },
<                       {
<                         "mode": "NULLABLE",
<                         "name": "log_sum_squares",
<                         "type": "FLOAT64"
<                       },
<                       {
<                         "mode": "REPEATED",
<                         "name": "range",
<                         "type": "INT64"
<                       },
<                       {
<                         "mode": "NULLABLE",
<                         "name": "sum",
<                         "type": "INT64"
<                       },
<                       {
<                         "mode": "NULLABLE",
<                         "name": "sum_squares_hi",
<                         "type": "INT64"
<                       },
<                       {
<                         "mode": "NULLABLE",
<                         "name": "sum_squares_lo",
<                         "type": "INT64"
<                       },
<                       {
<                         "fields": [
<                           {
<                             "mode": "NULLABLE",
<                             "name": "key",
<                             "type": "STRING"
<                           },
<                           {
<                             "mode": "NULLABLE",
<                             "name": "value",
<                             "type": "INT64"
<                           }
<                         ],
<                         "mode": "REPEATED",
<                         "name": "values",
<                         "type": "RECORD"
<                       }
<                     ],
<                     "mode": "NULLABLE",
<                     "name": "value",
<                     "type": "RECORD"
<                   }
<                 ],
<                 "mode": "REPEATED",
<                 "name": "value",
<                 "type": "RECORD"
<               }
<             ],
<             "mode": "REPEATED",
<             "name": "parent",
<             "type": "RECORD"
<           }
<         ],
<         "mode": "NULLABLE",
<         "name": "keyed_histograms",
<         "type": "RECORD"
<       },
<       {
<         "mode": "REPEATED",
<         "name": "previous_ecosystem_user_ids",
<         "type": "STRING"
<       },
<       {
<         "mode": "NULLABLE",
<         "name": "reason",
<         "type": "STRING"
<       }
<     ],
<     "mode": "NULLABLE",
<     "name": "payload",
<     "type": "RECORD"
<   },
<   {
<     "mode": "NULLABLE",
<     "name": "type",
<     "type": "STRING"
<   },
<   {
<     "mode": "NULLABLE",
<     "name": "version",
<     "type": "FLOAT64"
<   }
< ]

`compact_schema_5f213002-7172ea19.diff`

Click to expand!

diff --new-file --exclude '*.bq' /app/integration/5f213002/firefox-accounts.account-ecosystem.1.txt /app/integration/7172ea19/firefox-accounts.account-ecosystem.1.txt
1,7d0
< root.country STRING
< root.ecosystem_client_id STRING
< root.ecosystem_user_id STRING
< root.event STRING
< root.oauth_client_id STRING
< root.previous_ecosystem_user_ids.[] STRING
< root.region STRING
\ No newline at end of file
diff --new-file --exclude '*.bq' /app/integration/5f213002/metadata.pioneer-decoded.1.txt /app/integration/7172ea19/metadata.pioneer-decoded.1.txt
33d32
< root.rally_id STRING
diff --new-file --exclude '*.bq' /app/integration/5f213002/metadata.pioneer-error.1.txt /app/integration/7172ea19/metadata.pioneer-error.1.txt
26d25
< root.rally_id STRING
diff --new-file --exclude '*.bq' /app/integration/5f213002/metadata.pioneer-ingestion.1.txt /app/integration/7172ea19/metadata.pioneer-ingestion.1.txt
25d24
< root.rally_id STRING
diff --new-file --exclude '*.bq' /app/integration/5f213002/telemetry.account-ecosystem.4.txt /app/integration/7172ea19/telemetry.account-ecosystem.4.txt
1,38d0
< root.application.architecture STRING
< root.application.build_id STRING
< root.application.channel STRING
< root.application.display_version STRING
< root.application.name STRING
< root.application.platform_version STRING
< root.application.vendor STRING
< root.application.version STRING
< root.application.xpcom_abi STRING
< root.creation_date STRING
< root.environment.profile.creation_date FLOAT64
< root.environment.profile.first_use_date FLOAT64
< root.environment.settings.locale STRING
< root.environment.system.cpu.speed_m_hz FLOAT64
< root.environment.system.memory_mb FLOAT64
< root.environment.system.os.locale STRING
< root.environment.system.os.name STRING
< root.environment.system.os.version STRING
< root.id STRING
< root.payload.duration INT64
< root.payload.ecosystem_client_id STRING
< root.payload.ecosystem_user_id STRING
< root.payload.keyed_histograms.parent.[].key STRING
< root.payload.keyed_histograms.parent.[].value.[].key STRING
< root.payload.keyed_histograms.parent.[].value.[].value.bucket_count INT64
< root.payload.keyed_histograms.parent.[].value.[].value.histogram_type INT64
< root.payload.keyed_histograms.parent.[].value.[].value.log_sum FLOAT64
< root.payload.keyed_histograms.parent.[].value.[].value.log_sum_squares FLOAT64
< root.payload.keyed_histograms.parent.[].value.[].value.range.[] INT64
< root.payload.keyed_histograms.parent.[].value.[].value.sum INT64
< root.payload.keyed_histograms.parent.[].value.[].value.sum_squares_hi INT64
< root.payload.keyed_histograms.parent.[].value.[].value.sum_squares_lo INT64
< root.payload.keyed_histograms.parent.[].value.[].value.values.[].key STRING
< root.payload.keyed_histograms.parent.[].value.[].value.values.[].value INT64
< root.payload.previous_ecosystem_user_ids.[] STRING
< root.payload.reason STRING
< root.type STRING
< root.version FLOAT64
\ No newline at end of file

c2e18f4 2021-03-23 18:04:10 +0000 Bug 1697602 Remove AET schemas (#664) 5f21300 2021-03-17 19:04:16 +0000 Bug 1677567 - Add rallyId to pioneer metadata schemas (#665) 731721a 2021-03-09 10:03:38 -0800 Bug 1696074 - Add schema for uninstall-deletion ping in pioneer-core (#663) 793fc04 2021-03-08 09:55:50 -0500 Bug 1695795: Remove unused metric types from the Glean schema (#662) 04043f1 2021-03-02 08:30:04 -0500 Fix #660: Add the rate metric type (#661) 0a24af0 2021-02-24 10:43:58 -0500 [Bug 1694254] Add environment.settings.attribution.dltoken field c2dae92 2021-02-22 18:30:19 +0000 Bug 1693305 - Clone citp-news-disinfo study (#658) f3a1464 2021-02-17 16:05:26 -0600 Bug 1688698 - Add 'source' to TopSites schemas in Contextual Services (#657) 5904f8c 2021-02-12 16:29:41 -0600 Bug 1688698 - Add initial schemas for Contextual Services (#656)

c2e18f4 2021-03-23 18:04:10 +0000 Bug 1697602 Remove AET schemas (#664)

* Bug 1697602 Remove AET schemas associated See https://bugzilla.mozilla.org/show_bug.cgi?id=1697602 This will probably be in draft for a while. Merging this will require coordination with Data SRE to remove the underlying tables. * Update test

Bug 1697602 Remove AET schemas associated

c4869fb

See https://bugzilla.mozilla.org/show_bug.cgi?id=1697602 This will probably be in draft for a while. Merging this will require coordination with Data SRE to remove the underlying tables.

jklukas mentioned this pull request Mar 15, 2021

Bug 1697602 Remove configs for account-ecosystem pings mozilla/mozilla-schema-generator#185

Merged

Update test

7172ea1

jklukas changed the title ~~Bug 1697602 Remove AET schemas associated~~ Bug 1697602 Remove AET schemas Mar 18, 2021

whd self-requested a review March 23, 2021 00:11

whd marked this pull request as ready for review March 23, 2021 00:12

auto-assign bot requested a review from acmiyaguchi March 23, 2021 00:12

whd approved these changes Mar 23, 2021

View reviewed changes

acmiyaguchi approved these changes Mar 23, 2021

View reviewed changes

whd merged commit c2e18f4 into master Mar 23, 2021

whd deleted the remove-aet branch March 23, 2021 18:04

dataops-pipeline-schemas added a commit that referenced this pull request Mar 23, 2021

Auto-push from schema generation [ci skip]

05c773e

c2e18f4 2021-03-23 18:04:10 +0000 Bug 1697602 Remove AET schemas (#664)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Bug 1697602 Remove AET schemas #664

Bug 1697602 Remove AET schemas #664

Uh oh!

jklukas commented Mar 15, 2021 •

edited

Loading

Uh oh!

whd commented Mar 17, 2021

Uh oh!

dataops-ci-bot commented Mar 18, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Bug 1697602 Remove AET schemas #664

Bug 1697602 Remove AET schemas #664

Uh oh!

Conversation

jklukas commented Mar 15, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

whd commented Mar 17, 2021

Uh oh!

dataops-ci-bot commented Mar 18, 2021

Integration report for "Update test"

bq_schema_5f213002-7172ea19.diff

compact_schema_5f213002-7172ea19.diff

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

jklukas commented Mar 15, 2021 •

edited

Loading

`bq_schema_5f213002-7172ea19.diff`

`compact_schema_5f213002-7172ea19.diff`