Skip to content

Conversation

@jklukas
Copy link
Contributor

@jklukas jklukas commented Jun 26, 2020

This is a schema-incompatible change. These tables are still being used only
for early stage testing of AET, so it is entirely reasonable to throw away
existing data and recreate them entirely.

Checklist for reviewer:

  • Commits should reference a bug or github issue, if relevant (if a bug is referenced, the pull request should include the bug number in the title)
  • Scan the PR and verify that no changes (particularly to .circleci/config.yml) will cause environment variables (particularly credentials) to be exposed in test logs
  • If the PR comes from a fork, trigger the integration CI test by pushing this revision as discussed in the README and review the report posted in the comments.

For glean changes:

  • Update include/glean/CHANGELOG.md

This is a schema-incompatible change. These tables are still being used only
for early stage testing of AET, so it is entirely reasonable to throw away
existing data and recreate them entirely.
@jklukas jklukas requested a review from whd June 26, 2020 19:32
@jklukas
Copy link
Contributor Author

jklukas commented Jun 26, 2020

@whd - Is it fairly straight-forward on your end to delete and recreate these tables in order to remove these fields?

@auto-assign auto-assign bot requested a review from acmiyaguchi June 26, 2020 19:33
@jklukas jklukas removed the request for review from acmiyaguchi June 26, 2020 19:34
@dataops-ci-bot
Copy link

Integration report for "Remove ecosystem_device_id fields"

Report for upstream
Report for latest commit

2cfd30a-f883294.diff

No content detected.

bq_schema_2cfd30a-f883294.diff

Click to expand!
diff integration/2cfd30a/firefox-accounts.account-ecosystem.1.bq integration/f883294/firefox-accounts.account-ecosystem.1.bq
8,12d7
<     "mode": "NULLABLE",
<     "name": "ecosystem_device_id",
<     "type": "STRING"
<   },
<   {
diff integration/2cfd30a/telemetry.account-ecosystem.4.bq integration/f883294/telemetry.account-ecosystem.4.bq
72,76d71
<         "mode": "NULLABLE",
<         "name": "ecosystem_device_id",
<         "type": "STRING"
<       },
<       {

Copy link
Member

@whd whd left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@whd - Is it fairly straight-forward on your end to delete and recreate these tables in order to remove these fields?

It should be. Since we're recreating the tables, deleting the existing tables before the next schemas deploy that includes this change will likely suffice (i.e. no need to run bigquery-etl's list_broken_views script).

However, in case my understanding is incorrect, let's wait until Monday to merge this so that I don't have to make this change on Sunday when the next schemas deploy is scheduled.

Side note: this is the kind of change that I want CI to fail by default, but we'd obviously need some kind of override for cases like this (maybe in the commit message?).

@whd whd merged commit c47313b into master Jun 29, 2020
@whd whd deleted the aet-remove-deviceId branch June 29, 2020 18:40
dataops-pipeline-schemas added a commit that referenced this pull request Jun 29, 2020
c47313b	2020-06-29 18:40:32 +0000	Remove ecosystem_device_id fields (#567)
@whd
Copy link
Member

whd commented Jun 29, 2020

This has been deployed.

acmiyaguchi pushed a commit that referenced this pull request Jul 22, 2020
361c16d	2020-07-22 20:48:09 +0000	Add xfocsp-error-report schema (#581)
8b05ee6	2020-07-21 14:27:14 -0700	Add regrets-reporter schemas (#556)
e724d79	2020-07-17 11:02:54 -0400	Bug 1652842 - Add a blocklist field so the environment schema works with pre- and post-bug-1647225 data (#576)
7400dc8	2020-07-15 12:46:09 +0200	Bug 1652834 - Add gfx.EmbeddedInFirefoxReality field to environment.
490cfe0	2020-07-14 16:39:13 -0400	AET update: 64-char user_ids and relaxing validation
8f22589	2020-07-14 11:56:08 -0400	Specify yaml loader in extract_crash_annotation_fields script (#573)
f0f50c4	2020-07-14 11:55:40 -0400	Bug 1652777 - Add experimental features to crash ping schema (#574)
97bac7a	2020-06-30 15:16:36 -0700	Add 'openglCompositing' and 'wrCompositor' to schema (#569)
c47313b	2020-06-29 18:40:32 +0000	Remove ecosystem_device_id fields (#567)
2cfd30a	2020-06-26 15:09:52 -0400	Make ecosystem_device_id an optional field
3f11d8b	2020-06-23 18:03:10 +0200	Update sync ping schema to make all migration fields transpiled into BQ
e678002	2020-06-22 14:12:25 -0400	add history_average_days_per_month to schema
be61e13	2020-06-22 14:12:25 -0400	fix schema to contain all fields
6ecd201	2020-06-22 14:12:25 -0400	add schema for normandy-login-study bug 1643383
2e62aac	2020-06-22 12:53:32 -0400	Pin Centos image
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants