Skip to content

Conversation

@jklukas
Copy link
Contributor

@jklukas jklukas commented Jul 22, 2020

Closes #570

This pulls in the commit from that PR and duplicates it in a new namespace.
We will be rerouting these pings in the pipeline so that they can have
stricter access controls.

There is a test case added here so that CI will fail if these two schemas
get out of sync.

Checklist for reviewer:

  • Commits should reference a bug or github issue, if relevant (if a bug is referenced, the pull request should include the bug number in the title)
  • Scan the PR and verify that no changes (particularly to .circleci/config.yml) will cause environment variables (particularly credentials) to be exposed in test logs
  • If the PR comes from a fork, trigger the integration CI test by pushing this revision as discussed in the README and review the report posted in the comments.

For glean changes:

  • Update include/glean/CHANGELOG.md

artines1 and others added 2 commits July 10, 2020 14:07
Closes #570

This pulls in the commit from that PR and duplicates it in a new namespace.
We will be rerouting these pings in the pipeline so that they can have
stricter access controls.

There is a test case added here so that CI will fail if these two schemas
get out of sync.
@auto-assign auto-assign bot requested a review from wlach July 22, 2020 18:51
@jklukas jklukas requested review from acmiyaguchi and whd and removed request for wlach July 22, 2020 18:51
@jklukas jklukas changed the title Xfocsp Add xfocsp-error-report schema Jul 22, 2020
@dataops-ci-bot
Copy link

Integration report for "Bug 1654558 Add xfocsp-error-report schema"

Report for upstream
Report for latest commit

8b05ee6-be1e3f8.diff

No content detected.

bq_schema_8b05ee6-be1e3f8.diff

Click to expand!
diff integration/8b05ee6/firefox-accounts.account-ecosystem.1.bq integration/be1e3f8/firefox-accounts.account-ecosystem.1.bq
3c3
<     "mode": "NULLABLE",
---
>     "mode": "REQUIRED",
Only in integration/8b05ee6: regrets-reporter.regrets-reporter-update.1.bq
diff integration/8b05ee6/telemetry.crash.4.bq integration/be1e3f8/telemetry.crash.4.bq
1026,1031d1025
<                 "description": "Whether or not Firefox desktop is embedded by Firefox Reality. This is Windows only.",
<                 "mode": "NULLABLE",
<                 "name": "embedded_in_firefox_reality",
<                 "type": "BOOL"
<               },
<               {
1076d1069
<                         "description": "Indicates whether the d3d11 compositor was blocklisted due to driver bugs. (This field was renamed to blocklisted in bug 1647225).",
1082,1087d1074
<                         "description": "Indicates whether the d3d11 compositor was blocklisted due to driver bugs. (This field replaces blacklisted as of bug 1647225).",
<                         "mode": "NULLABLE",
<                         "name": "blocklisted",
<                         "type": "BOOL"
<                       },
<                       {
1535,1540d1521
<             "type": "STRING"
<           },
<           {
<             "description": "<string>, Comma-separated list of enabled experimental features from about:preferences#experimental.",
<             "mode": "NULLABLE",
<             "name": "experimental_features",
diff integration/8b05ee6/telemetry.event.4.bq integration/be1e3f8/telemetry.event.4.bq
1026,1031d1025
<                 "description": "Whether or not Firefox desktop is embedded by Firefox Reality. This is Windows only.",
<                 "mode": "NULLABLE",
<                 "name": "embedded_in_firefox_reality",
<                 "type": "BOOL"
<               },
<               {
1076d1069
<                         "description": "Indicates whether the d3d11 compositor was blocklisted due to driver bugs. (This field was renamed to blocklisted in bug 1647225).",
1079,1084d1071
<                         "type": "BOOL"
<                       },
<                       {
<                         "description": "Indicates whether the d3d11 compositor was blocklisted due to driver bugs. (This field replaces blacklisted as of bug 1647225).",
<                         "mode": "NULLABLE",
<                         "name": "blocklisted",
diff integration/8b05ee6/telemetry.first-shutdown.4.bq integration/be1e3f8/telemetry.first-shutdown.4.bq
1026,1031d1025
<                 "description": "Whether or not Firefox desktop is embedded by Firefox Reality. This is Windows only.",
<                 "mode": "NULLABLE",
<                 "name": "embedded_in_firefox_reality",
<                 "type": "BOOL"
<               },
<               {
1076d1069
<                         "description": "Indicates whether the d3d11 compositor was blocklisted due to driver bugs. (This field was renamed to blocklisted in bug 1647225).",
1079,1084d1071
<                         "type": "BOOL"
<                       },
<                       {
<                         "description": "Indicates whether the d3d11 compositor was blocklisted due to driver bugs. (This field replaces blacklisted as of bug 1647225).",
<                         "mode": "NULLABLE",
<                         "name": "blocklisted",
diff integration/8b05ee6/telemetry.heartbeat.4.bq integration/be1e3f8/telemetry.heartbeat.4.bq
1031,1036d1030
<                 "description": "Whether or not Firefox desktop is embedded by Firefox Reality. This is Windows only.",
<                 "mode": "NULLABLE",
<                 "name": "embedded_in_firefox_reality",
<                 "type": "BOOL"
<               },
<               {
1081d1074
<                         "description": "Indicates whether the d3d11 compositor was blocklisted due to driver bugs. (This field was renamed to blocklisted in bug 1647225).",
1084,1089d1076
<                         "type": "BOOL"
<                       },
<                       {
<                         "description": "Indicates whether the d3d11 compositor was blocklisted due to driver bugs. (This field replaces blacklisted as of bug 1647225).",
<                         "mode": "NULLABLE",
<                         "name": "blocklisted",
diff integration/8b05ee6/telemetry.main.4.bq integration/be1e3f8/telemetry.main.4.bq
1026,1031d1025
<                 "description": "Whether or not Firefox desktop is embedded by Firefox Reality. This is Windows only.",
<                 "mode": "NULLABLE",
<                 "name": "embedded_in_firefox_reality",
<                 "type": "BOOL"
<               },
<               {
1076d1069
<                         "description": "Indicates whether the d3d11 compositor was blocklisted due to driver bugs. (This field was renamed to blocklisted in bug 1647225).",
1079,1084d1071
<                         "type": "BOOL"
<                       },
<                       {
<                         "description": "Indicates whether the d3d11 compositor was blocklisted due to driver bugs. (This field replaces blacklisted as of bug 1647225).",
<                         "mode": "NULLABLE",
<                         "name": "blocklisted",
diff integration/8b05ee6/telemetry.modules.4.bq integration/be1e3f8/telemetry.modules.4.bq
1026,1031d1025
<                 "description": "Whether or not Firefox desktop is embedded by Firefox Reality. This is Windows only.",
<                 "mode": "NULLABLE",
<                 "name": "embedded_in_firefox_reality",
<                 "type": "BOOL"
<               },
<               {
1076d1069
<                         "description": "Indicates whether the d3d11 compositor was blocklisted due to driver bugs. (This field was renamed to blocklisted in bug 1647225).",
1079,1084d1071
<                         "type": "BOOL"
<                       },
<                       {
<                         "description": "Indicates whether the d3d11 compositor was blocklisted due to driver bugs. (This field replaces blacklisted as of bug 1647225).",
<                         "mode": "NULLABLE",
<                         "name": "blocklisted",
diff integration/8b05ee6/telemetry.new-profile.4.bq integration/be1e3f8/telemetry.new-profile.4.bq
1026,1031d1025
<                 "description": "Whether or not Firefox desktop is embedded by Firefox Reality. This is Windows only.",
<                 "mode": "NULLABLE",
<                 "name": "embedded_in_firefox_reality",
<                 "type": "BOOL"
<               },
<               {
1076d1069
<                         "description": "Indicates whether the d3d11 compositor was blocklisted due to driver bugs. (This field was renamed to blocklisted in bug 1647225).",
1079,1084d1071
<                         "type": "BOOL"
<                       },
<                       {
<                         "description": "Indicates whether the d3d11 compositor was blocklisted due to driver bugs. (This field replaces blacklisted as of bug 1647225).",
<                         "mode": "NULLABLE",
<                         "name": "blocklisted",
Only in integration/8b05ee6: telemetry.regrets-reporter-update.4.bq
diff integration/8b05ee6/telemetry.saved-session.4.bq integration/be1e3f8/telemetry.saved-session.4.bq
1026,1031d1025
<                 "description": "Whether or not Firefox desktop is embedded by Firefox Reality. This is Windows only.",
<                 "mode": "NULLABLE",
<                 "name": "embedded_in_firefox_reality",
<                 "type": "BOOL"
<               },
<               {
1076d1069
<                         "description": "Indicates whether the d3d11 compositor was blocklisted due to driver bugs. (This field was renamed to blocklisted in bug 1647225).",
1079,1084d1071
<                         "type": "BOOL"
<                       },
<                       {
<                         "description": "Indicates whether the d3d11 compositor was blocklisted due to driver bugs. (This field replaces blacklisted as of bug 1647225).",
<                         "mode": "NULLABLE",
<                         "name": "blocklisted",
diff integration/8b05ee6/telemetry.shield-icq-v1.4.bq integration/be1e3f8/telemetry.shield-icq-v1.4.bq
1026,1031d1025
<                 "description": "Whether or not Firefox desktop is embedded by Firefox Reality. This is Windows only.",
<                 "mode": "NULLABLE",
<                 "name": "embedded_in_firefox_reality",
<                 "type": "BOOL"
<               },
<               {
1076d1069
<                         "description": "Indicates whether the d3d11 compositor was blocklisted due to driver bugs. (This field was renamed to blocklisted in bug 1647225).",
1079,1084d1071
<                         "type": "BOOL"
<                       },
<                       {
<                         "description": "Indicates whether the d3d11 compositor was blocklisted due to driver bugs. (This field replaces blacklisted as of bug 1647225).",
<                         "mode": "NULLABLE",
<                         "name": "blocklisted",
diff integration/8b05ee6/telemetry.testpilot.4.bq integration/be1e3f8/telemetry.testpilot.4.bq
1026,1031d1025
<                 "description": "Whether or not Firefox desktop is embedded by Firefox Reality. This is Windows only.",
<                 "mode": "NULLABLE",
<                 "name": "embedded_in_firefox_reality",
<                 "type": "BOOL"
<               },
<               {
1076d1069
<                         "description": "Indicates whether the d3d11 compositor was blocklisted due to driver bugs. (This field was renamed to blocklisted in bug 1647225).",
1079,1084d1071
<                         "type": "BOOL"
<                       },
<                       {
<                         "description": "Indicates whether the d3d11 compositor was blocklisted due to driver bugs. (This field replaces blacklisted as of bug 1647225).",
<                         "mode": "NULLABLE",
<                         "name": "blocklisted",
diff integration/8b05ee6/telemetry.third-party-modules.4.bq integration/be1e3f8/telemetry.third-party-modules.4.bq
1026,1031d1025
<                 "description": "Whether or not Firefox desktop is embedded by Firefox Reality. This is Windows only.",
<                 "mode": "NULLABLE",
<                 "name": "embedded_in_firefox_reality",
<                 "type": "BOOL"
<               },
<               {
1076d1069
<                         "description": "Indicates whether the d3d11 compositor was blocklisted due to driver bugs. (This field was renamed to blocklisted in bug 1647225).",
1079,1084d1071
<                         "type": "BOOL"
<                       },
<                       {
<                         "description": "Indicates whether the d3d11 compositor was blocklisted due to driver bugs. (This field replaces blacklisted as of bug 1647225).",
<                         "mode": "NULLABLE",
<                         "name": "blocklisted",
diff integration/8b05ee6/telemetry.untrusted-modules.4.bq integration/be1e3f8/telemetry.untrusted-modules.4.bq
1026,1031d1025
<                 "description": "Whether or not Firefox desktop is embedded by Firefox Reality. This is Windows only.",
<                 "mode": "NULLABLE",
<                 "name": "embedded_in_firefox_reality",
<                 "type": "BOOL"
<               },
<               {
1076d1069
<                         "description": "Indicates whether the d3d11 compositor was blocklisted due to driver bugs. (This field was renamed to blocklisted in bug 1647225).",
1079,1084d1071
<                         "type": "BOOL"
<                       },
<                       {
<                         "description": "Indicates whether the d3d11 compositor was blocklisted due to driver bugs. (This field replaces blacklisted as of bug 1647225).",
<                         "mode": "NULLABLE",
<                         "name": "blocklisted",
diff integration/8b05ee6/telemetry.untrustedModules.4.bq integration/be1e3f8/telemetry.untrustedModules.4.bq
1026,1031d1025
<                 "description": "Whether or not Firefox desktop is embedded by Firefox Reality. This is Windows only.",
<                 "mode": "NULLABLE",
<                 "name": "embedded_in_firefox_reality",
<                 "type": "BOOL"
<               },
<               {
1076d1069
<                         "description": "Indicates whether the d3d11 compositor was blocklisted due to driver bugs. (This field was renamed to blocklisted in bug 1647225).",
1079,1084d1071
<                         "type": "BOOL"
<                       },
<                       {
<                         "description": "Indicates whether the d3d11 compositor was blocklisted due to driver bugs. (This field replaces blacklisted as of bug 1647225).",
<                         "mode": "NULLABLE",
<                         "name": "blocklisted",
diff integration/8b05ee6/telemetry.update.4.bq integration/be1e3f8/telemetry.update.4.bq
1026,1031d1025
<                 "description": "Whether or not Firefox desktop is embedded by Firefox Reality. This is Windows only.",
<                 "mode": "NULLABLE",
<                 "name": "embedded_in_firefox_reality",
<                 "type": "BOOL"
<               },
<               {
1076d1069
<                         "description": "Indicates whether the d3d11 compositor was blocklisted due to driver bugs. (This field was renamed to blocklisted in bug 1647225).",
1079,1084d1071
<                         "type": "BOOL"
<                       },
<                       {
<                         "description": "Indicates whether the d3d11 compositor was blocklisted due to driver bugs. (This field replaces blacklisted as of bug 1647225).",
<                         "mode": "NULLABLE",
<                         "name": "blocklisted",
diff integration/8b05ee6/telemetry.voice.4.bq integration/be1e3f8/telemetry.voice.4.bq
1026,1031d1025
<                 "description": "Whether or not Firefox desktop is embedded by Firefox Reality. This is Windows only.",
<                 "mode": "NULLABLE",
<                 "name": "embedded_in_firefox_reality",
<                 "type": "BOOL"
<               },
<               {
1076d1069
<                         "description": "Indicates whether the d3d11 compositor was blocklisted due to driver bugs. (This field was renamed to blocklisted in bug 1647225).",
1079,1084d1071
<                         "type": "BOOL"
<                       },
<                       {
<                         "description": "Indicates whether the d3d11 compositor was blocklisted due to driver bugs. (This field replaces blacklisted as of bug 1647225).",
<                         "mode": "NULLABLE",
<                         "name": "blocklisted",
Only in integration/be1e3f8: telemetry.xfocsp-error-report.4.bq
Only in integration/be1e3f8: xfocsp-error-report.xfocsp-error-report.4.bq

Copy link
Contributor

@acmiyaguchi acmiyaguchi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The schema itself looks fine. Why does the schema need to exist in the telemetry namespace if it will be rerouted to another pipeline family?

Copy link
Contributor

@acmiyaguchi acmiyaguchi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is actually consistent with the regrets-reporter schema in #556, though in that case I believe all the pings will actually be available in the Telemetry namespace as an exceptional case.

Copy link
Member

@whd whd left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll be responsible for merging this PR as we roll out the ACL pieces.

@jklukas
Copy link
Contributor Author

jklukas commented Jul 22, 2020

The schema itself looks fine. Why does the schema need to exist in the telemetry namespace if it will be rerouted to another pipeline family?

I had thought we would need to validate based on the telemetry schema, but then create the table based on the other schema, but it looks like we can do the rerouting immediately after parsing the URI.

For sanity, though, I think it makes sense to keep a copy under telemetry since that's what clients will be sending.

Assuming the pipeline metadata schema proposal moves forward, we would refactor this in the future such that only the schema under telemetry exists, and the routing information is encoded in that schema.

jklukas added a commit to mozilla/gcp-ingestion that referenced this pull request Jul 22, 2020
@whd whd merged commit 361c16d into master Jul 22, 2020
@whd whd deleted the xfocsp branch July 22, 2020 20:48
dataops-pipeline-schemas added a commit that referenced this pull request Jul 22, 2020
361c16d	2020-07-22 20:48:09 +0000	Add xfocsp-error-report schema (#581)
acmiyaguchi pushed a commit that referenced this pull request Jul 22, 2020
361c16d	2020-07-22 20:48:09 +0000	Add xfocsp-error-report schema (#581)
8b05ee6	2020-07-21 14:27:14 -0700	Add regrets-reporter schemas (#556)
e724d79	2020-07-17 11:02:54 -0400	Bug 1652842 - Add a blocklist field so the environment schema works with pre- and post-bug-1647225 data (#576)
7400dc8	2020-07-15 12:46:09 +0200	Bug 1652834 - Add gfx.EmbeddedInFirefoxReality field to environment.
490cfe0	2020-07-14 16:39:13 -0400	AET update: 64-char user_ids and relaxing validation
8f22589	2020-07-14 11:56:08 -0400	Specify yaml loader in extract_crash_annotation_fields script (#573)
f0f50c4	2020-07-14 11:55:40 -0400	Bug 1652777 - Add experimental features to crash ping schema (#574)
97bac7a	2020-06-30 15:16:36 -0700	Add 'openglCompositing' and 'wrCompositor' to schema (#569)
c47313b	2020-06-29 18:40:32 +0000	Remove ecosystem_device_id fields (#567)
2cfd30a	2020-06-26 15:09:52 -0400	Make ecosystem_device_id an optional field
3f11d8b	2020-06-23 18:03:10 +0200	Update sync ping schema to make all migration fields transpiled into BQ
e678002	2020-06-22 14:12:25 -0400	add history_average_days_per_month to schema
be61e13	2020-06-22 14:12:25 -0400	fix schema to contain all fields
6ecd201	2020-06-22 14:12:25 -0400	add schema for normandy-login-study bug 1643383
2e62aac	2020-06-22 12:53:32 -0400	Pin Centos image
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants