Bug 1631849 - Remove pioneer-study pings and update pioneer-study envelope

[acmiyaguchi]

bug link

This removes all of the pioneer-study pings under the respective namespaces. This also updates the telemetry/pioneer-study envelope schema to include a schemaNamespace for pioneer v2. This field is required, so it will start rejecting existing pioneer v1 pings into the error stream as a validation error.

Checklist for reviewer:

• Commits should reference a bug or github issue, if relevant (if a bug is referenced, the pull request should include the bug number in the title)
• Scan the PR and verify that no changes (particularly to .circleci/config.yml) will cause environment variables (particularly credentials) to be exposed in test logs
• If the PR comes from a fork, trigger the integration CI test by pushing this revision as discussed in the README and review the report posted in the comments.

For glean changes:

• Update include/glean/CHANGELOG.md

[dataops-ci-bot]

Integration report for "Add encryptionKeyId to validation ping"

Report for upstream
Report for latest commit

cfd06fe-8e1b6a2.diff

No content detected.

bq_schema_cfd06fe-8e1b6a2.diff

Click to expand!

diff integration/cfd06fe/telemetry.pioneer-study.4.bq integration/8e1b6a2/telemetry.pioneer-study.4.bq
91a92,97
>         "description": "The namespace of the schema used for validation and routing to a dataset.",
>         "mode": "REQUIRED",
>         "name": "schema_namespace",
>         "type": "STRING"
>       },
>       {

[fbertsch]

@acmiyaguchi, how do you feel about moving pioneer_study to pioneer_study_envelope or just pioneer_envelope to make it more obvious what this is? That would also allow us to ignore the long tail of pioneer_study pings we're getting.

[acmiyaguchi]

Moving a schema sounds involved, with the condition that the table is deleted before the next deploy. I'm in favor of reusing this since it requires no incompatible state changes in BigQuery and little to no change in the client code for submitting to the correct route.

As for the long tail, adding the new field will already allow us to reject the ping via a validation exception. Nothing stops us from marking this data as unwanted in the decoder since the study pings should get rerouted before they reach the telemetry decoder in the near future.

[fbertsch]

Sounds good, you convinced me :)