Bug 1677567 - Add rallyId to pioneer metadata schemas

[acmiyaguchi]

bug link

This adds the metadata field used by the ingestion system. The decoder will decide on how to fill in this field (which will likely be metrics.string.rally_id), which will be made available to shredder and consumers of the tables.

Checklist for reviewer:

• Commits should reference a bug or github issue, if relevant (if a bug is referenced, the pull request should include the bug number in the title)
• If adding a new field, the field should have a description (see Bug 1652842 - Add a blocklist field so the environment schema works with pre- and post-bug-1647225 data #576 for an example)
• Scan the PR and verify that no changes (particularly to .circleci/config.yml) will cause environment variables (particularly credentials) to be exposed in test logs
• If the PR comes from a fork, trigger the integration CI test by pushing this revision as discussed in the README and review the report posted in the comments.

For glean changes:

• Update templates/include/glean/CHANGELOG.md

[dataops-ci-bot]

Integration report for "Remove pioneerId as a required field (can be rallyId)"

bq_schema_731721ab-f44e9480.diff

Click to expand!

diff --new-file --exclude '*.txt' /app/integration/731721ab/metadata.pioneer-decoded.1.bq /app/integration/f44e9480/metadata.pioneer-decoded.1.bq
227a228,233
>     "description": "Bug 1677567 - Custom rally id reported by the core addon",
>     "mode": "NULLABLE",
>     "name": "rally_id",
>     "type": "STRING"
>   },
>   {
diff --new-file --exclude '*.txt' /app/integration/731721ab/metadata.pioneer-error.1.bq /app/integration/f44e9480/metadata.pioneer-error.1.bq
136a137,142
>     "description": "Bug 1677567 - Custom rally id reported by the core addon",
>     "mode": "NULLABLE",
>     "name": "rally_id",
>     "type": "STRING"
>   },
>   {
diff --new-file --exclude '*.txt' /app/integration/731721ab/metadata.pioneer-ingestion.1.bq /app/integration/f44e9480/metadata.pioneer-ingestion.1.bq
180a181,186
>     "description": "Bug 1677567 - Custom rally id reported by the core addon",
>     "mode": "NULLABLE",
>     "name": "rally_id",
>     "type": "STRING"
>   },
>   {

compact_schema_731721ab-f44e9480.diff

Click to expand!

diff --new-file --exclude '*.bq' /app/integration/731721ab/metadata.pioneer-decoded.1.txt /app/integration/f44e9480/metadata.pioneer-decoded.1.txt
32a33
> root.rally_id STRING
diff --new-file --exclude '*.bq' /app/integration/731721ab/metadata.pioneer-error.1.txt /app/integration/f44e9480/metadata.pioneer-error.1.txt
25a26
> root.rally_id STRING
diff --new-file --exclude '*.bq' /app/integration/731721ab/metadata.pioneer-ingestion.1.txt /app/integration/f44e9480/metadata.pioneer-ingestion.1.txt
24a25
> root.rally_id STRING

[whd]

This should be safe to merge since it is additive to existing operational schemas. I plan to merge this tomorrow pending :Dexter feedback.

[dataops-ci-bot]

Integration report for "Force the appearance of an id and add tests"

bq_schema_731721ab-3434aecc.diff

Click to expand!

diff --new-file --exclude '*.txt' /app/integration/731721ab/metadata.pioneer-decoded.1.bq /app/integration/3434aecc/metadata.pioneer-decoded.1.bq
227a228,233
>     "description": "Bug 1677567 - Custom rally id reported by the core addon",
>     "mode": "NULLABLE",
>     "name": "rally_id",
>     "type": "STRING"
>   },
>   {
diff --new-file --exclude '*.txt' /app/integration/731721ab/metadata.pioneer-error.1.bq /app/integration/3434aecc/metadata.pioneer-error.1.bq
136a137,142
>     "description": "Bug 1677567 - Custom rally id reported by the core addon",
>     "mode": "NULLABLE",
>     "name": "rally_id",
>     "type": "STRING"
>   },
>   {
diff --new-file --exclude '*.txt' /app/integration/731721ab/metadata.pioneer-ingestion.1.bq /app/integration/3434aecc/metadata.pioneer-ingestion.1.bq
180a181,186
>     "description": "Bug 1677567 - Custom rally id reported by the core addon",
>     "mode": "NULLABLE",
>     "name": "rally_id",
>     "type": "STRING"
>   },
>   {

compact_schema_731721ab-3434aecc.diff

Click to expand!

diff --new-file --exclude '*.bq' /app/integration/731721ab/metadata.pioneer-decoded.1.txt /app/integration/3434aecc/metadata.pioneer-decoded.1.txt
32a33
> root.rally_id STRING
diff --new-file --exclude '*.bq' /app/integration/731721ab/metadata.pioneer-error.1.txt /app/integration/3434aecc/metadata.pioneer-error.1.txt
25a26
> root.rally_id STRING
diff --new-file --exclude '*.bq' /app/integration/731721ab/metadata.pioneer-ingestion.1.txt /app/integration/3434aecc/metadata.pioneer-ingestion.1.txt
24a25
> root.rally_id STRING

[Dexterp37]

bug link

This adds the metadata field used by the ingestion system. The decoder will decide on how to fill in this field (which will likely be metrics.string.rally_id), which will be made available to shredder and consumers of the tables.

The field will likely bemetrics.uuid.rally_id (it's in the uuid section of the ping, not the string section).

[dataops-ci-bot]

Integration report for "Fix nits: extra newline and custom"

bq_schema_731721ab-dbb2fac8.diff

Click to expand!

diff --new-file --exclude '*.txt' /app/integration/731721ab/metadata.pioneer-decoded.1.bq /app/integration/dbb2fac8/metadata.pioneer-decoded.1.bq
227a228,233
>     "description": "Bug 1677567 - Rally id reported by the core addon",
>     "mode": "NULLABLE",
>     "name": "rally_id",
>     "type": "STRING"
>   },
>   {
diff --new-file --exclude '*.txt' /app/integration/731721ab/metadata.pioneer-error.1.bq /app/integration/dbb2fac8/metadata.pioneer-error.1.bq
136a137,142
>     "description": "Bug 1677567 - Rally id reported by the core addon",
>     "mode": "NULLABLE",
>     "name": "rally_id",
>     "type": "STRING"
>   },
>   {
diff --new-file --exclude '*.txt' /app/integration/731721ab/metadata.pioneer-ingestion.1.bq /app/integration/dbb2fac8/metadata.pioneer-ingestion.1.bq
180a181,186
>     "description": "Bug 1677567 - Rally id reported by the core addon",
>     "mode": "NULLABLE",
>     "name": "rally_id",
>     "type": "STRING"
>   },
>   {

compact_schema_731721ab-dbb2fac8.diff

Click to expand!

diff --new-file --exclude '*.bq' /app/integration/731721ab/metadata.pioneer-decoded.1.txt /app/integration/dbb2fac8/metadata.pioneer-decoded.1.txt
32a33
> root.rally_id STRING
diff --new-file --exclude '*.bq' /app/integration/731721ab/metadata.pioneer-error.1.txt /app/integration/dbb2fac8/metadata.pioneer-error.1.txt
25a26
> root.rally_id STRING
diff --new-file --exclude '*.bq' /app/integration/731721ab/metadata.pioneer-ingestion.1.txt /app/integration/dbb2fac8/metadata.pioneer-ingestion.1.txt
24a25
> root.rally_id STRING