Update CI to follow MSG and post compact columns

[acmiyaguchi]

This should fix #543 by using the --new-file option for diff. This should also post diffs of the compact column view.

Checklist for reviewer:

• Commits should reference a bug or github issue, if relevant (if a bug is referenced, the pull request should include the bug number in the title)
• If adding a new field, the field should have a description (see Bug 1652842 - Add a blocklist field so the environment schema works with pre- and post-bug-1647225 data #576 for an example)
• Scan the PR and verify that no changes (particularly to .circleci/config.yml) will cause environment variables (particularly credentials) to be exposed in test logs
• If the PR comes from a fork, trigger the integration CI test by pushing this revision as discussed in the README and review the report posted in the comments.

For glean changes:

• Update templates/include/glean/CHANGELOG.md

[dataops-ci-bot]

Integration report for "Update tests for new behaviors"

bq_schema_8e4c511f-9695b567.diff

No content detected.

compact_schema_8e4c511f-9695b567.diff

No content detected.

[dataops-ci-bot]

Integration report for "[testing] add schema to test diff"

bq_schema_8e4c511f-1e205f9b.diff

Click to expand!

Only in /app/integration/1e205f9b: test.testing.1.bq
Only in /app/integration/1e205f9b: test.testing.1.txt

compact_schema_8e4c511f-1e205f9b.diff

Click to expand!

Only in /app/integration/1e205f9b: test.testing.1.bq
Only in /app/integration/1e205f9b: test.testing.1.txt

[dataops-ci-bot]

Integration report for "Fix location of option"

bq_schema_8e4c511f-03f3b407.diff

Click to expand!

diff --new-file --exclude '*.txt' /app/integration/8e4c511f/test.testing.1.bq /app/integration/03f3b407/test.testing.1.bq
0a1,36
> [
>   {
>     "mode": "NULLABLE",
>     "name": "leaf",
>     "type": "INT64"
>   },
>   {
>     "fields": [
>       {
>         "mode": "NULLABLE",
>         "name": "leaf",
>         "type": "INT64"
>       }
>     ],
>     "mode": "NULLABLE",
>     "name": "nested",
>     "type": "RECORD"
>   },
>   {
>     "mode": "REPEATED",
>     "name": "repeated",
>     "type": "INT64"
>   },
>   {
>     "fields": [
>       {
>         "mode": "NULLABLE",
>         "name": "leaf",
>         "type": "INT64"
>       }
>     ],
>     "mode": "REPEATED",
>     "name": "repeated_nested",
>     "type": "RECORD"
>   }
> ]

compact_schema_8e4c511f-03f3b407.diff

Click to expand!

diff --new-file --exclude '*.bq' /app/integration/8e4c511f/test.testing.1.txt /app/integration/03f3b407/test.testing.1.txt
0a1,4
> root.leaf INT64
> root.nested.leaf INT64
> root.repeated.[] INT64
> root.repeated_nested.[].leaf INT64
\ No newline at end of file

[dataops-ci-bot]

Integration report for "Revert "[testing] add schema to test diff""

bq_schema_8e4c511f-b2f7209b.diff

No content detected.

compact_schema_8e4c511f-b2f7209b.diff

No content detected.

[acmiyaguchi]

The output of mps bigquery diff is available in the artifact tab of the circleci page now.

[fbertsch]

r+. Can you add the issue to the commit when you rebase?

[acmiyaguchi]

black doesn't change imports, that's the job of isort.

[acmiyaguchi]

black doesn't change imports, that's the job of isort.

[dataops-ci-bot]

Integration report for "Update mozilla_pipeline_schemas/cli/bigquery.py"

bq_schema_8e4c511f-61100f14.diff

No content detected.

compact_schema_8e4c511f-61100f14.diff

No content detected.