This repository was archived by the owner on May 17, 2024. It is now read-only.
Compare JSON, ARRAY, STRUCT types in BigQuery (simplistically) #533
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
nolar
changed the title
nolar
force-pushed
the
compare-bigquery-arrays-and-structs
branch
from
acd3b08 to
dcc2d44
Compare
nolar
referenced
this pull request
in nicolasaldecoa/sqeleton
May 3, 2023
…port Add support for column types: PostgreSQL JSON/JSONB and RedShift Super
nolar
force-pushed
the
simplify-json-columns
branch
from
74edbcb to
c2c5102
Compare
nolar
force-pushed
the
compare-bigquery-arrays-and-structs
branch
from
dcc2d44 to
0a4619c
Compare
dlawin
approved these changes
May 5, 2023
nolar
force-pushed
the
simplify-json-columns
branch
from
c2c5102 to
46c8867
Compare
nolar
force-pushed
the
compare-bigquery-arrays-and-structs
branch
from
0a4619c to
fc19d8e
Compare
nolar
force-pushed
the
compare-bigquery-arrays-and-structs
branch
from
fc19d8e to
c0d6570
Compare
nolar
force-pushed
the
compare-bigquery-arrays-and-structs
branch
from
c0d6570 to
7c8d058
Compare
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
An attempt to make BigQuery's
ARRAY&STRUCTcolumn types suitable for comparison.As of now, they are not supported at all, and data-diff does not even understand this type of columns. So the addition will only affect BigQuery and no other databases.
The comparison is rather simplistic — convert to JSON (lists & dicts correspondingly) and try to hash & compare those instead of the real arrays & structs.
This might be affected by different serialization results, such as different key ordering, different spacing & indenting in json, so on. This problem is to be solved separately in datafold/sqeleton#15.
Addresses: #445