BREAKING: fix json marshal unmarshal for namespace > 127 #7810
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
NamanJain8
requested review from
manishrjain,
pawanrawal and
vvbalaji-dgraph
as code owners
NamanJain8
changed the title
github-actions
bot
added
area/enterprise
NamanJain8
force-pushed
the
naman/fix-json-127
branch
from
9830cad
to
057b989
Compare
manishrjain
approved these changes
May 12, 2021
There was a problem hiding this comment.
0x in the front. Just the hex should be sufficient.
Reviewed 6 of 6 files at r1.
Reviewable status:complete! all files reviewed, all discussions resolved (waiting on @pawanrawal and @vvbalaji-dgraph)
NamanJain8
changed the title
NamanJain8
added a commit
that referenced
this pull request
May 13, 2021
We used to store predicate as <namespace>|<attribute> (pipe | signifies concatenation). We store this as a string. <namespace> is 8 bytes uint64, which when marshaled to JSON bytes mess up the predicate. This is because for the namespace greater than 127, the UTF-8 encoding might take up several bytes (also if the mapping does not exist, then it replaces it with some other rune). This affects three identified places in Dgraph: Live loader Backup and List Backup Http clients and Ratel Fix: Fix is to have a UTF-8 string when dealing with JSON. A better idea is to use UTF-8 string even for internal operations. Only when we read/write to badger we convert it into the format of the byte. New Format: <anmespace>-<attribute> (- is the hyphen literal) (cherry picked from commit 90d77f3)
NamanJain8
added a commit
that referenced
this pull request
May 13, 2021
This PR adds `dgraph updatemanifest` tool. This can be used to migrate the manifest of the backups taken on release/v21.03 or their parallels to the state after this change #7810
NamanJain8
added a commit
that referenced
this pull request
May 19, 2021
With #7810 change, we changed the format of the predicate. We missed updating the schema and predicate. This PR fixes it.
NamanJain8
added a commit
that referenced
this pull request
May 24, 2021
We used to store predicate as <namespace>|<attribute> (pipe | signifies concatenation). We store this as a string. <namespace> is 8 bytes uint64, which when marshaled to JSON bytes mess up the predicate. This is because for the namespace greater than 127, the UTF-8 encoding might take up several bytes (also if the mapping does not exist, then it replaces it with some other rune). This affects three identified places in Dgraph: Live loader Backup and List Backup Http clients and Ratel Fix: Fix is to have a UTF-8 string when dealing with JSON. A better idea is to use UTF-8 string even for internal operations. Only when we read/write to badger we convert it into the format of the byte. New Format: <anmespace>-<attribute> (- is the hyphen literal) (cherry picked from commit 90d77f3)
NamanJain8
added a commit
that referenced
this pull request
May 24, 2021
NamanJain8
added a commit
that referenced
this pull request
May 24, 2021
mangalaman93
pushed a commit
that referenced
this pull request
Jan 2, 2023
We used to store predicate as <namespace>|<attribute> (pipe | signifies concatenation). We store this as a string. <namespace> is 8 bytes uint64, which when marshaled to JSON bytes mess up the predicate. This is because for the namespace greater than 127, the UTF-8 encoding might take up several bytes (also if the mapping does not exist, then it replaces it with some other rune). This affects three identified places in Dgraph: * Live loader * Backup and List Backup * Http clients and Ratel Fix is to have a UTF-8 string when dealing with JSON. A better idea is to use UTF-8 string even for internal operations. Only when we read/write to badger we convert it into the format of the byte. New Format: <anmespace>-<attribute> (- is the hyphen literal)
mangalaman93
pushed a commit
that referenced
this pull request
Jan 2, 2023
This PR adds `dgraph updatemanifest` tool. This can be used to migrate the manifest of the backups taken on release/v21.03 or their parallels to the state after this change #7810
mangalaman93
pushed a commit
that referenced
this pull request
Jan 3, 2023
This PR adds `dgraph updatemanifest` tool. This can be used to migrate the manifest of the backups taken on release/v21.03 or their parallels to the state after this change #7810
harshil-goel
pushed a commit
that referenced
this pull request
Jan 13, 2023
With #7810 change, we changed the format of the predicate. We missed updating the schema and predicate. This PR fixes it.
harshil-goel
pushed a commit
that referenced
this pull request
Jan 30, 2023
We used to store predicate as <namespace>|<attribute> (pipe | signifies concatenation). We store this as a string. <namespace> is 8 bytes uint64, which when marshaled to JSON bytes mess up the predicate. This is because for the namespace greater than 127, the UTF-8 encoding might take up several bytes (also if the mapping does not exist, then it replaces it with some other rune). This affects three identified places in Dgraph: Live loader Backup and List Backup Http clients and Ratel Fix: Fix is to have a UTF-8 string when dealing with JSON. A better idea is to use UTF-8 string even for internal operations. Only when we read/write to badger we convert it into the format of the byte. New Format: <anmespace>-<attribute> (- is the hyphen literal)
harshil-goel
pushed a commit
that referenced
this pull request
Jan 30, 2023
With #7810 change, we changed the format of the predicate. We missed updating the schema and predicate. This PR fixes it.
harshil-goel
pushed a commit
that referenced
this pull request
Jan 31, 2023
We used to store predicate as <namespace>|<attribute> (pipe | signifies concatenation). We store this as a string. <namespace> is 8 bytes uint64, which when marshaled to JSON bytes mess up the predicate. This is because for the namespace greater than 127, the UTF-8 encoding might take up several bytes (also if the mapping does not exist, then it replaces it with some other rune). This affects three identified places in Dgraph: Live loader Backup and List Backup Http clients and Ratel Fix: Fix is to have a UTF-8 string when dealing with JSON. A better idea is to use UTF-8 string even for internal operations. Only when we read/write to badger we convert it into the format of the byte. New Format: <anmespace>-<attribute> (- is the hyphen literal)
harshil-goel
pushed a commit
that referenced
this pull request
Jan 31, 2023
With #7810 change, we changed the format of the predicate. We missed updating the schema and predicate. This PR fixes it.
harshil-goel
pushed a commit
that referenced
this pull request
Feb 2, 2023
We used to store predicate as <namespace>|<attribute> (pipe | signifies concatenation). We store this as a string. <namespace> is 8 bytes uint64, which when marshaled to JSON bytes mess up the predicate. This is because for the namespace greater than 127, the UTF-8 encoding might take up several bytes (also if the mapping does not exist, then it replaces it with some other rune). This affects three identified places in Dgraph: Live loader Backup and List Backup Http clients and Ratel Fix: Fix is to have a UTF-8 string when dealing with JSON. A better idea is to use UTF-8 string even for internal operations. Only when we read/write to badger we convert it into the format of the byte. New Format: <anmespace>-<attribute> (- is the hyphen literal)
harshil-goel
pushed a commit
that referenced
this pull request
Feb 2, 2023
With #7810 change, we changed the format of the predicate. We missed updating the schema and predicate. This PR fixes it.
harshil-goel
pushed a commit
that referenced
this pull request
Feb 3, 2023
We used to store predicate as <namespace>|<attribute> (pipe | signifies concatenation). We store this as a string. <namespace> is 8 bytes uint64, which when marshaled to JSON bytes mess up the predicate. This is because for the namespace greater than 127, the UTF-8 encoding might take up several bytes (also if the mapping does not exist, then it replaces it with some other rune). This affects three identified places in Dgraph: Live loader Backup and List Backup Http clients and Ratel Fix: Fix is to have a UTF-8 string when dealing with JSON. A better idea is to use UTF-8 string even for internal operations. Only when we read/write to badger we convert it into the format of the byte. New Format: <anmespace>-<attribute> (- is the hyphen literal) fix(restore): update the schema and type from 2103 (#7838) With #7810 change, we changed the format of the predicate. We missed updating the schema and predicate. This PR fixes it. fix(state): fix hex to uint64 response of list of namespaces (#8091) There is an issue in ExtractNamespaceFromPredicate. The issue is the parsing was done assuming ns in <ns>-<attr> to be decimal (actually it is hexadecimal). This leads to the following issues. A predicate a-name, it was skipped. A predicate 11-name was parsed as namespace 11, actually it is namespace 17 (0x11). fix(backup): handle manifest version logic, update manifest version to 2105 (#7825) The backward compatibility of the backup's manifest was broken by #7810, although the tool was added (#7815) that enables smooth migration of manifest. This PR makes backup backward compatible, by updating the manifest(in-memory) after reading. fix(updatemanifest): update the version of manifest after update (#7828) We were not updating the manifest version after the updation. This PR fixes that.
harshil-goel
pushed a commit
that referenced
this pull request
Feb 3, 2023
We used to store predicate as <namespace>|<attribute> (pipe | signifies concatenation). We store this as a string. <namespace> is 8 bytes uint64, which when marshaled to JSON bytes mess up the predicate. This is because for the namespace greater than 127, the UTF-8 encoding might take up several bytes (also if the mapping does not exist, then it replaces it with some other rune). This affects three identified places in Dgraph: Live loader Backup and List Backup Http clients and Ratel Fix: Fix is to have a UTF-8 string when dealing with JSON. A better idea is to use UTF-8 string even for internal operations. Only when we read/write to badger we convert it into the format of the byte. New Format: <anmespace>-<attribute> (- is the hyphen literal) fix(restore): update the schema and type from 2103 (#7838) With #7810 change, we changed the format of the predicate. We missed updating the schema and predicate. This PR fixes it. fix(state): fix hex to uint64 response of list of namespaces (#8091) There is an issue in ExtractNamespaceFromPredicate. The issue is the parsing was done assuming ns in <ns>-<attr> to be decimal (actually it is hexadecimal). This leads to the following issues. A predicate a-name, it was skipped. A predicate 11-name was parsed as namespace 11, actually it is namespace 17 (0x11). fix(backup): handle manifest version logic, update manifest version to 2105 (#7825) The backward compatibility of the backup's manifest was broken by #7810, although the tool was added (#7815) that enables smooth migration of manifest. This PR makes backup backward compatible, by updating the manifest(in-memory) after reading. fix(updatemanifest): update the version of manifest after update (#7828) We were not updating the manifest version after the updation. This PR fixes that.
harshil-goel
pushed a commit
that referenced
this pull request
Feb 3, 2023
We used to store predicate as <namespace>|<attribute> (pipe | signifies concatenation). We store this as a string. <namespace> is 8 bytes uint64, which when marshaled to JSON bytes mess up the predicate. This is because for the namespace greater than 127, the UTF-8 encoding might take up several bytes (also if the mapping does not exist, then it replaces it with some other rune). This affects three identified places in Dgraph: Live loader Backup and List Backup Http clients and Ratel Fix: Fix is to have a UTF-8 string when dealing with JSON. A better idea is to use UTF-8 string even for internal operations. Only when we read/write to badger we convert it into the format of the byte. New Format: <anmespace>-<attribute> (- is the hyphen literal)
harshil-goel
added a commit
that referenced
this pull request
Feb 3, 2023
…t json marshal issues (#8601) We used to store predicate as <namespace>|<attribute> (pipe | signifies concatenation). We store this as a string. <namespace> is 8 bytes uint64, which when marshaled to JSON bytes mess up the predicate. This is because for the namespace greater than 127, the UTF-8 encoding might take up several bytes (also if the mapping does not exist, then it replaces it with some other rune). This affects three identified places in Dgraph: - Live loader using guardian of galaxy - Backup and List Backup - Http clients and Ratel - Schema and predicate Fix: Fix is to have a UTF-8 string when dealing with JSON. A better idea is to use UTF-8 string even for internal operations. Only when we read/write to badger we convert it into the format of the byte. New Format: <namespace>-<attribute> (- is the hyphen literal) <namespace> is a string "81" in hex We also update the manifest version after update. This diff takes care that older backups are still compatible and can be used to restore. Contains: #7838 #7828 #7825 #7815 #7810
all-seeing-code
pushed a commit
that referenced
this pull request
Feb 8, 2023
…t json marshal issues (#8601) We used to store predicate as <namespace>|<attribute> (pipe | signifies concatenation). We store this as a string. <namespace> is 8 bytes uint64, which when marshaled to JSON bytes mess up the predicate. This is because for the namespace greater than 127, the UTF-8 encoding might take up several bytes (also if the mapping does not exist, then it replaces it with some other rune). This affects three identified places in Dgraph: - Live loader using guardian of galaxy - Backup and List Backup - Http clients and Ratel - Schema and predicate Fix: Fix is to have a UTF-8 string when dealing with JSON. A better idea is to use UTF-8 string even for internal operations. Only when we read/write to badger we convert it into the format of the byte. New Format: <namespace>-<attribute> (- is the hyphen literal) <namespace> is a string "81" in hex We also update the manifest version after update. This diff takes care that older backups are still compatible and can be used to restore. Contains: #7838 #7828 #7825 #7815 #7810
all-seeing-code
pushed a commit
that referenced
this pull request
Feb 8, 2023
…t json marshal issues (#8601) We used to store predicate as <namespace>|<attribute> (pipe | signifies concatenation). We store this as a string. <namespace> is 8 bytes uint64, which when marshaled to JSON bytes mess up the predicate. This is because for the namespace greater than 127, the UTF-8 encoding might take up several bytes (also if the mapping does not exist, then it replaces it with some other rune). This affects three identified places in Dgraph: - Live loader using guardian of galaxy - Backup and List Backup - Http clients and Ratel - Schema and predicate Fix: Fix is to have a UTF-8 string when dealing with JSON. A better idea is to use UTF-8 string even for internal operations. Only when we read/write to badger we convert it into the format of the byte. New Format: <namespace>-<attribute> (- is the hyphen literal) <namespace> is a string "81" in hex We also update the manifest version after update. This diff takes care that older backups are still compatible and can be used to restore. Contains: #7838 #7828 #7825 #7815 #7810
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
We used to store predicate as
<namespace>|<attribute>(pipe|signifies concatenation). We store this as a string.<namespace>is 8 bytes uint64, which when marshaled to JSON bytes mess up the predicate. This is because for the namespace greater than 127, the UTF-8 encoding might take up several bytes (also if the mapping does not exist, then it replaces it with some other rune). This affects three identified places in Dgraph:Fix:
Fix is to have a UTF-8 string when dealing with JSON. A better idea is to use UTF-8 string even for internal operations. Only when we read/write to badger we convert it into the format of the byte.
New Format:
<namespace>-<attribute>(-is the hyphen literal)<namespace>is a string "81" in hexThis change is