PS-10067 [8.0]: Consume per-field metadata with Rdb_convert_to_record_key_decoder::skip
#5707
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…or index `unpack_info` consist of: ``` (1) [ index flags (opt) ] (2) [ tag (1 byte) ] (3) [ length (2 bytes, if tag==DATA_TAG) ] (4) [ covered bitmap (2 bytes, opt) ] (5) [ per-field metadata ] (6) [ checksum section (opt) ] ``` `unp_reader` in `Rdb_key_def::unpack_record` consumes an `unpack_info` header ((1) to (4)) but for some data types and lenghts per-field metadata (5) is not be consumed and `Rdb_key_def::unpack_record` tries to read checksum section (6) still pointing at (5) what causes the checksum mismatch error. To fix the issue we may: a) skip per-field metadata (5) and move to checksum section (6) but it requires implementation for each data type that doesn't consume it's metadata b) check if `unp_reader` is already moved to checksum section (6) and skip checksum checking in other cases
…d_key_decoder::skip` Index fields consist of many `Rdb_field_packing` records. Each record consist of: ``` (1) [ index flags (opt) ] (2) [ tag (1 byte) ] (3) [ length (2 bytes, if tag==DATA_TAG) ] (4) [ covered bitmap (2 bytes, opt) ] (5) [ per-field metadata ] (6) [ checksum section (opt) ] ``` `Rdb_key_field_iterator::next()` calls `Rdb_convert_to_record_key_decoder::skip()` when iterating thru index fields. For data created with `make_unpack_unknown_varlength()` the skip function moves forward only a key slice reader but an unpack information reader should also be moved forward (per-field metadata (5) should be consumed). The solution is based on `unpack_unknown_varlength` at https://github.com/percona/percona-server/blob/4f842346452d/storage/rocksdb/rdb_datadic.cc#L3571-L3588 It simulates unpacking and skips unpacked data.
inikep
force-pushed
the
PS-10075-8.0
branch
from
d9fc2af to
89d7b97
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Index fields consist of many
Rdb_field_packingrecords.Each record consist of:
Rdb_key_field_iterator::next()callsRdb_convert_to_record_key_decoder::skip()wheniterating thru index fields. For data created with
make_unpack_unknown_varlength()theskip function moves forward only a key slice reader but an unpack information reader
should also be moved forward (per-field metadata (5) should be consumed).
The solution is based on
unpack_unknown_varlengthat https://github.com/percona/percona-server/blob/4f842346452d/storage/rocksdb/rdb_datadic.cc#L3571-L3588It simulates unpacking and skips unpacked data.