Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Introduce a blob file reader class #7461

Closed
wants to merge 60 commits into from
Closed

Introduce a blob file reader class #7461

Changes from 1 commit
Commits
Show all changes
60 commits
Select commit Hold shift + click to select a range
bdfc93a
Factor blob reading logic out into a separate class
ltamasi Sep 17, 2020
da2d08a
Introduce helpers to validate blob offset/calculate adjustment for bl…
ltamasi Sep 21, 2020
7131d4d
Make status messages more consistent
ltamasi Sep 21, 2020
c62c659
Read the entire blob log record and check key size, value size, and t…
ltamasi Sep 21, 2020
f8d010e
Only read blob record header/perform verification if verify_checksums…
ltamasi Sep 21, 2020
0e171ac
Factor blob record reading logic out into a helper method
ltamasi Sep 21, 2020
9025e85
Explicitly convert record_size to size_t
ltamasi Sep 21, 2020
c9c5a7f
Factor checksum verification out into a separate method
ltamasi Sep 21, 2020
fdf620b
Fix some integer types
ltamasi Sep 21, 2020
4536b06
Rename and generalize ReadBlobFromFile
ltamasi Sep 21, 2020
428ca5d
Turn ReadFromFile into a static helper
ltamasi Sep 21, 2020
21abe86
Read blob file header
ltamasi Sep 22, 2020
e7a7e5c
Make VerifyBlob a static method
ltamasi Sep 22, 2020
01a15b8
Validate column family ID
ltamasi Sep 22, 2020
e5d8fee
Store compression type from blob file header
ltamasi Sep 22, 2020
17efc7c
Pass in IOOptions as needed (for deadline/timeout support)
ltamasi Sep 22, 2020
cfe355f
Factor file opening out into a helper method
ltamasi Sep 22, 2020
76ecfbb
Factor header reading out into a separate method
ltamasi Sep 22, 2020
f0b8b87
Reorder methods a bit in the .cc file
ltamasi Sep 22, 2020
d29bac1
Initialize compression_type with kNoCompression
ltamasi Sep 22, 2020
c5993cb
Expose the compression type from BlobIndex and log it in DebugString
ltamasi Sep 22, 2020
b2d1ccc
Validate compression type when reading blob
ltamasi Sep 22, 2020
6ed012a
Implement uncompression in BlobFileReader
ltamasi Sep 22, 2020
a5353ef
Remove IOOptions related stuff for now
ltamasi Sep 23, 2020
8aafc5e
Use size_t uncompressed_size in BlobFileReader
ltamasi Sep 23, 2020
1a8c283
Read/validate footer as well; small cleanup
ltamasi Sep 24, 2020
6bae733
Add read latency histogram for blob files
ltamasi Sep 24, 2020
199b519
Adjust status message
ltamasi Sep 24, 2020
e04119a
Add a unit test case
ltamasi Sep 24, 2020
90d16de
Check blob file size before doing anything else
ltamasi Sep 24, 2020
e08a77e
Factor blob file writing out into a helper method in blob_file_reader…
ltamasi Sep 24, 2020
42acd1b
Add some test cases simulating I/O failures
ltamasi Sep 25, 2020
e084cd7
Change GetBlob to take a PinnableSlice instead of a GetContext
ltamasi Sep 25, 2020
79a74a2
Add tests for TTL related failure cases
ltamasi Sep 25, 2020
507aae7
Add test case for column family ID mismatch
ltamasi Sep 25, 2020
596ebbc
Test blob read with and without checksum verification
ltamasi Sep 25, 2020
f9f5311
Add test cases for invalid offset/compression type/key/value size
ltamasi Sep 25, 2020
e15b4e5
Small test cleanup
ltamasi Sep 26, 2020
5eab505
Add test case for incorrect key size
ltamasi Sep 28, 2020
1650df3
Move the definition of a test harness class
ltamasi Sep 28, 2020
84e6c0d
Add test cases for header/footer/record decoding failures
ltamasi Sep 29, 2020
147848a
Add a negative test case for blob CRC mismatch
ltamasi Sep 29, 2020
7596f8b
Add a test case for compression; small cleanup
ltamasi Sep 29, 2020
1569085
Add a test case for uncompression errors
ltamasi Sep 29, 2020
4d0d7f7
Delay adding the sync point callback until it's actually needed in Bl…
ltamasi Sep 29, 2020
6f291c6
Store file size in BlobFileReader
ltamasi Sep 29, 2020
5139bec
Validate offset against the end of file as well
ltamasi Sep 29, 2020
e4a847c
Add a test case for malformed blob files
ltamasi Sep 29, 2020
2bdbaef
Print compression type as string in BlobIndex::DebugString
ltamasi Sep 29, 2020
f7450a0
Remove unnecessary #include
ltamasi Sep 29, 2020
9a0cc44
Small const correctness improvement
ltamasi Sep 29, 2020
ad9236a
Remove redundant forward declaration
ltamasi Sep 29, 2020
2b270a1
Reorder forward declarations
ltamasi Sep 29, 2020
245f757
Update comment
ltamasi Sep 29, 2020
5cd84f5
Use std::unique_ptr<char[]> for buffer instead of abusing std::string
ltamasi Sep 29, 2020
e2d0d1d
Switch the order of arguments of BlobFileReader::SaveValue
ltamasi Oct 7, 2020
139236b
Do not pass key_size to VerifyBlob
ltamasi Oct 7, 2020
d7f90ea
Remove unnecessary #include
ltamasi Oct 7, 2020
f19c730
Add a comment for WriteBlobFile
ltamasi Oct 7, 2020
068d79b
Buckify again to make sure the new .cc file gets added to the new tar…
ltamasi Oct 7, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Small test cleanup
  • Loading branch information
@ltamasi
ltamasi committed Oct 7, 2020
commit e15b4e5e1e2d8dd9aba950b89625d086c9e6da64
6 changes: 3 additions & 3 deletions db/blob/blob_file_reader_test.cc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -116,22 +116,22 @@ TEST_F(BlobFileReaderTest, CreateReaderAndGetBlob) {

// Make sure the blob can be retrieved with and without checksum verification
ReadOptions read_options;
read_options.verify_checksums = false;

{
PinnableSlice value;

read_options.verify_checksums = false;

ASSERT_OK(reader->GetBlob(read_options, key, blob_offset, sizeof(blob) - 1,
kNoCompression, &value));
ASSERT_EQ(value, blob);
}

read_options.verify_checksums = true;

{
PinnableSlice value;

read_options.verify_checksums = true;

ASSERT_OK(reader->GetBlob(read_options, key, blob_offset, sizeof(blob) - 1,
kNoCompression, &value));
ASSERT_EQ(value, blob);
Expand Down