Skip to content

crash_test: failures are reported when running with the new memtable #98

@isaac-io

Description

@isaac-io

When running the crash tests, all of them fail quickly with:

WARNING: prefix_size is non-zero but memtablerep != prefix_hash
Verification failed for column family 0 key 000000000000004500000000000000E500000000000000310000000000000070 (6402): Value not found: NotFound: 
Crash-recovery verification failed :(
No writes or ops?
Verification failed :(

as well as

***Iterator diverged from control iterator which has value 000000000001FF3A000000000000007C000000000000007C000000000000007C000000000000007C000000000000007C000000000000001F78787878787878 total_order_seek: 0 auto_prefix_mode: 0 S 000000000001E67D78 NPNPPPPPPN; total_order_seek: 0 auto_prefix_mode: 0 SFP 0000000000020034000000000000007C000000000000000A7878 ***
stderr has error message:
***iterator has value 000000000001E3E2000000000000007C000000000000007C000000000000007C000000000000007C000000000000007C000000000000007C0000000000000006***
stderr has error message:
***Control CF 1***
stderr has error message:
***Verification failed :(***
TEST FAILED. See kill option and exit code above!!!

and

WARNING: prefix_size is non-zero but memtablerep != prefix_hash
Verification failed
Latest Sequence Number: 2100250
[default] 00000000000013FF000000000000017B00000000000000A9000000000000005900000000000000090000000000000013787878 => 82840000868784858A8B88898E8F8C8D92939091969794959A9B98999E9F9C9DA2A3A0A1A6A7A4A5AAABA8A9AEAFACADB2B3B0B1B6B7B4B5BABBB8B9BEBFBCBD
[1] 00000000000013FF000000000000017B00000000000000A9000000000000003F78787878787878 => 7F8B01007B7A797877767574737271706F6E6D6C6B6A69686766656463626160
Internal keys in CF 'default', [00000000000013FF000000000000017B00000000000000A9000000000000003F78787878787878, 00000000000013FF000000000000017B00000000000000A9000000000000005900000000000000090000000000000013787878] (max 8)
  key 00000000000013FF000000000000017B00000000000000A9000000000000005900000000000000090000000000000013787878 seq 702931 type 1
Internal keys in CF '1', [00000000000013FF000000000000017B00000000000000A9000000000000003F78787878787878, 00000000000013FF000000000000017B00000000000000A9000000000000005900000000000000090000000000000013787878] (max 8)
  key 00000000000013FF000000000000017B00000000000000A9000000000000003F78787878787878 seq 2097862 type 1
  key 00000000000013FF000000000000017B00000000000000A9000000000000005900000000000000090000000000000013787878 seq 702932 type 1
[2] 00000000000013FF000000000000017B00000000000000A9000000000000003F78787878787878 => 7F8B01007B7A797877767574737271706F6E6D6C6B6A69686766656463626160
Internal keys in CF '2', [00000000000013FF000000000000017B00000000000000A9000000000000003F78787878787878, 00000000000013FF000000000000017B00000000000000A9000000000000005900000000000000090000000000000013787878] (max 8)
  key 00000000000013FF000000000000017B00000000000000A9000000000000003F78787878787878 seq 0 type 1
  key 00000000000013FF000000000000017B00000000000000A9000000000000005900000000000000090000000000000013787878 seq 0 type 1
[3] 00000000000013FF000000000000017B00000000000000A9000000000000003F78787878787878 => 7F8B01007B7A797877767574737271706F6E6D6C6B6A69686766656463626160
Internal keys in CF '3', [00000000000013FF000000000000017B00000000000000A9000000000000003F78787878787878, 00000000000013FF000000000000017B00000000000000A9000000000000005900000000000000090000000000000013787878] (max 8)
  key 00000000000013FF000000000000017B00000000000000A9000000000000003F78787878787878 seq 0 type 1
  key 00000000000013FF000000000000017B00000000000000A9000000000000005900000000000000090000000000000013787878 seq 0 type 1
[4] 00000000000013FF000000000000017B00000000000000A9000000000000003F78787878787878 => 7F8B01007B7A797877767574737271706F6E6D6C6B6A69686766656463626160
Internal keys in CF '4', [00000000000013FF000000000000017B00000000000000A9000000000000003F78787878787878, 00000000000013FF000000000000017B00000000000000A9000000000000005900000000000000090000000000000013787878] (max 8)
  key 00000000000013FF000000000000017B00000000000000A9000000000000003F78787878787878 seq 2097865 type 1
  key 00000000000013FF000000000000017B00000000000000A9000000000000005900000000000000090000000000000013787878 seq 702935 type 1
[6] 00000000000013FF000000000000017B00000000000000A9000000000000003F78787878787878 => 7F8B01007B7A797877767574737271706F6E6D6C6B6A69686766656463626160
Internal keys in CF '6', [00000000000013FF000000000000017B00000000000000A9000000000000003F78787878787878, 00000000000013FF000000000000017B00000000000000A9000000000000005900000000000000090000000000000013787878] (max 8)
  key 00000000000013FF000000000000017B00000000000000A9000000000000003F78787878787878 seq 0 type 1
  key 00000000000013FF000000000000017B00000000000000A9000000000000005900000000000000090000000000000013787878 seq 0 type 1
[8] 00000000000013FF000000000000017B00000000000000A9000000000000003F78787878787878 => 7F8B01007B7A797877767574737271706F6E6D6C6B6A69686766656463626160
Internal keys in CF '8', [00000000000013FF000000000000017B00000000000000A9000000000000003F78787878787878, 00000000000013FF000000000000017B00000000000000A9000000000000005900000000000000090000000000000013787878] (max 8)
  key 00000000000013FF000000000000017B00000000000000A9000000000000003F78787878787878 seq 2097869 type 1
  key 00000000000013FF000000000000017B00000000000000A9000000000000005900000000000000090000000000000013787878 seq 702939 type 1
Crash-recovery verification failed :(
No writes or ops?
Verification failed :(

(all of the examples are taken from here).

This is happening only when running with the new memtable, which was merged as part of #22, and enabled in crash tests with #89 + #93.

To Reproduce

make crash_test

Expected behavior

Crash test succeeds.

System

  • OS: Ubuntu 20.04
  • Compiler and version: GCC 9.4.0

Additional context

N/A

Metadata

Metadata

Labels

bugSomething isn't working

Type

No type
No fields configured for issues without a type.

Projects

No projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions