db_bench crashes with segmentation fault

[stephenpmorgan]

Running db_bench with fillseq benchmark and large number of keys (4.5 billion) caused segmentation error after more than 1.959 billion keys were generated but well before device was filled. Using code as of yesterday afternoon.

Command was:
bpl=1048576;overlap=10;mcz=2;del=300000000;levels=6;ctrig=4; delay=8; stop=12; wbn=3; mbc=6; mb=67108864;wbs=134217728; dds=0; sync=0; r=4500000000; t=1; vs=800; bs=65536; cs=1048576; of=500000; si=1000000; ./db_bench --benchmarks=fillseq --disable_seek_compaction=1 --mmap_read=0 --statistics=1 --histogram=1 --num=$r --threads=$t --value_size=$vs --block_size=$bs --cache_size=$cs --bloom_bits=10 --cache_numshardbits=4 --open_files=$of --verify_checksum=1 --db=/media/stephen/f4098456-97de-4569-852d-d3eceb1e7a82/database --sync=$sync --disable_wal=1 --compression_type=zlib --stats_interval=$si --compression_ratio=0.50 --disable_data_sync=$dds --write_buffer_size=$wbs --target_file_size_base=$mb --max_write_buffer_number=$wbn --max_background_compactions=$mbc --level0_file_num_compaction_trigger=$ctrig --level0_slowdown_writes_trigger=$delay --level0_stop_writes_trigger=$stop --num_levels=$levels --delete_obsolete_files_period_micros=$del --min_level_to_compress=$mcz --max_grandparent_overlap_factor=$overlap --stats_per_interval=1 --max_bytes_for_level_base=$bpl --use_existing_db=0

Crash resulted in:
Received signal 11 (Segmentation fault)
#0 ./db_bench() [0x47b039] GetVarint32Ptr /home/stephen/Downloads/rocksdb/rocksdb-master/./util/coding.h:109
#1 ./db_bench() [0x41e7e9] rocksdb::BuildTable(std::string const&, rocksdb::Env_, rocksdb::ImmutableCFOptions const&, rocksdb::EnvOptions const&, rocksdb::TableCache_, rocksdb::Iterator_, rocksdb::FileMetaData_, rocksdb::InternalKeyComparator const&, unsigned long, unsigned long, ocksdb::CompressionType, rocksdb::CompressionOptions const&, rocksdb::Env::IOPriority) /home/stephen/Downloads/rocksdb/rocksdb-master/db/builder.cc:100
#2 ./db_bench() [0x444637] rocksdb::Status::operator=(rocksdb::Status const&) /home/stephen/Downloads/rocksdb/rocksdb-master/./include/rocksdb/status.h:145 (discriminator 1)
#3 ./db_bench() [0x444bed] rocksdb::Status::ok() const /home/stephen/Downloads/rocksdb/rocksdb-master/db/db_impl.cc:4899
#4 ./db_bench() [0x44576e] rocksdb::Status::operator=(rocksdb::Status const&) /home/stephen/Downloads/rocksdb/rocksdb-master/./include/rocksdb/status.h:145 (discriminator 1)
#5 ./db_bench() [0x4532fb] rocksdb::DBImpl::BackgroundCallFlush() /home/stephen/Downloads/rocksdb/rocksdb-master/db/db_impl.cc:2101 (discriminator 1)
#6 ./db_bench() [0x4e2024] BGThread /home/stephen/Downloads/rocksdb/rocksdb-master/util/env_posix.cc:1616
#7 /lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a) [0x7f9b6b48ee9a] ?? ??:0
#8 /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f9b69f543fd] ?? ??:0

./Test2: line 2: 6723 Segmentation fault (core dumped) ./db_bench --benchmarks=fillseq --disable_seek_compaction=1 --mmap_read=0 --statistics=1 --histogram=1 --num=$r --threads=$t --value_size=$vs --block_size=$bs --cache_size=$cs --bloom_bits=10 --cache_numshardbits=4 --open_files=$of --verify_checksum=1 --db=/media/stephen/f4098456-97de-4569-852d-d3eceb1e7a82/database --sync=$sync --disable_wal=1 --compression_type=zlib --stats_interval=$si --compression_ratio=0.50 --disable_data_sync=$dds --write_buffer_size=$wbs --target_file_size_base=$mb --max_write_buffer_number=$wbn --max_background_compactions=$mbc --level0_file_num_compaction_trigger=$ctrig --level0_slowdown_writes_trigger=$delay --level0_stop_writes_trigger=$stop --num_levels=$levels --delete_obsolete_files_period_micros=$del --min_level_to_compress=$mcz --max_grandparent_overlap_factor=$overlap --stats_per_interval=1 --max_bytes_for_level_base=$bpl --use_existing_db=0
Command exited with non-zero status 139

Last few lines of trace files were:
** Compaction Stats [default] **
Level Files Size(MB) Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) RW-Amp W-Amp Rd(MB/s) Wr(MB/s) Rn(cnt) Rnp1(cnt) Wnp1(cnt) Wnew(cnt) Comp(sec) Comp(cnt) Avg(sec) Stall(sec) Stall(cnt) Avg(ms) RecordIn RecordDrop

L0 5/1 608 1.0 0.0 0.0 0.0 1502.4 1502.4 0.0 0.0 0.0 38.5 0 0 0 0 39992 12657 3.160 0.04 37297 0.00 0 0
L2 1/1 122 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0 0 0 0 0.000 0.00 0 0.00 0 0
L4 8/0 972 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0 0 0 0 0.000 0.00 0 0.00 0 0
L5 12643/0 1536767 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0 0 0 0 0.000 0.00 0 0.00 0 0
Sum 12657/2 1538468 0.0 0.0 0.0 0.0 1502.4 1502.4 1.0 1.0 0.0 38.5 0 0 0 0 39992 12657 3.160 0.04 37297 0.00 0 0
Int 0/0 0 0.0 0.0 0.0 0.0 0.7 0.7 1.0 1.0 0.0 33.5 0 0 0 0 22 6 3.632 0.00 13 0.00 0 0
Flush(GB): accumulative 1502.410, interval 0.712
Stalls(secs): 0.000 level0_slowdown, 0.000 level0_numfiles, 0.037 memtable_compaction, 0.000 leveln_slowdown_soft, 0.000 leveln_slowdown_hard
Stalls(count): 0 level0_slowdown, 0 level0_numfiles, 37297 memtable_compaction, 0 leveln_slowdown_soft, 0 leveln_slowdown_hard

** DB Stats **
Uptime(secs): 14513.4 total, 7.9 interval
Cumulative writes: 1959000000 writes, 1959000000 batches, 1.0 writes per batch, 1517.95 GB user ingest
Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, 0.00 GB written
Interval writes: 1000000 writes, 1000000 batches, 1.0 writes per batch, 793.5 MB user ingest
Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, 0.00 MB written

df ouput showed:
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/sda1 476770320 250691264 201860484 56% /
udev 8062516 4 8062512 1% /dev
tmpfs 3251452 840 3250612 1% /run
none 5120 0 5120 0% /run/lock
none 8128624 156 8128468 1% /run/shm
none 102400 44 102356 1% /run/user
cgroup 8128624 0 8128624 0% /sys/fs/cgroup
/dev/sdb 3845710688 1576027268 2074332492 44% /media/stephen/f4098456-97de-4569-852d-d3eceb1e7a82

[iamjinlei]

@stephenpmorgan this is pretty bad if it is a real issue. Would you be able to reproduce the problem?

[stephenpmorgan]

Sure, what would you like me to do? Simply re-run it? It takes a few hours.

[igorcanadi]

I'll also try reproducing

[igorcanadi]

I got "put error: IO error: 006936.sst: No space left on device" after a billion writes :)

[igorcanadi]

@stephenpmorgan would you mind trying this out with RocksDB 2.4?

[stephenpmorgan]

I reran it with RocksDB version 3.6 (at least that's what it says in the output trace). It hung after 2.484 billion keys, No progress, no segmentation fault, no further output, just hanging there... I would be happy to try out RocksDB 2.4 if I knew how to do so. How do I do that? I don't really know how to use GitHub to extract 2.4.

[igorcanadi]

I tried reproducing 3 times, no luck. :(

Sorry, I meant RocksDB 3.4. Here is 3.4 release: https://github.com/facebook/rocksdb/releases/tag/rocksdb-3.4

Did you check the stack trace when the process was hanging? You can do that by attaching to the process using gdb and then issuing thread all apply bt

[stephenpmorgan]

(gdb) thread all apply bt
No symbol "all" in current context.
(gdb) bt
#0 0x00007fbd0db01d84 in pthread_cond_wait@@GLIBC_2.3.2 ()
from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00000000005503bd in rocksdb::port::CondVar::Wait (this=)
at port/port_posix.cc:83
#2 0x000000000040da41 in rocksdb::Benchmark::RunBenchmark (
this=this@entry=0x7fff6d00fac0, n=n@entry=1, name=..., method=
(void (rocksdb::Benchmark::_)(rocksdb::Benchmark * const, rocksdb::ThreadState *)) 0x419910 rocksdb::Benchmark::WriteSeq(rocksdb::ThreadState_))
at db/db_bench.cc:1479
#3 0x000000000041b567 in rocksdb::Benchmark::Run (
this=this@entry=0x7fff6d00fac0) at db/db_bench.cc:1390
#4 0x00000000004070cd in main (argc=1, argv=0x7fff6d00fdc0)
at db/db_bench.cc:2916

[igorcanadi]

Sorry, thread apply all by

[stephenpmorgan]

(gdb) thread apply all bt

Thread 9 (Thread 0x7fbd0c4ce700 (LWP 3405)):
#0 0x00007fbd0db01d84 in pthread_cond_wait@@GLIBC_2.3.2 ()
from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00000000004e1f14 in BGThread (thread_id=0, this=0x1137bc0)
at util/env_posix.cc:1560
#2 rocksdb::(anonymous namespace)::PosixEnv::ThreadPool::BGThreadWrapper (
arg=) at util/env_posix.cc:1633
#3 0x00007fbd0dafde9a in start_thread ()
from /lib/x86_64-linux-gnu/libpthread.so.0
#4 0x00007fbd0c5c33fd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#5 0x0000000000000000 in ?? ()

Thread 8 (Thread 0x7fbd0bccd700 (LWP 3406)):
#0 0x00007fbd0db01d84 in pthread_cond_wait@@GLIBC_2.3.2 ()
from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00000000004e1f14 in BGThread (thread_id=1, this=0x1137bc0)
at util/env_posix.cc:1560
#2 rocksdb::(anonymous namespace)::PosixEnv::ThreadPool::BGThreadWrapper (
arg=) at util/env_posix.cc:1633
#3 0x00007fbd0dafde9a in start_thread ()
from /lib/x86_64-linux-gnu/libpthread.so.0
#4 0x00007fbd0c5c33fd in clone () from /lib/x86_64-linux-gnu/libc.so.6
---Type to continue, or q to quit---
#5 0x0000000000000000 in ?? ()

Thread 7 (Thread 0x7fbd0b4cc700 (LWP 3407)):
#0 0x00007fbd0db01d84 in pthread_cond_wait@@GLIBC_2.3.2 ()
from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00000000004e1f14 in BGThread (thread_id=2, this=0x1137bc0)
at util/env_posix.cc:1560
#2 rocksdb::(anonymous namespace)::PosixEnv::ThreadPool::BGThreadWrapper (
arg=) at util/env_posix.cc:1633
#3 0x00007fbd0dafde9a in start_thread ()
from /lib/x86_64-linux-gnu/libpthread.so.0
#4 0x00007fbd0c5c33fd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#5 0x0000000000000000 in ?? ()

Thread 6 (Thread 0x7fbd0accb700 (LWP 3408)):
#0 0x00007fbd0db01d84 in pthread_cond_wait@@GLIBC_2.3.2 ()
from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00000000004e1f14 in BGThread (thread_id=3, this=0x1137bc0)
at util/env_posix.cc:1560
#2 rocksdb::(anonymous namespace)::PosixEnv::ThreadPool::BGThreadWrapper (
arg=) at util/env_posix.cc:1633
#3 0x00007fbd0dafde9a in start_thread ()
from /lib/x86_64-linux-gnu/libpthread.so.0
---Type to continue, or q to quit---
#4 0x00007fbd0c5c33fd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#5 0x0000000000000000 in ?? ()

Thread 5 (Thread 0x7fbd0a4ca700 (LWP 3409)):
#0 0x00007fbd0db01d84 in pthread_cond_wait@@GLIBC_2.3.2 ()
from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00000000004e1f14 in BGThread (thread_id=4, this=0x1137bc0)
at util/env_posix.cc:1560
#2 rocksdb::(anonymous namespace)::PosixEnv::ThreadPool::BGThreadWrapper (
arg=) at util/env_posix.cc:1633
#3 0x00007fbd0dafde9a in start_thread ()
from /lib/x86_64-linux-gnu/libpthread.so.0
#4 0x00007fbd0c5c33fd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#5 0x0000000000000000 in ?? ()

Thread 4 (Thread 0x7fbd09cc9700 (LWP 3410)):
#0 0x00007fbd0db01d84 in pthread_cond_wait@@GLIBC_2.3.2 ()
from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00000000004e1f14 in BGThread (thread_id=5, this=0x1137bc0)
at util/env_posix.cc:1560
#2 rocksdb::(anonymous namespace)::PosixEnv::ThreadPool::BGThreadWrapper (
arg=) at util/env_posix.cc:1633
#3 0x00007fbd0dafde9a in start_thread ()
---Type to continue, or q to quit---
from /lib/x86_64-linux-gnu/libpthread.so.0
#4 0x00007fbd0c5c33fd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#5 0x0000000000000000 in ?? ()

Thread 3 (Thread 0x7fbd094c8700 (LWP 3411)):
#0 0x00007fbd0db01d84 in pthread_cond_wait@@GLIBC_2.3.2 ()
from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00000000005503bd in rocksdb::port::CondVar::Wait (this=)
at port/port_posix.cc:83
#2 0x000000000043d408 in rocksdb::DBImpl::DelayWrite (
this=this@entry=0x1189e90, expiration_time=expiration_time@entry=0)
at db/db_impl.cc:4171
#3 0x000000000044fb88 in rocksdb::DBImpl::Write (this=0x1189e90, options=...,
my_batch=) at db/db_impl.cc:4039
#4 0x0000000000418cb4 in rocksdb::Benchmark::DoWrite (this=0x7fff6d00fac0,
thread=0x1192ee0, write_mode=) at db/db_bench.cc:2047
#5 0x000000000040c95a in rocksdb::Benchmark::ThreadBody (v=0x11968c0)
at db/db_bench.cc:1423
#6 0x00000000004e0c42 in rocksdb::(anonymous namespace)::StartThreadWrapper (
arg=0x118a5c0) at util/env_posix.cc:1757
#7 0x00007fbd0dafde9a in start_thread ()
from /lib/x86_64-linux-gnu/libpthread.so.0
#8 0x00007fbd0c5c33fd in clone () from /lib/x86_64-linux-gnu/libc.so.6
---Type to continue, or q to quit---
#9 0x0000000000000000 in ?? ()

Thread 2 (Thread 0x7fbd08ac7700 (LWP 3412)):
#0 0x00007fbd0db01d84 in pthread_cond_wait@@GLIBC_2.3.2 ()
from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00000000004e1f14 in BGThread (thread_id=0, this=0x1137c90)
at util/env_posix.cc:1560
#2 rocksdb::(anonymous namespace)::PosixEnv::ThreadPool::BGThreadWrapper (
arg=) at util/env_posix.cc:1633
#3 0x00007fbd0dafde9a in start_thread ()
from /lib/x86_64-linux-gnu/libpthread.so.0
#4 0x00007fbd0c5c33fd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#5 0x0000000000000000 in ?? ()

Thread 1 (Thread 0x7fbd0df14800 (LWP 3404)):
#0 0x00007fbd0db01d84 in pthread_cond_wait@@GLIBC_2.3.2 ()
from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00000000005503bd in rocksdb::port::CondVar::Wait (this=)
at port/port_posix.cc:83
#2 0x000000000040da41 in rocksdb::Benchmark::RunBenchmark (
this=this@entry=0x7fff6d00fac0, n=n@entry=1, name=..., method=
(void (rocksdb::Benchmark::_)(rocksdb::Benchmark * const, rocksdb::ThreadState *)) 0x419910 rocksdb::Benchmark::WriteSeq(rocksdb::ThreadState_))
---Type to continue, or q to quit---
at db/db_bench.cc:1479
#3 0x000000000041b567 in rocksdb::Benchmark::Run (
this=this@entry=0x7fff6d00fac0) at db/db_bench.cc:1390
#4 0x00000000004070cd in main (argc=1, argv=0x7fff6d00fdc0)
at db/db_bench.cc:2916
(gdb)

[igorcanadi]

Huh. This does not look good. Wonder why I can't repro. Can you also share your LOG as well?

[stephenpmorgan]

Are you sure? The log is enormous. How would you like me to share it?

[stephenpmorgan]

Here are the last 50 lines of the log:

Cumulative writes: 2482000000 writes, 2482000000 batches, 1.0 writes per batch, 1923.20 GB user ingest
Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, 0.00 GB written
Interval writes: 1000000 writes, 1000000 batches, 1.0 writes per batch, 793.5 MB user ingest
Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, 0.00 MB written

2014/10/09-22:22:24 ... thread 0: (1000000,2483000000) ops and (135188.1,143703.6) ops/second in (7.397102,17278.626286) seconds

** Compaction Stats [default] **

Level Files Size(MB) Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) RW-Amp W-Amp Rd(MB/s) Wr(MB/s) Rn(cnt) Rnp1(cnt) Wnp1(cnt) Wnew(cnt) Comp(sec) Comp(cnt) Avg(sec) Stall(sec) Stall(cnt) Avg(ms) RecordIn RecordDrop

L0 5/1 608 1.0 0.0 0.0 0.0 1904.3 1904.3 0.0 0.0 0.0 41.5 0 0 0 0 46975 16043 2.928 0.05 51680 0.00 0 0
L2 1/1 122 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0 0 0 0 0.000 0.00 0 0.00 0 0
L3 1/1 122 1.2 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0 0 0 0 0.000 0.00 0 0.00 0 0
L4 8/0 972 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0 0 0 0 0.000 0.00 0 0.00 0 0
L5 16028/0 1948151 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0 0 0 0 0.000 0.00 0 0.00 0 0
Sum 16043/3 1949974 0.0 0.0 0.0 0.0 1904.3 1904.3 1.0 1.0 0.0 41.5 0 0 0 0 46975 16043 2.928 0.05 51680 0.00 0 0
Int 0/0 0 0.0 0.0 0.0 0.0 0.7 0.7 1.0 1.0 0.0 36.4 0 0 0 0 20 6 3.336 0.00 18 0.00 0 0
Flush(GB): accumulative 1904.271, interval 0.712
Stalls(secs): 0.000 level0_slowdown, 0.000 level0_numfiles, 0.052 memtable_compaction, 0.000 leveln_slowdown_soft, 0.000 leveln_slowdown_hard
Stalls(count): 0 level0_slowdown, 0 level0_numfiles, 51680 memtable_compaction, 0 leveln_slowdown_soft, 0 leveln_slowdown_hard

** DB Stats **
Uptime(secs): 17278.6 total, 7.4 interval
Cumulative writes: 2483000000 writes, 2483000000 batches, 1.0 writes per batch, 1923.98 GB user ingest
Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, 0.00 GB written
Interval writes: 1000000 writes, 1000000 batches, 1.0 writes per batch, 793.5 MB user ingest
Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, 0.00 MB written

2014/10/09-22:22:31 ... thread 0: (1000000,2484000000) ops and (143864.0,143703.6) ops/second in (6.951008,17285.577294) seconds

** Compaction Stats [default] **

Level Files Size(MB) Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) RW-Amp W-Amp Rd(MB/s) Wr(MB/s) Rn(cnt) Rnp1(cnt) Wnp1(cnt) Wnew(cnt) Comp(sec) Comp(cnt) Avg(sec) Stall(sec) Stall(cnt) Avg(ms) RecordIn RecordDrop

L0 6/1 729 1.2 0.0 0.0 0.0 1905.0 1905.0 0.0 0.0 0.0 41.5 0 0 0 0 46994 16049 2.928 0.05 51698 0.00 0 0
L3 1/1 122 1.2 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0 0 0 0 0.000 0.00 0 0.00 0 0
L4 8/0 972 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0 0 0 0 0.000 0.00 0 0.00 0 0
L5 16034/0 1948880 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0 0 0 0 0.000 0.00 0 0.00 0 0
Sum 16049/2 1950703 0.0 0.0 0.0 0.0 1905.0 1905.0 1.0 1.0 0.0 41.5 0 0 0 0 46994 16049 2.928 0.05 51698 0.00 0 0
Int 0/0 0 0.0 0.0 0.0 0.0 0.7 0.7 1.0 1.0 0.0 38.4 0 0 0 0 19 6 3.165 0.00 18 0.00 0 0
Flush(GB): accumulative 1904.984, interval 0.712
Stalls(secs): 0.000 level0_slowdown, 0.000 level0_numfiles, 0.052 memtable_compaction, 0.000 leveln_slowdown_soft, 0.000 leveln_slowdown_hard
Stalls(count): 0 level0_slowdown, 0 level0_numfiles, 51698 memtable_compaction, 0 leveln_slowdown_soft, 0 leveln_slowdown_hard

** DB Stats **
Uptime(secs): 17285.6 total, 7.0 interval
Cumulative writes: 2484000000 writes, 2484000000 batches, 1.0 writes per batch, 1924.75 GB user ingest
Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, 0.00 GB written
Interval writes: 1000000 writes, 1000000 batches, 1.0 writes per batch, 793.5 MB user ingest
Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, 0.00 MB written

[igorcanadi]

@stephenpmorgan Can you zip it and upload it somewhere?

[iamjinlei]

The config has write_buffer_size = 128MB, target_file_size = 64MB, compression=zlib, ratio=0.5, but the LOG seems to show that average file size is ~128MB. I may misunderstand something, but it seems fishy to me.