Database size 20% larger than LevelDB

[karalabe]

This isn't really a bugreport, or rather may or may not be.

While doing our preliminary tests vs. LevelDB, we've noticed that Pebble's on-disk size is about 20% larger than that of go-leveldb. Wondering if this is an expected known fact / feature of Pebble, or if it might be a bug or a misconfiguration on our side? I'm still tweaking parameters, so the comparison might not be completely fair, just wondering if this size increase is something I should expect or if it's something I should debug.

Thanks

[petermattis]

That is unexpected. I can't think of any reason for that to be the case. The on-disk size should be similar. The sstable format is essentially the same. And both use snappy compression (unless you accidentally disabled it in Pebble).

[karalabe]

Thanks for the note I'll keep digging them. In this particular run I've inited Pebble via:

	db, err := pebble.Open(file, &pebble.Options{
		BytesPerSync: 32 * 1024 * 1024, // No idea what's a correct number here
		Cache:        pebble.NewCache(int64(cache * 1024 * 1024 / 2)),
		MaxOpenFiles: handles,
		MemTableSize: cache * 1024 * 1024 / 4,
		MaxConcurrentCompactions: runtime.NumCPU(),
	})

where cache is 512 2048 and handles something insane, 500K.

[petermattis]

If you're trying to do an apples-to-apples comparison, you should start with the same configuration you're using for go-leveldb. A 512MB memtable is quite large. The LevelDB/Pebble default is 4MB. CRDB uses 64MB.

Pebble subtracts space from the cache for memtables. So your 1GB cache will actually only be 512MB if there is a memtable, or 0 if there are 2 memtables (because 1 is being flushed and 1 is being written to).

You can leave off the BytesPerSync setting. The default of 512KB is fine.

I don't think go-leveldb supports concurrent compactions, so I'd start with leaving off MaxConcurrentCompactions. I doubt that is causing problems, but if you're trying to do a comparison that is something to check.

[karalabe]

Yes, that's why I said it's not completely fair currently. I'm mostly doing parameter exploratory tests, playing with file sizes, concurrency, whatnot and try to evaluate how Pebble behaves.

Pebble subtracts space from the cache for memtables.

Ah, that explains why the memory use was half of leveldb!

I don't think go-leveldb supports concurrent compactions, so I'd start with leaving off MaxConcurrentCompactions. I doubt that is causing problems, but if you're trying to do a comparison that is something to check.

Yup, that's on my list-to-test, just have a limited number of machines to screw with and the tests run for 1 or 7 days :P

[karalabe]

Unrelated to the issue, but curious chart wrt memtable sizes. Green is leveldb (512 MB memtables), yellow is Pebble (first hald 64MB, second half 512MB memtable). Please ignore the absolute values (ldb did a compaction after restart) and only look at the rate of growth.

[petermattis]

I'm not sure what I'm supposed to be noticing. Certainly, a smaller memtable size will cause data to be flushed to disk faster and thus can result in more disk reads. It would be helpful if you described what your application is doing. What is the mix of reads and writes? What is the data set size? Is the data set size increasing over time?

[karalabe]

I've been investigating this. Couldn't repro it via an external script, so I tried to take our production benchmark-run weighing about 160GB in LevelDB and 200GB in Pebble and dropped a compaction request on top of both. Pebble reduced its database size by a whopping 40GB (20%), reaching the same sizes as LevelDB.

Before compaction:

__level_____count____size___score______in__ingest(sz_cnt)____move(sz_cnt)___write(sz_cnt)____read___r-amp___w-amp
    WAL         1    64 M       -   284 G       -       -       -       -   284 G       -       -       -     1.0
      0         2   682 M    0.50   284 G     0 B       0     0 B       0   231 G     707     0 B       2     0.8
      1         0     0 B    0.00     0 B     0 B       0     0 B       0     0 B       0     0 B       0     0.0
      2        31    63 M    0.98   149 G     0 B       0     0 B       0   152 G    77 K   156 G       1     1.0
      3       236   468 M    1.00   136 G     0 B       0    83 G    42 K   183 G    94 K   188 G       1     1.4
      4      1868   3.4 G    1.00   171 G     0 B       0    51 G    26 K   454 G   249 K   461 G       1     2.7
      5     15534    25 G    1.00   190 G     0 B       0    22 G    12 K   678 G   370 K   686 G       1     3.6
      6     97844   172 G       -   177 G     0 B       0   1.9 G   1.0 K   522 G   295 K   530 G       1     2.9
  total    115515   202 G       -   284 G     0 B       0   158 G    81 K   2.4 T   1.1 M   2.0 T       7     8.8
  flush       707
compact    335158    14 G          (size == estimated-debt)
 memtbl         1   512 M
zmemtbl         0     0 B
   ztbl         0     0 B
 bcache     761 K   1.5 G   90.1%  (score == hit-rate)
 tcache     103 K    60 M  100.0%  (score == hit-rate)
 titers         0
 filter         -       -    0.0%  (score == utility)

After compaction:

__level_____count____size___score______in__ingest(sz_cnt)____move(sz_cnt)___write(sz_cnt)____read___r-amp___w-amp
    WAL         1    21 M       -   297 G       -       -       -       -   297 G       -       -       -     1.0
      0         1    39 M    0.00   297 G     0 B       0     0 B       0   241 G     920     0 B       1     0.8
      1         0     0 B    0.00     0 B     0 B       0     0 B       0     0 B       0     0 B       0     0.0
      2       831   1.6 G    6.63   160 G     0 B       0     0 B       0   445 G   226 K   449 G       1     2.8
      3       983   1.7 G    6.48   142 G     0 B       0    86 G    44 K   196 G   100 K   201 G       1     1.4
      4      1016   1.9 G    0.63   177 G     0 B       0    52 G    26 K   465 G   255 K   472 G       1     2.6
      5      5663   358 M    0.02   199 G     0 B       0    22 G    12 K   713 G   434 K   721 G       1     3.6
      6     76890   152 G       -   210 G     0 B       0   1.9 G   1.0 K   677 G   374 K   738 G       1     3.2
  total     85384   158 G       -   297 G     0 B       0   161 G    83 K   3.0 T   1.4 M   2.5 T       6    10.2
  flush       920
compact    340215    12 G          (size == estimated-debt)
 memtbl         1   512 M
zmemtbl         0     0 B
   ztbl         0     0 B
 bcache     806 K   1.5 G   89.2%  (score == hit-rate)
 tcache      79 K    46 M  100.0%  (score == hit-rate)
 titers         2
 filter         -       -    0.0%  (score == utility)

Based on this, the data usage of Pebble seems to indeed be in line with LevelDB, but I'm wondering how we could get it to compact it more aggressively to that it only uses as much space as needed. I wouldn't mind a bit of a blowup, but 20% is not something I can afford (our biggest db is around 6TB).

[petermattis]

The before and after compaction numbers are confusing to me. Before the compaction there were 2 L0 sstables and 31 L2 sstables. After compaction there was 1 L0 sstable and 831 L2 sstables. And the data size increased!

dropped a compaction request on top of both

What does that mean? You manually compacted the entire DB. If you did that I would expect the after to look quite different from what you've shown.

The LevelDB and Pebble compaction heuristics are quite different. I don't know off-hand what is causing the discrepancy in compaction aggressiveness on your workload. Hard for me to even speculate unless I could run the workload myself, though even if I could do that I don't have any spare time to do so.

[holiman]

@petermattis some more context and new benchmark results can be found here: ethereum/go-ethereum#21575 (comment)

[petermattis]

@holiman The issue you're pointing to has a lot of charts, yet I'm missing a ton of context as to what you're doing. It would be helpful if you could describe the load your system is performing. Even better if you could encode that description in a simple binary.

[github-actions]

We have marked this issue as stale because it has been inactive for
18 months. If this issue is still relevant, removing the stale label
or adding a comment will keep it active. Otherwise, we'll close it
in 10 days to keep the issue queue tidy. Thank you for your
contribution to Pebble!