Data file size are different between nodes

[zeinali0]

Why the tserver data size are different between nodes(logs size not counted)?
I have a 3 node universe with rf 3 but the size of data between nodes are too much different:

2 tserver are using about 10GB (number 1 & 3) while one of them using about 1.5GB(number 3)

while yb-admin UI is showing tablets are balanced between nodes

[bmatican]

cc @amitanandaiyer , maybe similar to #3666 ?

@zeinali0 Are your disks xfs? Also which version of yugabyte are you using?

[kmuthukk]

From the screenshot there's ~1200 tablets on each node. So ~1200 MANIFEST* files.

So, like @bmatican suspects, if it is the issue #3666 (which has recently been fixed) -- that could explain about 1200*4MB = 4.8GB overhead on nodes where the yb-tserver was restarted (because of the unintended 4MB pre-allocation/fallocate call). This is a one-time cost per tablet in older release - and will be avoided in releases that include the fix for #3666.

@zeinali0 -- can you look for some MANIFEST* files inside the data directory and compare the size reported by du -hs <filename> vs. ls -lh <filename> to see if it is a duplicate of #3666?

Thx!

[zeinali0]

i'm using version 2.1.0.0.
i reset another node tserver and now it is using about 1.5GB.

[zeinali0]

can you look for some MANIFEST* files inside the data directory and compare the size reported by "du -hs <filename>" vs. "ls -lh <filename>" to see if it is a duplicate of #3666?

Yes it is duplicate. 66 byte file size and 4.0 MB on disk
But from which release does it fixed? i'm using 2.1.0.0

[zeinali0]

it increased to 10GB again

[ddorian]

Looks like the patch is in 2.1.2

yugabyte-db/src/yb/rocksdb/util/options.cc

Line 264 in 50aaf8d

manifest_preallocation_size(64 * 1024),

I tried locally and reports to 64KB. Can you retry @zeinali0 ?

[kmuthukk]

Thx for confirming @zeinali0 that it indeed is a duplicate of #3666.

Like @ddorian mentioned, this issue is fixed in 2.1.2 release. Please see release notes here:

https://github.com/yugabyte/yugabyte-db/releases/tag/v2.1.2

However, please note that the fix will not apply to tables already created. It will only apply to new tables created after the upgrade.

[zeinali0]

The issue fixed after upgrading to 2.1.2.0.