-
Notifications
You must be signed in to change notification settings - Fork 177
Mv filecache tuning2
- developer tested code checked-in May 17, 2013
- development started May 14, 2013
The mv-level-work1 branch changed levels 1 and 2 from sorted files to overlapped files. This change allows writes to occur more quickly and almost halves the write amplification. The overlapped strategy takes all files from level 0 and creates one file at level 1 during a compaction. Similarly all files of from level 1 become one file at level 2. Level 2 files are "large". They compact across all the sorted files of level 3.
The above process works fine when fewer files exist in levels 0 through 3 than the max_open_files limit. Many of the lower level files are likely close during the level 2 to 3 compaction if max_open_files is small. Reopening the larger, overlapped level files is CPU and disk intensive ... and these are files that are most likely to be read.
mv-level-work1 branch continued a previous concept of creating larger and larger files as the levels increased. This concept was based upon write performance observations. Recent analysis shows that larger files really hurt random reads and iterators/iterations.
One quick concept was to leave the overlapped files of levels 0, 1, and 2 large. Then make the sort files of levels 3 and above smaller. Reducing the size of the sorted files in level 3 and beyond however increased the chances that the lower levels' larger files would be closed and later require the expensive reopen.