[core] Support MULTI_WRITE mode for KeyValueFileStoreWrite when multiple jobs are writing to one table simultaneously… #5301
Conversation
| <td><h5>write.mode</h5></td> | ||
| <td style="word-wrap: break-word;">SINGLE_WRITE</td> | ||
| <td><p>Enum</p></td> | ||
| <td>Specify the write mode for table. When multiple jobs are writing to one table, the MULTI_WRITE mode should be used for the write-and-compact job. In this case, the CompactManager can scan files generated by other users. Notice: This option does not work for dedicated compaction job.<br /><br />Possible values:<ul><li>"SINGLE_WRITE"</li><li>"MULTI_WRITE"</li></ul></td> |
There was a problem hiding this comment.
Based on my understanding of this PR ：
1: Specify the write mode for table. -> Specifies the write mode for the current write job.
2: the MULTI_WRITE mode should be used for the write-and-compact job that wirte-only=false and other write jobs should be write-only=true.
|
|
||
| /** Multiple writes are creating new files for current table. */ | ||
| MULTI_WRITE | ||
| } |
There was a problem hiding this comment.
Suggestion ： SINGLE_WRITER / MULTI_WRITER will be better.
|
|
||
| KeyValueFileStoreWrite write1 = (KeyValueFileStoreWrite) store1.newWrite(user1); | ||
| KeyValueFileStoreWrite write2 = (KeyValueFileStoreWrite) store2.newWrite(user2); | ||
|
|
There was a problem hiding this comment.
I tried adding a writer-3 in this test, the result is not as expected, the file written by the writer-1 is not be compacted. Please correct me if I understand wrongly.
There was a problem hiding this comment.
writer-1 is the write and compact job, writer-3 should be write only. The file added by writer-3 should be compacted by writer-1.
There was a problem hiding this comment.
Sorry, my expression is wrong. the file written by the writer-2 is not be compacted. write-1 and write-3 will be compacted.
There was a problem hiding this comment.
@LinMingQiang There is a debug in the previous implementable. Pls check the latest version. Also I've added the writer-3 in the test.
…ple jobs are writing to one table simultaneously
|
@LinMingQiang I've discussed with @JingsongLi about this feature. We need to avoid continuously file scan in writer. So this pr will only be a PoC temporarily. |
Purpose
Linked issue: close #5214
Tests
MergeTreeCompactManagerTest#testSyncOrphanFiles()
KeyValueFileStoreWriteTest#testMultiWriteModeEnabled()
KeyValueFileStoreWriteTest#testMultiWriteModeDisabled()
API and Format
Documentation