[Enhancement]: L0 Delta - DataNode support write L0 segment

[congqixia]

Is there an existing issue for this?

• I have searched the existing issues

What would you like to be added?

Refactor datanode to support L0 segment sync

• Refine datanode component
  • Replace all grpc operation with broker
    • Add Broker Add broker for datanode grpc operations #27631
    • Replace usage Replace manual composed grpc call with Broker methods in DataNode #27676
  • Refine meta cache Add meta cache to datanode for L0 Delta #27768 Refine datanode metacache and implement CoW #27985
  • Move buffer from Segment struct to buffer manager (with delta policy) Add WriteBuffer to provided absbraction for delta policy #27874
  • Refine flush manager Add SyncManager to replace flush manager #27873
  • Use write node, writeBuffer & sync Manager Use writebuffer, sync manager refactory in datanode #28320
• Enable Level zero delta policy
  • Sync policy adapts LevelZero segment enhance: fix LevelZero segment sync logic #28482
• Metrics
  • datacoord enhance: add l0 metric and fix datacoord no need drop l0 segment when flush #28373
  • datanode
• Code cleanup, Improvement & fixup logics
  • Comment out code & naming fix enhance: Remove commented code and fix naming issue #28450
  • Add compacted segment cleanup logic back enhance: Add back clean compacted segment info logic #28646
  • Output earliest buffer information enhance: Log channel checkpoint source info in writebuffer #28993
  • Separate storage/storagev2 into different abstraction from syncmgr enhance: separate serializer logic from sync task  #29413
  • Add memory watermark sync policy
  • Fill entry numbers & timestamp range fix: Forget to set EntriesNum for deltalogs #28858 enhance: Write buffer time range when syncing logs #28970
  • Allocate logID at most only once for each sync task enhance: reduce SyncTask AllocID call and refine code #29701
• Bugfix
  • Merged statslog may not generate if flush sync task has no insert data fix: Refine sync task field binlog compose logic #28494
  • Fix l0 segment deadlock & field missing fix: datacoord l0 segment bugs #28490
  • Fix unstable syncmgr TestCompact unit test fix: syncmgr unstable TestCompact unittest logic #28630
  • Flush hang & datacoord panic with zero-length compacted result [Bug]: Flush hang and mixcoord panic: runtime error: index out of range [0] with length 0 #28628
  • Standalone or datanode panic due to SegmentNotFound error
    • issue [Bug]: Datanode panic: segment not found and search failed: context canceled with timeout 10min #28748 [Bug]: Standalone panic: segment not found[segment=xxx] [recovered] and search error Timestamp lag too large #28736
    • Fix fix: Update segment compactTo when compactTo segment is compacted #28755
  • Level zero segments synced as non-flushed state fix: always sync level zero segments as flushed #29569
  • Explicitly pass LevelZero segment ids in vchan info enhance: Explicitly pass LevelZero segment ids in vchan info #29612
  • serializer shall bypass L0 segment merge stats step fix: serializer shall bypass L0 segment merge stats step #29636

Why is this needed?

See also #27349

Anything else?

No response

[congqixia]

The old write flow is

After the refactor of datanode, the component structure and data flow shall be:

[congqixia]

With multiple implementation of write buffer, it could be controlled by configuration which delta policy shall datanode use to buffer delta data:
BF Write Buffer


L0 Write Buffer

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Rotten issues close after 30d of inactivity. Reopen the issue with /reopen.