[Enhancement]: Add pursuit mode for mqtt msgstream

[congqixia]

Is there an existing issue for this?

• I have searched the existing issues

What would you like to be added?

Add pursuit mode for mqtt msgstream

Why is this needed?

When system recover from some abnormal state, datanode & querynode may need to consume lots of msg
Using current policy is time consuming and cause system recovery tool slow.
The proposal is to add pursuit mode: merge msg packs from mqtt msgstream if the time lag is too large(lag threshold configurable). So flowgraphs could can handle these messages in larger batch and skip lots of useless processing.

Anything else?

No response

[congqixia]

For consuming 100 batch *10000 768 dim vectors, older implementation cost about 60 seconds while pursuit mode costs only 7 seconds

[xiaofan-luan]

/assign @chyezh
could you pleas help on review the implementation

[xiaofan-luan]

For consuming 100 batch *10000 768 dim vectors, older implementation cost about 60 seconds while pursuit mode costs only 7 seconds

what about the all timetick case?

[congqixia]

what about the all timetick case?

@xiaofan-luan with mock unit test case (mock datacoord time ticker as empty op), normal mode and pursuit mode all takes about 2 seconds to consume time tick msg of 1 hour (56060).
I guess in real world scenario, operations for time tick sender was the step that consume most of the time for empty time tick msg packs.

[congqixia]

@xiaofan-luan
also @yanliang567 is doing tests in test deployment to get baseline before this enhancement. FYI.

[yanliang567]

the baseline on master-20230802-df26b909 is
50m_768d entities(295 growing segments) was inserted during the datanode was scale down to 0. It costs 55 mins to catch up the tt lag with one datanode on, almost regardless of insert batch(1000, 2000 or 5000) or waiting time after insertion(1h, 4h or 10h).
now I am running tests on master-20230805-241117dd...

[xiaofan-luan]

looking forward to the result!
And the test way is actually very smart! I would expect 2-3 times performance improvement

[yanliang567]

for now it was blocked by #26177