[Multi-stage] Optimize query plan serialization

[Jackie-Jiang]

Optimize the query plan ser-de by:

• Only sending the worker metadata that will be executed. Currently all worker metadata are sent to every server
• Pre-serialize the proto object to bytes for the shared objects (e.g. plan node)

Backward Incompatible

This PR changes the proto object structure, which will cause backward incompatibility when broker and server are running different version. Since we are not maintaining backward compatibility for multi-stage engine as of now, user should upgrade both brokers and servers, then start querying.

[xiangfu0]

this is not used.

[Jackie-Jiang]

This is used to ser/de the map

[codecov-commenter]

Codecov Report

Attention: 23 lines in your changes are missing coverage. Please review.

Comparison is base (0761a65) 61.71% compared to head (af2c506) 61.69%.

Files	Patch %	Lines
...apache/pinot/query/service/server/QueryServer.java	59.09%	9 Missing ⚠️
...ery/planner/physical/MailboxAssignmentVisitor.java	80.95%	1 Missing and 3 partials ⚠️
...va/org/apache/pinot/query/runtime/QueryRunner.java	80.95%	4 Missing ⚠️
...org/apache/pinot/query/routing/WorkerMetadata.java	85.00%	3 Missing ⚠️
...rg/apache/pinot/query/routing/MailboxMetadata.java	87.50%	1 Missing ⚠️
...y/runtime/operator/BaseMailboxReceiveOperator.java	50.00%	0 Missing and 1 partial ⚠️
...ry/runtime/plan/server/ServerPlanRequestUtils.java	95.00%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##             master   #12370      +/-   ##
============================================
- Coverage     61.71%   61.69%   -0.03%     
  Complexity      207      207              
============================================
  Files          2426     2425       -1     
  Lines        132776   132707      -69     
  Branches      20541    20529      -12     
============================================
- Hits          81946    81871      -75     
- Misses        44817    44823       +6     
  Partials       6013     6013              

Flag	Coverage Δ
custom-integration1	<0.01% <0.00%> (ø)
integration	<0.01% <0.00%> (ø)
integration1	<0.01% <0.00%> (ø)
integration2	0.00% <0.00%> (ø)
java-11	61.65% <89.30%> (-0.02%)	⬇️
java-21	61.57% <89.30%> (-0.03%)	⬇️
skip-bytebuffers-false	61.68% <89.30%> (-0.03%)	⬇️
skip-bytebuffers-true	61.55% <89.30%> (+26.69%)	⬆️
temurin	61.69% <89.30%> (-0.03%)	⬇️
unittests	61.68% <89.30%> (-0.03%)	⬇️
unittests1	46.85% <89.30%> (-0.05%)	⬇️
unittests2	27.73% <0.00%> (+0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[walterddr]

lgtm. only reviewed the non-test portion.

[walterddr]

lol we should do license check better

[walterddr]

seems like all auto-formatting? (skipping most of these kinds)

[walterddr]

do we still need stage info . I saw it got added during #12358 then removed in #12363 ?

[Jackie-Jiang]

This is a simple helper class to help organize the info for each stage.

[walterddr]

was this change related to Ser/De or just test optimization?

[Jackie-Jiang]

Moving the per method setup to per class is test optimization