[multistage] fix usage of metadata overrides

[walterddr]

custom properties in StageMetadata and
requestMetadata from WorkerRequest are immutable

mutating it during execution causes race-condition in set b/c multiple worker shares the same metadata

thus

1. create an immutable wrapper around
2. move the consolidation logic to create a new local map of requestMetadata for each queryRunner call and set it context, never use the customProperties field from StageMetadata again.

[Jackie-Jiang]

Mostly good

[Jackie-Jiang]

I think we should put request level metadata first, then override the stage level custom property

[walterddr]

this changes the previous behavior. i will separate that into a different PR outside of this bugfix

[codecov-commenter]

Codecov Report

Merging #11587 (e1ed225) into master (3c7cc4f) will decrease coverage by 0.01%.
Report is 23 commits behind head on master.
The diff coverage is 85.71%.

❗ Current head e1ed225 differs from pull request most recent head 87c31b5. Consider uploading reports for the commit 87c31b5 to get more accurate results

@@             Coverage Diff              @@
##             master   #11587      +/-   ##
============================================
- Coverage     63.04%   63.03%   -0.01%     
+ Complexity     1109     1105       -4     
============================================
  Files          2320     2326       +6     
  Lines        124667   124884     +217     
  Branches      19033    19063      +30     
============================================
+ Hits          78600    78726     +126     
- Misses        40476    40549      +73     
- Partials       5591     5609      +18     

Flag	Coverage Δ
integration	<0.01% <0.00%> (-0.01%)	⬇️
integration1	<0.01% <0.00%> (-0.01%)	⬇️
integration2	0.00% <0.00%> (ø)
java-11	63.01% <85.71%> (+<0.01%)	⬆️
java-17	14.49% <0.00%> (-48.41%)	⬇️
java-20	14.48% <0.00%> (-48.43%)	⬇️
temurin	63.03% <85.71%> (-0.01%)	⬇️
unittests	63.03% <85.71%> (-0.01%)	⬇️
unittests1	67.42% <85.71%> (-0.04%)	⬇️
unittests2	14.49% <0.00%> (+<0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files Changed	Coverage Δ
...va/org/apache/pinot/query/runtime/QueryRunner.java	78.07% <82.75%> (+0.79%)	⬆️
...inot/query/runtime/operator/AggregateOperator.java	94.40% <100.00%> (-0.04%)	⬇️
...pinot/query/runtime/operator/HashJoinOperator.java	92.94% <100.00%> (-0.05%)	⬇️
...ot/query/runtime/plan/OpChainExecutionContext.java	96.15% <100.00%> (ø)
...apache/pinot/query/runtime/plan/StageMetadata.java	76.00% <100.00%> (ø)

... and 88 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more