WIP: Support delete in upserts

[KKcorps]

Use two new configs -

      "deleteRecordKey": "delete"

[Jackie-Jiang]

There is one corner case not handled. When a record is deleted, we might want to track its timestamp to guarantee consistency for late event. This could cause much bigger map size thus revoke the benefit of deletion. We can further discuss the tradeoffs

[kishoreg]

can we see all the configs related to upsert? lets try to keep the config keys intuitive and follow some pattern

[KKcorps]

There is one corner case not handled. When a record is deleted, we might want to track its timestamp to guarantee consistency for late event. This could cause much bigger map size thus revoke the benefit of deletion. We can further discuss the tradeoffs

I think we can keep that in the state for now. The current ask is to just support deleting the record afaik (i.e. to mark it as invalid). Reducing the state size can be thought of as an optimization.

[KKcorps]

can we see all the configs related to upsert? lets try to keep the config keys intuitive and follow some pattern

Definitely. This is just a Draft PR. I thought about using the existing Upsert.Mode config but it was not sufficient

[codecov-commenter]

Codecov Report

Merging #10035 (1625af5) into master (431b918) will decrease coverage by 0.07%.
The diff coverage is 62.85%.

@@             Coverage Diff              @@
##             master   #10035      +/-   ##
============================================
- Coverage     70.42%   70.34%   -0.08%     
- Complexity     5691     5697       +6     
============================================
  Files          1994     1994              
  Lines        107508   107579      +71     
  Branches      16340    16353      +13     
============================================
- Hits          75709    75679      -30     
- Misses        26502    26599      +97     
- Partials       5297     5301       +4     

Flag	Coverage Δ
integration1	24.99% <8.57%> (-0.04%)	⬇️
integration2	24.46% <8.57%> (-0.02%)	⬇️
unittests1	67.91% <62.85%> (-0.02%)	⬇️
unittests2	13.57% <0.00%> (-0.04%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
...local/indexsegment/mutable/MutableSegmentImpl.java	57.46% <33.33%> (-0.29%)	⬇️
...rg/apache/pinot/spi/config/table/UpsertConfig.java	72.34% <33.33%> (-2.66%)	⬇️
...t/ConcurrentMapPartitionUpsertMetadataManager.java	67.22% <61.53%> (-0.70%)	⬇️
...ache/pinot/segment/local/utils/IngestionUtils.java	29.77% <71.42%> (+1.70%)	⬆️
...manager/realtime/LLRealtimeSegmentDataManager.java	70.85% <100.00%> (+0.03%)	⬆️
...ent/local/realtime/impl/RealtimeSegmentConfig.java	92.42% <100.00%> (ø)
...g/apache/pinot/server/api/resources/ErrorInfo.java	0.00% <0.00%> (-100.00%)	⬇️
.../apache/pinot/common/metrics/MinionQueryPhase.java	0.00% <0.00%> (-100.00%)	⬇️
...data/manager/realtime/DefaultSegmentCommitter.java	0.00% <0.00%> (-80.00%)	⬇️
...t/server/api/resources/DefaultExceptionMapper.java	0.00% <0.00%> (-75.00%)	⬇️
... and 39 more

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more