Add LASTWITHTIME aggregate function support #7315

[weixiangsun]

Description

Adding aggregate function to return last value of time-based data set.

Upgrade Notes

Does this PR prevent a zero down-time upgrade? (Assume upgrade order: Controller, Broker, Server, Minion)

• Yes (Please label as backward-incompat, and complete the section below on Release Notes)

Does this PR fix a zero-downtime upgrade introduced earlier?

• Yes (Please label this as backward-incompat, and complete the section below on Release Notes)

Does this PR otherwise need attention when creating release notes? Things to consider:

• New configuration options
• Deprecation of configurations
• Signature changes to public methods/interfaces
• New plugins added or old plugins removed

• Yes (Please label this PR as release-notes and complete the section on Release Notes)

Release Notes

Documentation

Will raise an another PR to update the documentation once this PR is approved and the changes are finalized.

[Jackie-Jiang]

Good in general, great job

[weixiangsun]

I addressed all the comments and finished the testing. Please take another look! @Jackie-Jiang @lakshmanan-v @pjpringle

[codecov-commenter]

Codecov Report

Merging #7584 (0c37328) into master (5eee297) will decrease coverage by 4.56%.
The diff coverage is 14.17%.

❗ Current head 0c37328 differs from pull request most recent head 6d60f1e. Consider uploading reports for the commit 6d60f1e to get more accurate results

@@            Coverage Diff             @@
##           master    #7584      +/-   ##
==========================================
- Coverage   32.16%   27.60%   -4.57%     
==========================================
  Files        1513     1566      +53     
  Lines       75557    79579    +4022     
  Branches    11055    11795     +740     
==========================================
- Hits        24305    21965    -2340     
- Misses      49156    55619    +6463     
+ Partials     2096     1995     -101     

Flag	Coverage Δ
integration1	?
integration2	27.60% <14.17%> (-1.35%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
...e/pinot/common/minion/MergeRollupTaskMetadata.java	0.00% <ø> (-78.27%)	⬇️
.../minion/RealtimeToOfflineSegmentsTaskMetadata.java	100.00% <ø> (+26.66%)	⬆️
...ot/common/request/context/RequestContextUtils.java	53.26% <0.00%> (-22.26%)	⬇️
...e/pinot/common/utils/FileUploadDownloadClient.java	53.24% <0.00%> (-6.07%)	⬇️
...java/org/apache/pinot/common/utils/StringUtil.java	80.00% <ø> (+15.29%)	⬆️
...apache/pinot/common/utils/config/TagNameUtils.java	64.28% <0.00%> (-24.61%)	⬇️
...g/apache/pinot/common/utils/helix/HelixHelper.java	41.26% <0.00%> (-6.53%)	⬇️
...ller/api/resources/PinotTenantRestletResource.java	4.21% <0.00%> (-0.39%)	⬇️
...ler/api/resources/TableConfigsRestletResource.java	0.00% <0.00%> (ø)
...realtime/segment/DefaultFlushThresholdUpdater.java	83.33% <ø> (ø)
... and 376 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5eee297...6d60f1e. Read the comment docs.

[weixiangsun]

@Jackie-Jiang @lakshmanan-v I have already addressed the comments. Please take another look and leave any comment you have! Thanks a lot!

[lakshmanan-v]

lgtm. Please wait for @Jackie-Jiang's approval before merging.

[Jackie-Jiang]

LGTM with minor comments. Great work!

[pjpringle]

thanks! looking forward to giving this a go.

can we now add functionality to the segment compaction to use this operator, e.g. take last value every hour

[weixiangsun]

@pjpringle any sample change? I can add it.