add segment size metric on segment addition rather than push

[jadami10]

Description

The initial implementation of this in #8387 was insufficient. Many of us do metadata pushes, so that metric was only capturing the metadata size in those cases.

This implementation uses a single point where segments are "added" to a server to capture the size. The benefit here is that we can now also get this metric for realtime, committed segments and not just offline segments.

I deployed this to an internal cluster and confirmed we see sizes for realtime and offline tables. I confirmed the segment size on disk is the same as what we see on disk.

Upgrade Notes

Does this PR prevent a zero down-time upgrade? (Assume upgrade order: Controller, Broker, Server, Minion)

• Yes (Please label as backward-incompat, and complete the section below on Release Notes)

Does this PR fix a zero-downtime upgrade introduced earlier?

• Yes (Please label this as backward-incompat, and complete the section below on Release Notes)

Does this PR otherwise need attention when creating release notes? Things to consider:

• New configuration options
• Deprecation of configurations
• Signature changes to public methods/interfaces
• New plugins added or old plugins removed

• Yes (Please label this PR as release-notes and complete the section on Release Notes)

Release Notes

For segments that are available on disk we will emit a metric for their uncompressed size.

Documentation

[Jackie-Jiang]

This won't work properly because when the server restarts, there is no guarantee which segment is loaded first.
If it is okay to track the compressed segment size (segment size in the deep store), @timsants is working on adding the compressed segment size to the ZK metadata in #8358. Then you may use that segment size which is always the actual compressed file size

[codecov-commenter]

Codecov Report

Merging #8402 (2b74e7d) into master (a388f67) will decrease coverage by 6.65%.
The diff coverage is 100.00%.

@@             Coverage Diff              @@
##             master    #8402      +/-   ##
============================================
- Coverage     70.69%   64.03%   -6.66%     
- Complexity     4275     4280       +5     
============================================
  Files          1654     1609      -45     
  Lines         86475    84600    -1875     
  Branches      13040    12841     -199     
============================================
- Hits          61136    54177    -6959     
- Misses        21111    26506    +5395     
+ Partials       4228     3917     -311     

Flag	Coverage Δ
integration1	?
integration2	?
unittests1	66.96% <100.00%> (+<0.01%)	⬆️
unittests2	14.13% <0.00%> (+<0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
...g/apache/pinot/common/metrics/ControllerGauge.java	0.00% <ø> (-97.44%)	⬇️
...ces/PinotSegmentUploadDownloadRestletResource.java	39.05% <ø> (-18.82%)	⬇️
...a/org/apache/pinot/common/metrics/ServerGauge.java	92.00% <100.00%> (-3.84%)	⬇️
.../pinot/core/data/manager/BaseTableDataManager.java	86.49% <100.00%> (-0.32%)	⬇️
...local/recordtransformer/ExpressionTransformer.java	100.00% <100.00%> (ø)
...va/org/apache/pinot/core/routing/RoutingTable.java	0.00% <0.00%> (-100.00%)	⬇️
...va/org/apache/pinot/common/config/NettyConfig.java	0.00% <0.00%> (-100.00%)	⬇️
...a/org/apache/pinot/common/metrics/MinionMeter.java	0.00% <0.00%> (-100.00%)	⬇️
...g/apache/pinot/common/metrics/ControllerMeter.java	0.00% <0.00%> (-100.00%)	⬇️
.../apache/pinot/common/metrics/BrokerQueryPhase.java	0.00% <0.00%> (-100.00%)	⬇️
... and 383 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a9ae53e...2b74e7d. Read the comment docs.

[jadami10]

thank you for the pointer, jackie. closing this in favor of extending #8358