skip late cron job with max allowed delay

[klsince]

Add a feature flag to allow to skip late cron schedules to shed load from the cron scheduler. Otherwise the scheduler would lag behind further and further, and one has to restart controller to shed all schedules and catch up.

Release note

1. config: controller.task.skipLateCronSchedule - to turn on this feature
2. config: controller.task.maxCronScheduleDelayInSeconds - to tune how late a schedule is allowed, if later than that, the schedule is skipped i.e. the task generation
3. a meter: cronSchedulerSkipped - to monitor on this

[codecov-commenter]

Codecov Report

Merging #9372 (5a440cd) into master (c8a114d) will increase coverage by 0.05%.
The diff coverage is 35.00%.

@@             Coverage Diff              @@
##             master    #9372      +/-   ##
============================================
+ Coverage     69.75%   69.80%   +0.05%     
- Complexity     4707     4785      +78     
============================================
  Files          1884     1884              
  Lines        100273   100296      +23     
  Branches      15252    15254       +2     
============================================
+ Hits          69942    70014      +72     
+ Misses        25380    25336      -44     
+ Partials       4951     4946       -5     

Flag	Coverage Δ
integration1	26.25% <25.00%> (+0.07%)	⬆️
integration2	24.84% <25.00%> (+0.06%)	⬆️
unittests1	66.96% <0.00%> (+0.02%)	⬆️
unittests2	15.30% <30.00%> (+0.02%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
...ntroller/helix/core/minion/CronJobScheduleJob.java	0.00% <0.00%> (ø)
...g/apache/pinot/common/metrics/ControllerMeter.java	100.00% <100.00%> (ø)
...va/org/apache/pinot/controller/ControllerConf.java	57.64% <100.00%> (+0.33%)	⬆️
...controller/helix/core/minion/PinotTaskManager.java	68.03% <100.00%> (+0.37%)	⬆️
.../helix/core/minion/MinionInstancesCleanupTask.java	77.27% <0.00%> (-4.55%)	⬇️
...lix/core/realtime/PinotRealtimeSegmentManager.java	76.43% <0.00%> (-4.19%)	⬇️
.../org/apache/pinot/core/startree/StarTreeUtils.java	76.28% <0.00%> (-2.07%)	⬇️
...e/pinot/segment/local/io/util/PinotDataBitSet.java	95.62% <0.00%> (-1.46%)	⬇️
.../helix/core/realtime/SegmentCompletionManager.java	72.15% <0.00%> (-1.02%)	⬇️
...perator/filter/SortedIndexBasedFilterOperator.java	85.71% <0.00%> (-0.96%)	⬇️
... and 26 more

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more