Allow to configure multi-datadirs as instance configs and a Quickstart example about them

[klsince]

This PR is part of the work to support multi-datadir for Pinot server as tracked by #8843.

This one makes it simpler and more consistent to configure the feature. And added a QuickStart example about how this works with the typical configs.

Release Note

One can config tier and its datadir as instance configs to be consistent across tables, e.g.

"pinot.server.instance.tierConfigs.0.tierName": "hotTier"
"pinot.server.instance.tierConfigs.0.dataDir: "/tmp/multidir_test/hotTier"
"pinot.server.instance.tierConfigs.1.tierName": "coldTier"
"pinot.server.instance.tierConfigs.1.dataDir": "/tmp/multidir_test/coldTier"

The datadir from instance configs can be overwritten by tierConfigs in TableConfig, e.g.

  "tierConfigs": [
    {
      "name": "hotTier",
      "segmentSelectorType": "time",
      "segmentAge": "3130d",
      "storageType": "pinot_server",
      "serverTag": "DefaultTenant_OFFLINE",
      "tierBackendProperties": {
          "dataDir": "/foo/bar/baz", <--- overwrite the instance config
      }
    },
    {
      "name": "coldTier",
      "segmentSelectorType": "time",
      "segmentAge": "3140d",
      "storageType": "pinot_server",
      "serverTag": "DefaultTenant_OFFLINE"
    }
  ]

But the instance configs are more recommended to keep datadirs consistent across tables within a cluster.

[codecov-commenter]

Codecov Report

Merging #9705 (2ec5616) into master (173916e) will decrease coverage by 1.43%.
The diff coverage is 78.26%.

@@             Coverage Diff              @@
##             master    #9705      +/-   ##
============================================
- Coverage     70.46%   69.02%   -1.44%     
+ Complexity     5535     5064     -471     
============================================
  Files          1982     1982              
  Lines        106449   106496      +47     
  Branches      16131    16140       +9     
============================================
- Hits          75006    73512    -1494     
- Misses        26213    27833    +1620     
+ Partials       5230     5151      -79     

Flag	Coverage Δ
integration1	?
integration2	24.65% <5.79%> (+0.16%)	⬆️
unittests1	67.96% <83.33%> (+0.01%)	⬆️
unittests2	15.81% <5.79%> (-0.02%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
...roller/helix/core/relocation/SegmentRelocator.java	47.14% <0.00%> (-16.43%)	⬇️
...ent/local/data/manager/TableDataManagerConfig.java	36.20% <55.00%> (+9.89%)	⬆️
.../apache/pinot/controller/util/TableTierReader.java	84.21% <57.14%> (-4.26%)	⬇️
...che/pinot/common/utils/config/TierConfigUtils.java	74.54% <95.83%> (+4.09%)	⬆️
.../pinot/core/data/manager/BaseTableDataManager.java	74.86% <100.00%> (+0.28%)	⬆️
...indexsegment/immutable/ImmutableSegmentLoader.java	85.04% <100.00%> (+0.14%)	⬆️
.../local/loader/TierBasedSegmentDirectoryLoader.java	90.81% <100.00%> (+1.92%)	⬆️
...local/segment/index/loader/IndexLoadingConfig.java	81.56% <100.00%> (+0.21%)	⬆️
...ment/spi/loader/SegmentDirectoryLoaderContext.java	92.50% <100.00%> (+0.83%)	⬆️
...g/apache/pinot/server/api/resources/ErrorInfo.java	0.00% <0.00%> (-100.00%)	⬇️
... and 148 more

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

[npawar]

thanks for the config change. lgtm