You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Currently, when a minion node processes a segment it calls uploadSegmentAsMultiPart API of controller. The controller processes and uploads the segment to deepstore and then issues a refresh call to the respective servers.
Raising this issue to gather the following context:
Why didn't we allow the minion nodes to upload the files to deepstore (just like servers do)? And then use uploadtype as URI to refresh data from deepstore to servers via controllers?
Do we expect the controllers under heavy load if we continue the present behaviour? We are trying to use UpsertCompaction at scale in Uber and are a bit skeptical that if we enable it for a lot of tables with the present config, the controllers might be under heavy resource utilization. We run our clusters with very minimal controller nodes currently.
One of the ideas to resolve this:
Can we allow minion-nodes to re-upload segments to deepstore post processing? Maybe behind a config initially - allowDeepstoreUpload? Or is this considered like any anti-pattern? At present, minion nodes do interact with deepstore to download the segment file before processing.
@tibrewalpratik17 Actually all controller regular jobs should have be placed in minion. I also have a requirement to place the job of deleting tmp deep store files in minion for better scaling up.
Currently, when a minion node processes a segment it calls
uploadSegmentAsMultiPart
API of controller. The controller processes and uploads the segment to deepstore and then issues a refresh call to the respective servers.pinot/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotSegmentUploadDownloadRestletResource.java
Line 546 in 4abb2d1
Raising this issue to gather the following context:
Why didn't we allow the minion nodes to upload the files to deepstore (just like servers do)? And then use uploadtype as URI to refresh data from deepstore to servers via controllers?
pinot/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotSegmentUploadDownloadRestletResource.java
Line 507 in 4abb2d1
Do we expect the controllers under heavy load if we continue the present behaviour? We are trying to use UpsertCompaction at scale in Uber and are a bit skeptical that if we enable it for a lot of tables with the present config, the controllers might be under heavy resource utilization. We run our clusters with very minimal controller nodes currently.
One of the ideas to resolve this:
allowDeepstoreUpload
? Or is this considered like any anti-pattern? At present, minion nodes do interact with deepstore to download the segment file before processing.cc @ankitsultana
The text was updated successfully, but these errors were encountered: