Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Using local copy of segment instead of downloading from remote #14429

Merged
merged 1 commit into from
Nov 14, 2024
Merged

Using local copy of segment instead of downloading from remote #14429

merged 1 commit into from
Nov 14, 2024
+14 −19

Conversation

rajagopr
Copy link
Contributor

Re-introducing PR-12863 post testing in an internal cluster.

Testing

Minion Logs

2024/11/12 17:04:12.231 INFO [FileIngestionTaskExecutor] [TaskStateModelFactory-task_thread-0] Generated 1 segments with duration: 12261ms

2024/11/12 17:04:12.239 INFO [FileIngestionTaskExecutor] [TaskStateModelFactory-task_thread-0] Adding new segments: 1 created from multiple input files: 1

2024/11/12 17:04:12.273 INFO [HttpClient] [TaskStateModelFactory-task_thread-0] Sending request: /segments/suspects001/startDataIngestRequest?tableType=OFFLINE&taskType=FileIngestionTask to controller: pinot-pinot-controller-0.pinot-pinot-controller-headless.managed.svc.cluster.local, version: Unknown

2024/11/12 17:04:12.273 INFO [FileIngestionTaskExecutor] [TaskStateModelFactory-task_thread-0] Submitted checkpoint: FileIngestionTask_1731431035906_0 for table: suspects001_OFFLINE with new segments: [suspects001_2014-06-01_2022-12-12_FileIngestionTask_1731431035906_0_0]

2024/11/12 17:04:14.949 INFO [FileIngestionTaskExecutor] [TaskStateModelFactory-task_thread-0] Total uncompressed segment size for task 60613536 bytes

2024/11/12 17:04:14.949 INFO [FileIngestionTaskExecutor] [TaskStateModelFactory-task_thread-0] Compressed segments with duration: 2676ms

2024/11/12 17:04:14.951 INFO [FileIngestionTaskExecutor] [TaskStateModelFactory-task_thread-0] Uploading compressed segment: suspects001_2014-06-01_2022-12-12_FileIngestionTask_1731431035906_0_0.tar.gz with name: suspects001_2014-06-01_2022-12-12_FileIngestionTask_1731431035906_0_0

2024/11/12 17:04:14.953 INFO [FileIngestionTaskExecutor] [TaskStateModelFactory-task_thread-0] Using push mode: METADATA to upload segment: /home/pinot/data/minionData/FileIngestionTask/tmp-97a38a61-a5bd-4ab7-af46-50fc537c5214/workingDir/tarredSegments/suspects001_2014-06-01_2022-12-12_FileIngestionTask_1731431035906_0_0.tar.gz

2024/11/12 17:04:15.102 INFO [S3PinotFS] [TaskStateModelFactory-task_thread-0] Copy /home/pinot/data/minionData/FileIngestionTask/tmp-97a38a61-a5bd-4ab7-af46-50fc537c5214/workingDir/tarredSegments/suspects001_2014-06-01_2022-12-12_FileIngestionTask_1731431035906_0_0.tar.gz from local to s3://sc-dev-dmtest-testdm-pinot-fs/sc-dev/managed/pinot/suspects001/suspects001_2014-06-01_2022-12-12_FileIngestionTask_1731431035906_0_0.tar.gz

2024/11/12 17:04:15.411 INFO [IngestionTaskUtils] [TaskStateModelFactory-task_thread-0] Moved generated segment from: /home/pinot/data/minionData/FileIngestionTask/tmp-97a38a61-a5bd-4ab7-af46-50fc537c5214/workingDir/tarredSegments/suspects001_2014-06-01_2022-12-12_FileIngestionTask_1731431035906_0_0.tar.gz to: s3://sc-dev-dmtest-testdm-pinot-fs/sc-dev/managed/pinot/suspects001/suspects001_2014-06-01_2022-12-12_FileIngestionTask_1731431035906_0_0.tar.gz

2024/11/12 17:04:15.412 INFO [SegmentPushUtils] [TaskStateModelFactory-task_thread-0] Start pushing segment metadata: {s3://sc-dev-dmtest-testdm-pinot-fs/sc-dev/managed/pinot/suspects001/suspects001_2014-06-01_2022-12-12_FileIngestionTask_1731431035906_0_0.tar.gz=/home/pinot/data/minionData/FileIngestionTask/tmp-97a38a61-a5bd-4ab7-af46-50fc537c5214/workingDir/tarredSegments/suspects001_2014-06-01_2022-12-12_FileIngestionTask_1731431035906_0_0.tar.gz} to locations: [org.apache.pinot.spi.ingestion.batch.spec.PinotClusterSpec@4d2aba53] for table: suspects001_OFFLINE

2024/11/12 17:04:15.413 INFO [SegmentPushUtils] [TaskStateModelFactory-task_thread-0] Checking if metadata tar gz file /home/pinot/data/minionData/FileIngestionTask/tmp-97a38a61-a5bd-4ab7-af46-50fc537c5214/workingDir/tarredSegments/suspects001_2014-06-01_2022-12-12_FileIngestionTask_1731431035906_0_0.metadata.tar.gz exists

2024/11/12 17:04:15.413 INFO [SegmentPushUtils] [TaskStateModelFactory-task_thread-0] Trying to untar Metadata file from: [/home/pinot/data/minionData/FileIngestionTask/tmp-97a38a61-a5bd-4ab7-af46-50fc537c5214/workingDir/tarredSegments/suspects001_2014-06-01_2022-12-12_FileIngestionTask_1731431035906_0_0.tar.gz] to [/tmp/segmentMetadataDir-18f05bf5-6295-4c98-b96b-f84efe53110a]

[Metadata file getting generated from the local segment file]

2024/11/12 17:04:15.419 INFO [SegmentPushUtils] [TaskStateModelFactory-task_thread-0] Trying to untar CreationMeta file from: [/home/pinot/data/minionData/FileIngestionTask/tmp-97a38a61-a5bd-4ab7-af46-50fc537c5214/workingDir/tarredSegments/suspects001_2014-06-01_2022-12-12_FileIngestionTask_1731431035906_0_0.tar.gz] to [/tmp/segmentMetadataDir-18f05bf5-6295-4c98-b96b-f84efe53110a]

2024/11/12 17:04:15.574 INFO [SegmentPushUtils] [TaskStateModelFactory-task_thread-0] Trying to tar segment metadata dir [/tmp/segmentMetadataDir-18f05bf5-6295-4c98-b96b-f84efe53110a] to [/tmp/segmentMetadata-18f05bf5-6295-4c98-b96b-f84efe53110a.tar.gz]

2024/11/12 17:04:15.580 INFO [SegmentPushUtils] [TaskStateModelFactory-task_thread-0] Pushing segments: [suspects001_2014-06-01_2022-12-12_FileIngestionTask_1731431035906_0_0] to location: https://pinot-pinot-controller-headless.managed.svc.cluster.local:9000 for table: suspects001_OFFLINE

2024/11/12 17:04:15.832 INFO [HttpClient] [TaskStateModelFactory-task_thread-0] Sending request: /segments/batchUpload?tableName=suspects001_OFFLINE&tableType=OFFLINE to controller: pinot-pinot-controller-0.pinot-pinot-controller-headless.managed.svc.cluster.local, version: Unknown

2024/11/12 17:04:15.833 INFO [SegmentPushUtils] [TaskStateModelFactory-task_thread-0] Response for pushing table suspects001_OFFLINE segments [suspects001_2014-06-01_2022-12-12_FileIngestionTask_1731431035906_0_0] to location https://pinot-pinot-controller-headless.managed.svc.cluster.local:9000 - 200: {"status":"Successfully uploaded segments: [suspects001_2014-06-01_2022-12-12_FileIngestionTask_1731431035906_0_0] of table: suspects001_OFFLINE in 206 ms"}

2024/11/12 17:04:15.833 INFO [FileIngestionTaskExecutor] [TaskStateModelFactory-task_thread-0] Uploaded compressed segment: suspects001_2014-06-01_2022-12-12_FileIngestionTask_1731431035906_0_0.tar.gz with name: suspects001_2014-06-01_2022-12-12_FileIngestionTask_1731431035906_0_0

2024/11/12 17:04:15.838 INFO [FileIngestionTaskExecutor] [TaskStateModelFactory-task_thread-0] Uploaded segments: 1 with duration: 886ms

2024/11/12 17:04:15.874 INFO [HttpClient] [TaskStateModelFactory-task_thread-0] Sending request: /segments/suspects001/endDataIngestRequest?tableType=OFFLINE&taskType=FileIngestionTask&checkpointEntryKey=FileIngestionTask_1731431035906_0 to controller: pinot-pinot-controller-0.pinot-pinot-controller-headless.managed.svc.cluster.local, version: Unknown

@codecov-commenter
Copy link

codecov-commenter commented Nov 12, 2024 •
edited
Loading

Codecov Report

Attention: Patch coverage is 9.09091% with 10 lines in your changes missing coverage. Please review.

Project coverage is 63.83%. Comparing base (59551e4) to head (f4dbbeb).
Report is 1313 commits behind head on master.

Files with missing lines Patch % Lines
.../tasks/BaseMultipleSegmentsConversionExecutor.java 9.09% 10 Missing ⚠️
Additional details and impacted files 
@@             Coverage Diff              @@
##             master   #14429      +/-   ##
============================================
+ Coverage     61.75%   63.83%   +2.08%     
- Complexity      207     1566    +1359     
============================================
  Files          2436     2663     +227     
  Lines        133233   146225   +12992     
  Branches      20636    22433    +1797     
============================================
+ Hits          82274    93348   +11074     
- Misses        44911    45989    +1078     
- Partials       6048     6888     +840
Flag Coverage Δ
custom-integration1 100.00% <ø> (+99.99%) ⬆️
integration 100.00% <ø> (+99.99%) ⬆️
integration1 100.00% <ø> (+99.99%) ⬆️
integration2 0.00% <ø> (ø)
java-11 63.80% <9.09%> (+2.09%) ⬆️
java-21 63.73% <9.09%> (+2.10%) ⬆️
skip-bytebuffers-false 63.83% <9.09%> (+2.09%) ⬆️
skip-bytebuffers-true 63.63% <9.09%> (+35.90%) ⬆️
temurin 63.83% <9.09%> (+2.08%) ⬆️
unittests 63.83% <9.09%> (+2.08%) ⬆️
unittests1 55.52% <ø> (+8.63%) ⬆️
unittests2 34.16% <9.09%> (+6.43%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@Jackie-Jiang Jackie-Jiang merged commit b90d1da into apache:master Nov 14, 2024
40 of 42 checks passed
@rajagopr rajagopr deleted the use-local-segment branch November 14, 2024 16:32
rajagopr added a commit to rajagopr/pinot that referenced this pull request Nov 18, 2024
@rajagopr
Revert "Using local copy of segment instead of downloading from remote (
234474b 
apache#14429)"

This reverts commit b90d1da.
@rajagopr rajagopr mentioned this pull request Nov 18, 2024
Merged
Jackie-Jiang pushed a commit that referenced this pull request Nov 19, 2024
@rajagopr
Revert "Using local copy of segment instead of downloading from remote (
d9b4723 
#14429)" (#14484)

This reverts commit b90d1da.
davecromberge pushed a commit to davecromberge/pinot that referenced this pull request Nov 22, 2024
@rajagopr @davecromberge
Using local copy of segment instead of downloading from remote (apach…
eb1fe53 
…e#14429)
davecromberge pushed a commit to davecromberge/pinot that referenced this pull request Nov 22, 2024
@rajagopr @davecromberge
Revert "Using local copy of segment instead of downloading from remote (
1d937f9 
apache#14429)" (apache#14484)

This reverts commit b90d1da.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@swaminathanmanish swaminathanmanish swaminathanmanish approved these changes

Assignees
No one assigned
Labels
enhancement ingestion minion
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@rajagopr @codecov-commenter @swaminathanmanish @Jackie-Jiang