[Backport 2.x] Support batch ingestion in bulk API (#12457) (#13306) #13462

chishui · 2024-04-30T04:15:30Z

Description

Backport 1219c56 from #13306.

This PR is to enable batch ingestion in _bulk API. Please refer to #12457 for proposal and discussion.

Related Issues

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Failing checks are inspected and point to the corresponding known issue(s) (See: Troubleshooting Failing Builds)
Commits are signed per the DCO using --signoff
Commit changes are listed out in CHANGELOG.md file (See: Changelog)
Public documentation issue/PR created

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

…earch-project#13306) * [PoC][issues-12457] Support Batch Ingestion Signed-off-by: Liyun Xiu <xiliyun@amazon.com> * Rewrite batch interface and handle error and metrics Signed-off-by: Liyun Xiu <xiliyun@amazon.com> * Remove unnecessary change Signed-off-by: Liyun Xiu <xiliyun@amazon.com> * Revert some unnecessary test change Signed-off-by: Liyun Xiu <xiliyun@amazon.com> * Keep executeBulkRequest main logic untouched Signed-off-by: Liyun Xiu <xiliyun@amazon.com> * Add UT Signed-off-by: Liyun Xiu <xiliyun@amazon.com> * Add UT & yamlRest test, fix BulkRequest se/deserialization Signed-off-by: Liyun Xiu <xiliyun@amazon.com> * Add missing java docs Signed-off-by: Liyun Xiu <xiliyun@amazon.com> * Remove Writable from BatchIngestionOption Signed-off-by: Liyun Xiu <xiliyun@amazon.com> * Add more UTs Signed-off-by: Liyun Xiu <xiliyun@amazon.com> * Fix spotlesscheck Signed-off-by: Liyun Xiu <xiliyun@amazon.com> * Rename parameter name to batch_size Signed-off-by: Liyun Xiu <xiliyun@amazon.com> * Add more rest yaml tests & update rest spec Signed-off-by: Liyun Xiu <xiliyun@amazon.com> * Remove batch_ingestion_option and only use batch_size Signed-off-by: Liyun Xiu <xiliyun@amazon.com> * Throw invalid request exception for invalid batch_size Signed-off-by: Liyun Xiu <xiliyun@amazon.com> * Update server/src/main/java/org/opensearch/action/bulk/BulkRequest.java Co-authored-by: Andriy Redko <drreta@gmail.com> Signed-off-by: Liyun Xiu <chishui2@gmail.com> * Remove version constant Signed-off-by: Liyun Xiu <xiliyun@amazon.com> --------- Signed-off-by: Liyun Xiu <xiliyun@amazon.com> Signed-off-by: Liyun Xiu <chishui2@gmail.com> Co-authored-by: Andriy Redko <drreta@gmail.com> (cherry picked from commit 1219c56)

github-actions · 2024-04-30T04:48:55Z

❌ Gradle check result for 0c1fdea: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Signed-off-by: Liyun Xiu <xiliyun@amazon.com>

github-actions · 2024-04-30T08:15:03Z

❕ Gradle check result for d78b7c2: UNSTABLE

TEST FAILURES:

      1 org.opensearch.cluster.MinimumClusterManagerNodesIT.testThreeNodesNoClusterManagerBlock

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

codecov · 2024-04-30T08:15:40Z

Codecov Report

Attention: Patch coverage is 83.23353% with 56 lines in your changes are missing coverage. Please review.

Project coverage is 71.22%. Comparing base (0dd892c) to head (d78b7c2).
Report is 213 commits behind head on 2.x.

❗ Current head d78b7c2 differs from pull request most recent head 587aed7. Consider uploading reports for the commit 587aed7 to get more accurate results

Files	Patch %	Lines
...main/java/org/opensearch/ingest/IngestService.java	79.81%	29 Missing and 14 partials ⚠️
...n/java/org/opensearch/action/bulk/BulkRequest.java	40.00%	5 Missing and 1 partial ⚠️
.../java/org/opensearch/ingest/CompoundProcessor.java	93.75%	0 Missing and 4 partials ⚠️
...a/org/opensearch/ingest/IngestDocumentWrapper.java	72.72%	3 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##                2.x   #13462      +/-   ##
============================================
- Coverage     71.28%   71.22%   -0.06%     
- Complexity    60145    61071     +926     
============================================
  Files          4957     5025      +68     
  Lines        282799   287340    +4541     
  Branches      41409    41996     +587     
============================================
+ Hits         201591   204657    +3066     
- Misses        64189    65381    +1192     
- Partials      17019    17302     +283

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

chishui · 2024-04-30T08:19:00Z

@dblock please help merge this backport

Signed-off-by: Liyun Xiu <xiliyun@amazon.com>

github-actions · 2024-04-30T13:30:49Z

❌ Gradle check result for 587aed7: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

dblock · 2024-04-30T13:54:36Z

❌ Gradle check result for 587aed7: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

org.opensearch.gateway.RecoveryFromGatewayIT.testShardStoreFetchMultiNodeMultiIndexesUsingBatchAction

#13307

org.opensearch.remotemigration.RemoteReplicaRecoveryIT.testReplicaRecovery

#8919

github-actions · 2024-04-30T14:46:31Z

❌ Gradle check result for 587aed7: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

dblock · 2024-04-30T14:48:07Z

❌ Gradle check result for 587aed7: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

org.opensearch.indices.CacheStatsAPIIndicesRequestCacheIT.testNullLevels {p0={"opensearch.experimental.feature.pluggable.caching.enabled":"true"}}

#13458

github-actions · 2024-04-30T15:38:46Z

❌ Gradle check result for 587aed7: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

dblock · 2024-04-30T15:39:57Z

❌ Gradle check result for 587aed7: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

org.opensearch.cluster.allocation.ClusterRerouteIT.testDelayWithALargeAmountOfShards

#10558

github-actions · 2024-04-30T16:30:09Z

❌ Gradle check result for 587aed7: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

chishui · 2024-04-30T16:33:57Z

❌ Gradle check result for 587aed7: FAILURE

org.opensearch.ratelimitting.admissioncontrol.AdmissionForClusterManagerIT.testAdmissionControlResponseStatus

#13045

org.opensearch.remotestore.SegmentReplicationUsingRemoteStoreIT.testIndexReopenClose

#13084

github-actions · 2024-04-30T17:35:35Z

❌ Gradle check result for 587aed7: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

chishui · 2024-04-30T17:38:42Z

❌ Gradle check result for 587aed7: FAILURE

org.opensearch.indices.CacheStatsAPIIndicesRequestCacheIT.testNullLevels {p0={"opensearch.experimental.feature.pluggable.caching.enabled":"true"}}

#13458

org.opensearch.remotemigration.RemoteReplicaRecoveryIT.testReplicaRecovery

#13473

github-actions · 2024-04-30T18:28:36Z

❕ Gradle check result for 587aed7: UNSTABLE

TEST FAILURES:

      1 org.opensearch.cluster.allocation.ClusterRerouteIT.testDelayWithALargeAmountOfShards

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

chishui requested review from anasalkouz, andrross, Bukhtawar, CEHENKLE, dblock, dbwiddis, dreamer-89, gbbafna, kotwanikunal, mch2, msfroh, nknize, owaiskazi19, reta, Rishikesh1159, sachinpkale, saratvemulapalli, shwetathareja, sohami, tlfeng and VachaShah as code owners April 30, 2024 04:15

Adjust changelog item position to trigger CI

d78b7c2

Signed-off-by: Liyun Xiu <xiliyun@amazon.com>

Merge branch '2.x' into backport/backport-13306-to-2.x

587aed7

Signed-off-by: Liyun Xiu <xiliyun@amazon.com>

dblock approved these changes Apr 30, 2024

View reviewed changes

dblock merged commit e7828c0 into opensearch-project:2.x Apr 30, 2024
30 checks passed

chishui deleted the backport/backport-13306-to-2.x branch May 6, 2024 05:27

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Backport 2.x] Support batch ingestion in bulk API (#12457) (#13306) #13462

[Backport 2.x] Support batch ingestion in bulk API (#12457) (#13306) #13462

chishui commented Apr 30, 2024 •

edited by dblock

Loading

github-actions bot commented Apr 30, 2024

github-actions bot commented Apr 30, 2024

codecov bot commented Apr 30, 2024 •

edited

Loading

chishui commented Apr 30, 2024

github-actions bot commented Apr 30, 2024

dblock commented Apr 30, 2024

github-actions bot commented Apr 30, 2024

dblock commented Apr 30, 2024

github-actions bot commented Apr 30, 2024

dblock commented Apr 30, 2024

github-actions bot commented Apr 30, 2024

chishui commented Apr 30, 2024 •

edited

Loading

github-actions bot commented Apr 30, 2024

chishui commented Apr 30, 2024 •

edited

Loading

github-actions bot commented Apr 30, 2024

[Backport 2.x] Support batch ingestion in bulk API (#12457) (#13306) #13462

[Backport 2.x] Support batch ingestion in bulk API (#12457) (#13306) #13462

Conversation

chishui commented Apr 30, 2024 • edited by dblock Loading

Description

Related Issues

Check List

github-actions bot commented Apr 30, 2024

github-actions bot commented Apr 30, 2024

codecov bot commented Apr 30, 2024 • edited Loading

Codecov Report

chishui commented Apr 30, 2024

github-actions bot commented Apr 30, 2024

dblock commented Apr 30, 2024

github-actions bot commented Apr 30, 2024

dblock commented Apr 30, 2024

github-actions bot commented Apr 30, 2024

dblock commented Apr 30, 2024

github-actions bot commented Apr 30, 2024

chishui commented Apr 30, 2024 • edited Loading

github-actions bot commented Apr 30, 2024

chishui commented Apr 30, 2024 • edited Loading

github-actions bot commented Apr 30, 2024

chishui commented Apr 30, 2024 •

edited by dblock

Loading

codecov bot commented Apr 30, 2024 •

edited

Loading

chishui commented Apr 30, 2024 •

edited

Loading

chishui commented Apr 30, 2024 •

edited

Loading