Changes for the Upload Workflow for merged segments #18610
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
❕ Gradle check result for 1999d32: UNSTABLE Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure. |
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #18610 +/- ##
============================================
+ Coverage 72.55% 72.75% +0.20%
- Complexity 68185 68262 +77
============================================
Files 5555 5556 +1
Lines 313857 313868 +11
Branches 45522 45521 -1
============================================
+ Hits 227707 228359 +652
+ Misses 67616 66935 -681
- Partials 18534 18574 +40 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
|
❌ Gradle check result for ffb007a: FAILURE Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change? |
2 tasks
|
❌ Gradle check result for d358dd1: FAILURE Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change? |
ashking94
requested changes
Jul 4, 2025
server/src/main/java/org/opensearch/index/shard/RemoteStoreUploaderService.java
Outdated
Show resolved
Hide resolved
server/src/main/java/org/opensearch/index/shard/RemoteStoreUploaderService.java
Show resolved
Hide resolved
server/src/test/java/org/opensearch/index/shard/RemoteStoreUploaderServiceTests.java
Outdated
Show resolved
Hide resolved
|
❌ Gradle check result for a2a04c1: FAILURE Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change? |
|
❕ Gradle check result for 3c61f11: UNSTABLE Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure. |
added 7 commits
July 7, 2025 14:48
Signed-off-by: Aditi Goyal <adgoyal@amazon.com>
Signed-off-by: Aditi Goyal <adgoyal@amazon.com>
Signed-off-by: Aditi Goyal <adgoyal@amazon.com>
Signed-off-by: Aditi Goyal <adgoyal@amazon.com>
Signed-off-by: Aditi Goyal <adgoyal@amazon.com>
Signed-off-by: Aditi Goyal <adgoyal@amazon.com>
Signed-off-by: Aditi Goyal <adgoyal@amazon.com>
Adiiigo
force-pushed
the
upload-segregation
branch
from
3c61f11 to
1ec78e0
Compare
Signed-off-by: Aditi Goyal <adgoyal@amazon.com>
|
❌ Gradle check result for e0f7998: FAILURE Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change? |
added 2 commits
July 9, 2025 14:02
Signed-off-by: Aditi Goyal <adgoyal@amazon.com>
This reverts commit e542b99. Signed-off-by: Aditi Goyal <adgoyal@amazon.com>
Adiiigo
force-pushed
the
upload-segregation
branch
from
aa5b97b to
71614ea
Compare
Adiiigo
requested review from
a team,
Bukhtawar,
CEHENKLE,
Rishikesh1159,
VachaShah,
cwperks,
dbwiddis,
gbbafna,
jed326,
kotwanikunal,
mch2,
msfroh,
owaiskazi19,
reta,
sachinpkale,
saratvemulapalli,
shwetathareja and
sohami
as code owners
3 tasks
ashking94
approved these changes
Jul 11, 2025
vishdivs
pushed a commit
to vishdivs/OpenSearch
that referenced
this pull request
Jul 14, 2025
…ct#18610) * Changes for the Upload Workflow for merged segments Signed-off-by: Aditi Goyal <adgoyal@amazon.com> * Addressed Todos Signed-off-by: Aditi Goyal <adgoyal@amazon.com> * Added Test Cases Signed-off-by: Aditi Goyal <adgoyal@amazon.com> * Removed the usage of Forbidden APIs Signed-off-by: Aditi Goyal <adgoyal@amazon.com> * Updated the code to include priority upload as paramter Signed-off-by: Aditi Goyal <adgoyal@amazon.com> * Updated the remote listener to use remote upploader interface Signed-off-by: Aditi Goyal <adgoyal@amazon.com> * Added JavaDocs across the changes Signed-off-by: Aditi Goyal <adgoyal@amazon.com> * Addressed spotless check Signed-off-by: Aditi Goyal <adgoyal@amazon.com> * Dummy commit(can be reverted) Signed-off-by: Aditi Goyal <adgoyal@amazon.com> * Revert "Dummy commit(can be reverted)" This reverts commit e542b99. Signed-off-by: Aditi Goyal <adgoyal@amazon.com> --------- Signed-off-by: Aditi Goyal <adgoyal@amazon.com> Co-authored-by: Aditi Goyal <adgoyal@amazon.com>
animodak7
added a commit
to animodak7/OpenSearch
that referenced
this pull request
Jul 16, 2025
* Fix regex query to work with field alias (opensearch-project#18215) --------- Signed-off-by: bowenlan-amzn <bowenlan23@gmail.com> Signed-off-by: Ankit Jain <akjain@amazon.com> Co-authored-by: Ankit Jain <akjain@amazon.com> * Add workflow to automatically trigger manifest generation process (opensearch-project#18629) Signed-off-by: Sayali Gaikawad <gaiksaya@amazon.com> * Bump com.squareup.okio:okio from 3.13.0 to 3.14.0 in /test/fixtures/hdfs-fixture (opensearch-project#18645) * Bump com.squareup.okio:okio in /test/fixtures/hdfs-fixture Bumps [com.squareup.okio:okio](https://github.com/square/okio) from 3.13.0 to 3.14.0. - [Release notes](https://github.com/square/okio/releases) - [Changelog](https://github.com/square/okio/blob/master/CHANGELOG.md) - [Commits](square/okio@parent-3.13.0...parent-3.14.0) --- updated-dependencies: - dependency-name: com.squareup.okio:okio dependency-version: 3.14.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> * Update changelog Signed-off-by: dependabot[bot] <support@github.com> --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Fix credentials format (opensearch-project#18651) Signed-off-by: Sayali Gaikawad <gaiksaya@amazon.com> * Provide aux transport to SecureAuxTransportSettingsProvider (opensearch-project#18616) * Pass aux transport to secure settings parameters(). Signed-off-by: Finn Carroll <carrofin@amazon.com> * Update javadocs, fix UTs. Signed-off-by: Finn Carroll <carrofin@amazon.com> * Spotless apply Signed-off-by: Finn Carroll <carrofin@amazon.com> * Pass aux transport key as unique identifier instead of transport ref. Signed-off-by: Finn Carroll <carrofin@amazon.com> * Consistent naming. auxTransportSettingKey -> auxTransportType. Signed-off-by: Finn Carroll <carrofin@amazon.com> * Changelog. Signed-off-by: Finn Carroll <carrofin@amazon.com> * Spotless apply Signed-off-by: Finn Carroll <carrofin@amazon.com> --------- Signed-off-by: Finn Carroll <carrofin@amazon.com> * Making snapshot store/restore rate dynamic (opensearch-project#18069) * Making snapshot store/restore rate dynamic Signed-off-by: kkewwei <kewei.11@bytedance.com> Signed-off-by: kkewwei <kkewwei@163.com> * simple the code Signed-off-by: kkewwei <kewei.11@bytedance.com> Signed-off-by: kkewwei <kkewwei@163.com> --------- Signed-off-by: kkewwei <kewei.11@bytedance.com> Signed-off-by: kkewwei <kkewwei@163.com> * Bump com.netflix.nebula.ospackage-base from 11.11.2 to 12.0.0 in /distribution/packages (opensearch-project#18646) * Bump com.netflix.nebula.ospackage-base in /distribution/packages Bumps com.netflix.nebula.ospackage-base from 11.11.2 to 12.0.0. --- updated-dependencies: - dependency-name: com.netflix.nebula.ospackage-base dependency-version: 12.0.0 dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> * Update changelog Signed-off-by: dependabot[bot] <support@github.com> --------- Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: Craig Perkins <cwperx@amazon.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Craig Perkins <cwperx@amazon.com> * Extend Profiler to allow for non-timing info (opensearch-project#18631) * copied from other branch, ready for pr Signed-off-by: TJ Neuenfeldt <tjneu@amazon.com> * cleanup from merging Signed-off-by: TJ Neuenfeldt <tjneu@amazon.com> * another spotless apply and fixed breaking change Signed-off-by: TJ Neuenfeldt <tjneu@amazon.com> * another spotlessApply Signed-off-by: TJ Neuenfeldt <tjneu@amazon.com> * forgot to change back concurrent agg stuff Signed-off-by: TJ Neuenfeldt <tjneu@amazon.com> * forgot to spotless after prev fix Signed-off-by: TJ Neuenfeldt <tjneu@amazon.com> * added more unit tests for coverage Signed-off-by: TJ Neuenfeldt <tjneu@amazon.com> * changed to supplier, removed timer ctors Signed-off-by: TJ Neuenfeldt <tjneu@amazon.com> * made feedback changes Signed-off-by: TJ Neuenfeldt <tjneu@amazon.com> * forgot javadoc for new class Signed-off-by: TJ Neuenfeldt <tjneu@amazon.com> * forgot to revert name to toString Signed-off-by: TJ Neuenfeldt <tjneu@amazon.com> * fix count check in concurrentquerypb Signed-off-by: TJ Neuenfeldt <tjneu@amazon.com> * unmodifiable map add Signed-off-by: TJ Neuenfeldt <tjneu@amazon.com> --------- Signed-off-by: TJ Neuenfeldt <tjneu@amazon.com> Signed-off-by: TJ Neuenfeldt <56700214+neuenfeldttj@users.noreply.github.com> * [Bug Fix] Fix the backward compatibility regression with `COMPLEMENT` for Regexp queries introduced in OpenSearch 3.0 (opensearch-project#18640) * FIx ~ bug Signed-off-by: Prudhvi Godithi <pgodithi@amazon.com> * Update changelog Signed-off-by: Prudhvi Godithi <pgodithi@amazon.com> --------- Signed-off-by: Prudhvi Godithi <pgodithi@amazon.com> * [WLM] Make resource usage value cacheable for NodeDuressTrackers (opensearch-project#18649) --------- Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> Signed-off-by: Ankit Jain <akjain@amazon.com> Co-authored-by: Ankit Jain <akjain@amazon.com> * Fix bugs in replication lag computation (opensearch-project#18602) * Fix bug in replication lag computation. This change fixes a bug with replication lag computation to correctly use epoch reference point with Instant.now() and DateUtils. This change also fixes pruning logic to correctly remove the latest synced to checkpoint from tracking. Previously we would only prune up to the latest. This ensures that when a new checkpoint is eventually received we aren't incorrectly computing lag from the synced-to checkpoint. Signed-off-by: Marc Handalian <marc.handalian@gmail.com> * add changelog entry Signed-off-by: Marc Handalian <marc.handalian@gmail.com> * Fix flaky test to hardcode time lag between checkpoint arrival. Signed-off-by: Marc Handalian <marc.handalian@gmail.com> --------- Signed-off-by: Marc Handalian <marc.handalian@gmail.com> * Replace LegacyESVersion.fromString with Version.fromString (opensearch-project#18567) --------- Signed-off-by: Du Tran <quangdutran809@gmail.com> * Inject QueryCollectorSpecFactory from Plugins to create QueryCollectorContext in QueryPhase (opensearch-project#18637) * Initial Commit Signed-off-by: Varun Jain <varunudr@amazon.com> * Javadocs Signed-off-by: Varun Jain <varunudr@amazon.com> * Factory design pattern for QueryCollectorContextSpec Signed-off-by: Varun Jain <varunudr@amazon.com> * Fixing Concurrent Segment Search Signed-off-by: Varun Jain <varunudr@amazon.com> * Remove spec Signed-off-by: Varun Jain <varunudr@amazon.com> * Renaming method Signed-off-by: Varun Jain <varunudr@amazon.com> * Javadoc Signed-off-by: Varun Jain <varunudr@amazon.com> * Removing support method from QueryCollectorContextSpecFactory Signed-off-by: Varun Jain <varunudr@amazon.com> * Adding javadoc Signed-off-by: Varun Jain <varunudr@amazon.com> * Add changelog Signed-off-by: Varun Jain <varunudr@amazon.com> * Add experimentalAPI annotation Signed-off-by: Varun Jain <varunudr@amazon.com> * remove redundant check from getQueryCollectorContextSpec method Signed-off-by: Varun Jain <varunudr@amazon.com> * Add unit tests Signed-off-by: Varun Jain <varunudr@amazon.com> * Remove Test annotation Signed-off-by: Varun Jain <varunudr@amazon.com> * Reformatting Signed-off-by: Varun Jain <varunudr@amazon.com> --------- Signed-off-by: Varun Jain <varunudr@amazon.com> * Bump com.azure:azure-storage-blob from 12.30.0 to 12.30.1 in /plugins/repository-azure (opensearch-project#18644) * Bump com.azure:azure-storage-blob in /plugins/repository-azure Bumps [com.azure:azure-storage-blob](https://github.com/Azure/azure-sdk-for-java) from 12.30.0 to 12.30.1. - [Release notes](https://github.com/Azure/azure-sdk-for-java/releases) - [Commits](Azure/azure-sdk-for-java@azure-storage-blob_12.30.0...azure-storage-blob_12.30.1) --- updated-dependencies: - dependency-name: com.azure:azure-storage-blob dependency-version: 12.30.1 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> * Updating SHAs Signed-off-by: dependabot[bot] <support@github.com> * Update changelog Signed-off-by: dependabot[bot] <support@github.com> --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Fix checkpoint handling to prevent segment replication infinite loop (opensearch-project#18636) * Fix checkpoint handling to prevent segment replication infinite loop Signed-off-by: Ashish Singh <ssashish@amazon.com> * Modify existing test to verify code change behaviour Signed-off-by: Ashish Singh <ssashish@amazon.com> --------- Signed-off-by: Ashish Singh <ssashish@amazon.com> * Extend Approximation Framework to other numeric types (opensearch-project#18530) * Extend approximation to other numeric types Signed-off-by: Prudhvi Godithi <pgodithi@amazon.com> * Fix basic tests Signed-off-by: Prudhvi Godithi <pgodithi@amazon.com> * Fix basic tests Signed-off-by: Prudhvi Godithi <pgodithi@amazon.com> * Add Parameterized tests for all numeric types Signed-off-by: Prudhvi Godithi <pgodithi@amazon.com> * Fix FieldSortIT Signed-off-by: Prudhvi Godithi <pgodithi@amazon.com> * Fix FieldSortIT Signed-off-by: Prudhvi Godithi <pgodithi@amazon.com> * Address comments Signed-off-by: Prudhvi Godithi <pgodithi@amazon.com> * Update tests and changelog Signed-off-by: Prudhvi Godithi <pgodithi@amazon.com> * Update tests and changelog Signed-off-by: Prudhvi Godithi <pgodithi@amazon.com> --------- Signed-off-by: Prudhvi Godithi <pgodithi@amazon.com> * Bump BouncyCastle and password4j (opensearch-project#18668) Changes: - Bump org.bouncycastle:bctls-fips from 2.0.19 to 2.0.20 - Bump org.bouncycastle:bcpkix-fips from 2.0.7 to 2.0.8 - Bump org.bouncycastle:bcpg-fips from 2.0.10 to 2.0.11 - Bump com.password4j:password4j from 1.8.2 to 1.8.3 Signed-off-by: Andrey Pleskach <ples@aiven.io> * Add support for custom index name resolver from cluster plugin (opensearch-project#18593) Signed-off-by: Gagan Singh Saini <gagasa@amazon.com> * Remove clone holder and add Cleaner logic to clean up clones in FullFileCachedIndexInput (opensearch-project#18662) Signed-off-by: Shreyansh Ray <rayshrey@amazon.com> * Disallow resize for Warm Index, add Parameterized ITs for close in remote store (opensearch-project#18686) Signed-off-by: Gagan Singh Saini <gagasa@amazon.com> Co-authored-by: Gaurav Bafna <gbbafna@amazon.com> * Added support for range version support in semver (opensearch-project#18557) * Added support for range version support in semver Signed-off-by: Shruti Garg <shrugarg@amazon.com> * Added test coverage for missing lines Signed-off-by: Shruti Garg <shrugarg@amazon.com> * Fixed for non reachable code line and tests Signed-off-by: Shruti Garg <shrugarg@amazon.com> --------- Signed-off-by: Shruti Garg <shrugarg@amazon.com> * An attempt to fix flaky SecureReactorNetty4HttpServerTransportTests (opensearch-project#18680) Signed-off-by: Andriy Redko <drreta@gmail.com> * Bump com.squareup.okio:okio from 3.14.0 to 3.15.0 in /test/fixtures/hdfs-fixture (opensearch-project#18689) * Bump com.squareup.okio:okio in /test/fixtures/hdfs-fixture Bumps [com.squareup.okio:okio](https://github.com/square/okio) from 3.14.0 to 3.15.0. - [Release notes](https://github.com/square/okio/releases) - [Changelog](https://github.com/square/okio/blob/master/CHANGELOG.md) - [Commits](square/okio@parent-3.14.0...parent-3.15.0) --- updated-dependencies: - dependency-name: com.squareup.okio:okio dependency-version: 3.15.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> * Update changelog Signed-off-by: dependabot[bot] <support@github.com> --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Add Semantic Version field type mapper (opensearch-project#18454) * Add Semantic Version field type mapper and extensive unit tests Signed-off-by: Siddhant Deshmukh <deshsid@amazon.com> * Refactor SemanticVersionFieldMapper for flexible index, store, and doc_values support Signed-off-by: Siddhant Deshmukh <deshsid@amazon.com> --------- Signed-off-by: Siddhant Deshmukh <deshsid@amazon.com> * Bump com.azure:azure-core from 1.55.3 to 1.55.5 in /plugins/repository-azure (opensearch-project#18691) * Bump com.azure:azure-core in /plugins/repository-azure Bumps [com.azure:azure-core](https://github.com/Azure/azure-sdk-for-java) from 1.55.3 to 1.55.5. - [Release notes](https://github.com/Azure/azure-sdk-for-java/releases) - [Commits](Azure/azure-sdk-for-java@azure-core_1.55.3...azure-core_1.55.5) --- updated-dependencies: - dependency-name: com.azure:azure-core dependency-version: 1.55.5 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> * Updating SHAs Signed-off-by: dependabot[bot] <support@github.com> * Update changelog Signed-off-by: dependabot[bot] <support@github.com> --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Exposing default codec from codec service (opensearch-project#18698) * Exposing default codec from codec service Signed-off-by: bharath-techie <bharath78910@gmail.com> * addressing comments Signed-off-by: bharath-techie <bharath78910@gmail.com> --------- Signed-off-by: bharath-techie <bharath78910@gmail.com> * Replace CPU load average logic with AverageTracker class and modify default thresholds (opensearch-project#18666) Signed-off-by: Harsh Kothari <techarsh@amazon.com> * Rename all query group references to workload group (opensearch-project#18711) Signed-off-by: Ruirui Zhang <mariazrr@amazon.com> * Rename WorkloadGroupTestUtil to WorkloadManagementTestUtil (opensearch-project#18709) Signed-off-by: Ruirui Zhang <mariazrr@amazon.com> * Add Prudhvi Godithi (GH: prudhvigodithi) as a maintainer (opensearch-project#18690) * Add Prudhvi Godithi (GH: prudhvigodithi) as a maintainer Signed-off-by: Craig Perkins <cwperx@amazon.com> * Keep spaces Signed-off-by: Craig Perkins <cwperx@amazon.com> * Fix spacing Signed-off-by: Craig Perkins <cwperx@amazon.com> --------- Signed-off-by: Craig Perkins <cwperx@amazon.com> * Pass index settings to system ingest processor factories. (opensearch-project#18708) Signed-off-by: Bo Zhang <bzhangam@amazon.com> * Include named queries from rescore contexts in matched_queries array (opensearch-project#18697) * Fix: Include named queries from rescore contexts in matched_queries array - Modified QueryRescoreContext to store ParsedQuery instead of just Query - Updated QueryRescorerBuilder to use context.toQuery() for capturing named queries - Enhanced MatchedQueriesPhase to collect named queries from all rescore contexts - Added comprehensive tests for both unit and REST API scenarios - Resolves inconsistency with Elasticsearch behavior where rescore named queries were not surfaced - Cleaned up unnecessary comments for better code readability Signed-off-by: Girish Jeyakumar <girishjeyakumar@dropbox.com> * Address maintainer feedback: update RescoreContext interface design - Added parsedQuery() method to RescoreContext interface - Added getParsedQueries() method to RescoreContext interface - Updated QueryRescoreContext to override interface methods - Removed instanceof check in MatchedQueriesPhase by using interface method - Simplified assignment in QueryRescorerBuilder to one line Signed-off-by: Girish Jeyakumar <girishjeyakumar@dropbox.com> * Complete interface replacement: replace query() with parsedQuery() and getQueries() with getParsedQueries() - Removed query() method from QueryRescoreContext - Removed getQueries() method from RescoreContext interface - Updated all callers to use parsedQuery().query() instead of query() - Updated DfsPhase to use getParsedQueries() instead of getQueries() - Updated QueryRescorer to use parsedQuery().query() for rescoring and explanations - Updated QueryRescorerBuilderTests to use parsedQuery().query() - This addresses the maintainer's feedback for a cleaner interface design Signed-off-by: Girish Jeyakumar <girishjeyakumar@dropbox.com> * Finalize interface design and move changelog entry to Added section - Removed parsedQuery() method from RescoreContext base interface - Renamed getParsedQuery() to parsedQuery() in QueryRescoreContext for consistency - Updated all callers to use parsedQuery() method - Moved changelog entry to 'Added' section (more appropriate than 'Fixed') - Added proper imports for ParsedQuery to improve code readability - All tests pass and code formatting is correct Final interface design: - RescoreContext: only getParsedQueries() (returns List<ParsedQuery>) - QueryRescoreContext: parsedQuery() + getParsedQueries() override - MatchedQueriesPhase: uses getParsedQueries().forEach() for clean iteration This addresses all maintainer feedback from @mch2 and provides a clean, well-tested implementation that surfaces named queries from rescore contexts in the matched_queries array. Signed-off-by: Girish Jeyakumar <girishjeyakumar@dropbox.com> * Maintain backward compatibility for RescoreContext public API - Restore getQueries() method for backward compatibility - Add getParsedQueries() method for new functionality Signed-off-by: Girish Jeyakumar <girishjeyakumar@dropbox.com> * Fix mixed cluster test failure by updating skip version to 3.2.0 Signed-off-by: girish jeyakumar <girishjeyakumar@dropbox.com> --------- Signed-off-by: Girish Jeyakumar <girishjeyakumar@dropbox.com> Signed-off-by: girish jeyakumar <girishjeyakumar@dropbox.com> Signed-off-by: Marc Handalian <marc.handalian@gmail.com> Co-authored-by: Girish Jeyakumar <girishjeyakumar@dropbox.com> Co-authored-by: Marc Handalian <marc.handalian@gmail.com> * Move gRPC transitive dependencies to expected version (opensearch-project#18672) * Some explicit transient gRPC deps are not the expected version. grpc-netty-shaded:1.68.2 expects perfmark-api:0.27.0 guava:33.2.1-jre expects failureaccess:1.0.2 Signed-off-by: Finn Carroll <carrofin@amazon.com> * Update license shas. Signed-off-by: Finn Carroll <carrofin@amazon.com> * Changelog. Signed-off-by: Finn Carroll <carrofin@amazon.com> * Rebase changelog. Signed-off-by: Finn Carroll <carrofin@amazon.com> --------- Signed-off-by: Finn Carroll <carrofin@amazon.com> * Modify wlm logging message (opensearch-project#18712) Signed-off-by: Ruirui Zhang <mariazrr@amazon.com> * [WLM] Introduce rule cardinality check (opensearch-project#18663) Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * [WLM] Add WLM mode validation for workload group CRUD requests (opensearch-project#18652) Signed-off-by: Ruirui Zhang <mariazrr@amazon.com> * Allow upload of translog-1 generation while migrating to remote (opensearch-project#18713) Signed-off-by: Gaurav Bafna <gbbafna@amazon.com> * Fix max_score is null when using _score as a primary sort (opensearch-project#18715) * Fix max_score is null when sorting on score firstly Signed-off-by: Binlong Gao <gbinlong@amazon.com> * modify changelog Signed-off-by: Binlong Gao <gbinlong@amazon.com> * Fix test failure Signed-off-by: Binlong Gao <gbinlong@amazon.com> --------- Signed-off-by: Binlong Gao <gbinlong@amazon.com> * Changes for the Upload Workflow for merged segments (opensearch-project#18610) * Changes for the Upload Workflow for merged segments Signed-off-by: Aditi Goyal <adgoyal@amazon.com> * Addressed Todos Signed-off-by: Aditi Goyal <adgoyal@amazon.com> * Added Test Cases Signed-off-by: Aditi Goyal <adgoyal@amazon.com> * Removed the usage of Forbidden APIs Signed-off-by: Aditi Goyal <adgoyal@amazon.com> * Updated the code to include priority upload as paramter Signed-off-by: Aditi Goyal <adgoyal@amazon.com> * Updated the remote listener to use remote upploader interface Signed-off-by: Aditi Goyal <adgoyal@amazon.com> * Added JavaDocs across the changes Signed-off-by: Aditi Goyal <adgoyal@amazon.com> * Addressed spotless check Signed-off-by: Aditi Goyal <adgoyal@amazon.com> * Dummy commit(can be reverted) Signed-off-by: Aditi Goyal <adgoyal@amazon.com> * Revert "Dummy commit(can be reverted)" This reverts commit e542b99. Signed-off-by: Aditi Goyal <adgoyal@amazon.com> --------- Signed-off-by: Aditi Goyal <adgoyal@amazon.com> Co-authored-by: Aditi Goyal <adgoyal@amazon.com> * Modify merged segment pre-copy classes licenses (opensearch-project#18737) Signed-off-by: guojialiang <guojialiang.2012@bytedance.com> * `Lucene.java` Code Improvement (opensearch-project#18710) * Code Improvement Signed-off-by: Prudhvi Godithi <pgodithi@amazon.com> * Code Improvement with a new method Signed-off-by: Prudhvi Godithi <pgodithi@amazon.com> --------- Signed-off-by: Prudhvi Godithi <pgodithi@amazon.com> * Bump org.apache.commons:commons-lang3 from 3.17.0 to 3.18.0 in /plugins/repository-hdfs (opensearch-project#18745) * Bump org.apache.commons:commons-lang3 in /plugins/repository-hdfs Bumps org.apache.commons:commons-lang3 from 3.17.0 to 3.18.0. --- updated-dependencies: - dependency-name: org.apache.commons:commons-lang3 dependency-version: 3.18.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> * Updating SHAs Signed-off-by: dependabot[bot] <support@github.com> * Update changelog Signed-off-by: dependabot[bot] <support@github.com> --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump org.apache.logging.log4j:log4j-core from 2.25.0 to 2.25.1 in /buildSrc/src/testKit/thirdPartyAudit/sample_jars (opensearch-project#18744) * Bump org.apache.logging.log4j:log4j-core Bumps org.apache.logging.log4j:log4j-core from 2.25.0 to 2.25.1. --- updated-dependencies: - dependency-name: org.apache.logging.log4j:log4j-core dependency-version: 2.25.1 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> * Update changelog Signed-off-by: dependabot[bot] <support@github.com> --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump com.azure:azure-storage-common from 12.29.0 to 12.29.1 in /plugins/repository-azure (opensearch-project#18742) * Bump com.azure:azure-storage-common in /plugins/repository-azure Bumps [com.azure:azure-storage-common](https://github.com/Azure/azure-sdk-for-java) from 12.29.0 to 12.29.1. - [Release notes](https://github.com/Azure/azure-sdk-for-java/releases) - [Commits](Azure/azure-sdk-for-java@azure-storage-blob_12.29.0...azure-storage-blob_12.29.1) --- updated-dependencies: - dependency-name: com.azure:azure-storage-common dependency-version: 12.29.1 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> * Updating SHAs Signed-off-by: dependabot[bot] <support@github.com> * Update changelog Signed-off-by: dependabot[bot] <support@github.com> --------- Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: gaobinlong <gbinlong@amazon.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: gaobinlong <gbinlong@amazon.com> * Bump com.google.jimfs:jimfs from 1.3.0 to 1.3.1 in /distribution/tools/plugin-cli (opensearch-project#18743) * Bump com.google.jimfs:jimfs in /distribution/tools/plugin-cli Bumps [com.google.jimfs:jimfs](https://github.com/google/jimfs) from 1.3.0 to 1.3.1. - [Release notes](https://github.com/google/jimfs/releases) - [Commits](google/jimfs@v1.3.0...v1.3.1) --- updated-dependencies: - dependency-name: com.google.jimfs:jimfs dependency-version: 1.3.1 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> * Update changelog Signed-off-by: dependabot[bot] <support@github.com> --------- Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: gaobinlong <gbinlong@amazon.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: gaobinlong <gbinlong@amazon.com> * Fix sync method to account for block files in Composite Directory (opensearch-project#18660) Signed-off-by: Shreyansh Ray <rayshrey@amazon.com> * [Cleanup] Remove upgrade-cli tool and related build references (opensearch-project#18494) The `upgrade-cli` tool is deprecated and no longer needed. - Removed: distribution/tools/upgrade-cli - Cleaned up: settings.gradle, distribution/build.gradle - Deleted: distribution/src/bin/opensearch-upgrade(.bat) Signed-off-by: BeomSeogKim <kbs4520@daum.net> * Improve test instructions in developer guide (opensearch-project#18736) Signed-off-by: Shahbaz Aamir <shahbaz07dbit@gmail.com> * [Experimental] Start without joining a cluster if a "clusterless" ClusterPlugin is loaded (opensearch-project#18479) These are the core changes to allow ClusterPlugins to manage cluster state on behalf of an OpenSearch node. If a "clusterless" ClusterPlugin is loaded, then the node will start up with dummy versions of Discovery and ClusterManagerService, and will load the minimum cluster state required for startup (essentially, the node needs to see itself). From there, the ClusterPlugin can interact with ClusterApplierService to apply some cluster state to the node. (The details of constructing/receiving that cluster state is up to the plugin.) --------- Signed-off-by: Michael Froh <msfroh@apache.org> * Check CircuitBreaker before put/compute in FileCache to avoid entry removal (opensearch-project#18661) Signed-off-by: Shreyansh Ray <rayshrey@amazon.com> * Fix UNIX domain socket permission checks on Windows (opensearch-project#18764) Signed-off-by: Andriy Redko <drreta@gmail.com> --------- Signed-off-by: bowenlan-amzn <bowenlan23@gmail.com> Signed-off-by: Ankit Jain <akjain@amazon.com> Signed-off-by: Sayali Gaikawad <gaiksaya@amazon.com> Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: Finn Carroll <carrofin@amazon.com> Signed-off-by: kkewwei <kewei.11@bytedance.com> Signed-off-by: kkewwei <kkewwei@163.com> Signed-off-by: Craig Perkins <cwperx@amazon.com> Signed-off-by: TJ Neuenfeldt <tjneu@amazon.com> Signed-off-by: TJ Neuenfeldt <56700214+neuenfeldttj@users.noreply.github.com> Signed-off-by: Prudhvi Godithi <pgodithi@amazon.com> Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> Signed-off-by: Marc Handalian <marc.handalian@gmail.com> Signed-off-by: Du Tran <quangdutran809@gmail.com> Signed-off-by: Varun Jain <varunudr@amazon.com> Signed-off-by: Ashish Singh <ssashish@amazon.com> Signed-off-by: Andrey Pleskach <ples@aiven.io> Signed-off-by: Gagan Singh Saini <gagasa@amazon.com> Signed-off-by: Shreyansh Ray <rayshrey@amazon.com> Signed-off-by: Shruti Garg <shrugarg@amazon.com> Signed-off-by: Andriy Redko <drreta@gmail.com> Signed-off-by: Siddhant Deshmukh <deshsid@amazon.com> Signed-off-by: bharath-techie <bharath78910@gmail.com> Signed-off-by: Harsh Kothari <techarsh@amazon.com> Signed-off-by: Ruirui Zhang <mariazrr@amazon.com> Signed-off-by: Bo Zhang <bzhangam@amazon.com> Signed-off-by: Girish Jeyakumar <girishjeyakumar@dropbox.com> Signed-off-by: girish jeyakumar <girishjeyakumar@dropbox.com> Signed-off-by: Gaurav Bafna <gbbafna@amazon.com> Signed-off-by: Binlong Gao <gbinlong@amazon.com> Signed-off-by: Aditi Goyal <adgoyal@amazon.com> Signed-off-by: guojialiang <guojialiang.2012@bytedance.com> Signed-off-by: gaobinlong <gbinlong@amazon.com> Signed-off-by: BeomSeogKim <kbs4520@daum.net> Signed-off-by: Shahbaz Aamir <shahbaz07dbit@gmail.com> Signed-off-by: Michael Froh <msfroh@apache.org> Co-authored-by: bowenlan-amzn <bowenlan23@gmail.com> Co-authored-by: Ankit Jain <akjain@amazon.com> Co-authored-by: Sayali Gaikawad <gaiksaya@amazon.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Finn <carrofin@amazon.com> Co-authored-by: kkewwei <kewei.11@bytedance.com> Co-authored-by: Craig Perkins <cwperx@amazon.com> Co-authored-by: TJ Neuenfeldt <56700214+neuenfeldttj@users.noreply.github.com> Co-authored-by: Prudhvi Godithi <pgodithi@amazon.com> Co-authored-by: Kaushal Kumar <ravi.kaushal97@gmail.com> Co-authored-by: Marc Handalian <marc.handalian@gmail.com> Co-authored-by: Trần Quang Dự <quangdutran809@gmail.com> Co-authored-by: Varun Jain <varunudr@amazon.com> Co-authored-by: Ashish Singh <ssashish@amazon.com> Co-authored-by: Andrey Pleskach <ples@aiven.io> Co-authored-by: Gagan6164 <32809202+Gagan6164@users.noreply.github.com> Co-authored-by: rayshrey <121871912+rayshrey@users.noreply.github.com> Co-authored-by: Gaurav Bafna <gbbafna@amazon.com> Co-authored-by: shrugarg-amzn <shrugarg@amazon.com> Co-authored-by: Andriy Redko <drreta@gmail.com> Co-authored-by: Siddhant Deshmukh <deshsid@amazon.com> Co-authored-by: Bharathwaj G <bharath78910@gmail.com> Co-authored-by: Harsh Kothari <harsh.77424@gmail.com> Co-authored-by: Ruirui Zhang <mariazrr@amazon.com> Co-authored-by: Bo Zhang <bzhangam@amazon.com> Co-authored-by: Girish Jeyakumar <9464353+girishjeyakumar@users.noreply.github.com> Co-authored-by: Girish Jeyakumar <girishjeyakumar@dropbox.com> Co-authored-by: Gaurav Bafna <85113518+gbbafna@users.noreply.github.com> Co-authored-by: gaobinlong <gbinlong@amazon.com> Co-authored-by: Aditi Goyal <31741158+Adiiigo@users.noreply.github.com> Co-authored-by: Aditi Goyal <adgoyal@amazon.com> Co-authored-by: guojialiang <guojialiang.2012@bytedance.com> Co-authored-by: Adrian-Kim <110332047+BeomSeogKim@users.noreply.github.com> Co-authored-by: Shahbaz Aamir <shahbaz07dbit@gmail.com> Co-authored-by: Michael Froh <msfroh@apache.org>
pranikum
pushed a commit
to pranikum/OpenSearch
that referenced
this pull request
Jul 17, 2025
…ct#18610) * Changes for the Upload Workflow for merged segments Signed-off-by: Aditi Goyal <adgoyal@amazon.com> * Addressed Todos Signed-off-by: Aditi Goyal <adgoyal@amazon.com> * Added Test Cases Signed-off-by: Aditi Goyal <adgoyal@amazon.com> * Removed the usage of Forbidden APIs Signed-off-by: Aditi Goyal <adgoyal@amazon.com> * Updated the code to include priority upload as paramter Signed-off-by: Aditi Goyal <adgoyal@amazon.com> * Updated the remote listener to use remote upploader interface Signed-off-by: Aditi Goyal <adgoyal@amazon.com> * Added JavaDocs across the changes Signed-off-by: Aditi Goyal <adgoyal@amazon.com> * Addressed spotless check Signed-off-by: Aditi Goyal <adgoyal@amazon.com> * Dummy commit(can be reverted) Signed-off-by: Aditi Goyal <adgoyal@amazon.com> * Revert "Dummy commit(can be reverted)" This reverts commit e542b99. Signed-off-by: Aditi Goyal <adgoyal@amazon.com> --------- Signed-off-by: Aditi Goyal <adgoyal@amazon.com> Co-authored-by: Aditi Goyal <adgoyal@amazon.com>
tandonks
pushed a commit
to tandonks/OpenSearch
that referenced
this pull request
Aug 5, 2025
…ct#18610) * Changes for the Upload Workflow for merged segments Signed-off-by: Aditi Goyal <adgoyal@amazon.com> * Addressed Todos Signed-off-by: Aditi Goyal <adgoyal@amazon.com> * Added Test Cases Signed-off-by: Aditi Goyal <adgoyal@amazon.com> * Removed the usage of Forbidden APIs Signed-off-by: Aditi Goyal <adgoyal@amazon.com> * Updated the code to include priority upload as paramter Signed-off-by: Aditi Goyal <adgoyal@amazon.com> * Updated the remote listener to use remote upploader interface Signed-off-by: Aditi Goyal <adgoyal@amazon.com> * Added JavaDocs across the changes Signed-off-by: Aditi Goyal <adgoyal@amazon.com> * Addressed spotless check Signed-off-by: Aditi Goyal <adgoyal@amazon.com> * Dummy commit(can be reverted) Signed-off-by: Aditi Goyal <adgoyal@amazon.com> * Revert "Dummy commit(can be reverted)" This reverts commit e542b99. Signed-off-by: Aditi Goyal <adgoyal@amazon.com> --------- Signed-off-by: Aditi Goyal <adgoyal@amazon.com> Co-authored-by: Aditi Goyal <adgoyal@amazon.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
The PR aims to highlight the changes on how to segregate the syncing segments to Remote Store which is highly coupled to the Refresh workflow.
The segregation of the code will help to sync the segments at any action. Some are listed below
The overall purpose of the above mentioned feature to make the segments available as soon as possible reducing the search latency since the ingestion of the document.
Related Issues
Resolves #[Issue number to be closed when this PR is merged]
Check List
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.