[Pull-based Ingestion] Support consumer reset in Resume API #18332

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Merged

andrross merged 3 commits into opensearch-project:main from varunbharadwaj:vb/offsetupdate

May 28, 2025

Contributor

varunbharadwaj commented May 19, 2025

Description

This PR is a follow up to ingestion management API effort, to support consumer reset in the resume API. This PR includes the following changes.

Support shard pointer reset in resume API.
Include batchStartPointer in GetIngestionState API response.
Add metric to track number of duplicate messages that are skipped on rewind.
Rename reset state enum from rewind to reset. Example: REWIND_BY_OFFSET to RESET_BY_OFFSET. This will be a backward incompatible change, applicable from 3.1.0 release onwards.

Related Issues

Resolves #17442

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

github-actions bot added enhancement Indexing labels

Contributor

github-actions bot commented May 19, 2025

❌ Gradle check result for 5bde213: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

varunbharadwaj changed the title ~~[Pull-based Ingestion] Support consumer reset in Resume API~~ [WIP][Pull-based Ingestion] Support consumer reset in Resume API

varunbharadwaj force-pushed the vb/offsetupdate branch 2 times, most recently from 0bd390a to cf6b842 Compare

May 20, 2025 06:59

varunbharadwaj changed the title ~~[WIP][Pull-based Ingestion] Support consumer reset in Resume API~~ [Pull-based Ingestion] Support consumer reset in Resume API

Contributor

github-actions bot commented May 20, 2025

❌ Gradle check result for cf6b842: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

varunbharadwaj force-pushed the vb/offsetupdate branch from cf6b842 to 812d39c Compare

May 20, 2025 16:42

Contributor

github-actions bot commented May 20, 2025

❌ Gradle check result for 812d39c: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

varunbharadwaj force-pushed the vb/offsetupdate branch from 812d39c to 38fc627 Compare

May 20, 2025 17:32

Contributor

github-actions bot commented May 20, 2025

❌ Gradle check result for 38fc627: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

varunbharadwaj closed this

varunbharadwaj reopened this

Contributor

github-actions bot commented May 20, 2025

❌ Gradle check result for 38fc627: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

varunbharadwaj force-pushed the vb/offsetupdate branch from 38fc627 to 0a3b372 Compare

May 20, 2025 18:41

Contributor

github-actions bot commented May 20, 2025

❌ Gradle check result for 0a3b372: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

varunbharadwaj closed this

varunbharadwaj reopened this

Contributor

github-actions bot commented May 20, 2025

❌ Gradle check result for 0a3b372: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

varunbharadwaj closed this

varunbharadwaj reopened this

Contributor

github-actions bot commented May 20, 2025

❌ Gradle check result for 0a3b372: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

varunbharadwaj force-pushed the vb/offsetupdate branch from 0a3b372 to a862103 Compare

May 20, 2025 22:46

Contributor

github-actions bot commented May 20, 2025

❌ Gradle check result for a862103: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

varunbharadwaj force-pushed the vb/offsetupdate branch 2 times, most recently from d7c4b92 to 6948986 Compare

May 20, 2025 22:56

Contributor

github-actions bot commented May 20, 2025

❌ Gradle check result for 6948986: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

varunbharadwaj closed this

varunbharadwaj reopened this

varunbharadwaj requested review from a team, Rishikesh1159, VachaShah, owaiskazi19, peternied, reta, sachinpkale, saratvemulapalli, shwetathareja and sohami as code owners

May 28, 2025 02:16

varunbharadwaj added 3 commits

May 27, 2025 20:00


          Support shard pointer reset in resume API

587a45c

Signed-off-by: Varun Bharadwaj <varunbharadwaj1995@gmail.com>


          Undo asString interface change in consumer

b77d9bd

Signed-off-by: Varun Bharadwaj <varunbharadwaj1995@gmail.com>


          Create builder for IngestionState

c6f578b

Signed-off-by: Varun Bharadwaj <varunbharadwaj1995@gmail.com>

varunbharadwaj force-pushed the vb/offsetupdate branch from 98e0207 to c6f578b Compare

May 28, 2025 03:03

yupeng9 approved these changes

View reviewed changes

Contributor

github-actions bot commented May 28, 2025

❌ Gradle check result for c6f578b: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

varunbharadwaj closed this

varunbharadwaj reopened this

Contributor

github-actions bot commented May 28, 2025

✅ Gradle check result for c6f578b: SUCCESS

andrross approved these changes

View reviewed changes

andrross merged commit a29e6e8 into opensearch-project:main

55 of 58 checks passed

opensearch-ci-bot mentioned this pull request

[AUTOCUT] Gradle Check Flaky Test Report for SearchIT #18129

Closed

Gagan6164 pushed a commit to Gagan6164/OpenSearch that referenced this pull request


          [Pull-based Ingestion] Support consumer reset in Resume API (opensear…

82f2196

…ch-project#18332)

* Support shard pointer reset in resume API

Signed-off-by: Varun Bharadwaj <varunbharadwaj1995@gmail.com>

* Undo asString interface change in consumer

Signed-off-by: Varun Bharadwaj <varunbharadwaj1995@gmail.com>

* Create builder for IngestionState

Signed-off-by: Varun Bharadwaj <varunbharadwaj1995@gmail.com>

---------

Signed-off-by: Varun Bharadwaj <varunbharadwaj1995@gmail.com>

Gagan6164 pushed a commit to Gagan6164/OpenSearch that referenced this pull request


          [Pull-based Ingestion] Support consumer reset in Resume API (opensear…

c656559

…ch-project#18332)

* Support shard pointer reset in resume API

Signed-off-by: Varun Bharadwaj <varunbharadwaj1995@gmail.com>

* Undo asString interface change in consumer

Signed-off-by: Varun Bharadwaj <varunbharadwaj1995@gmail.com>

* Create builder for IngestionState

Signed-off-by: Varun Bharadwaj <varunbharadwaj1995@gmail.com>

---------

Signed-off-by: Varun Bharadwaj <varunbharadwaj1995@gmail.com>

BrewTestBot mentioned this pull request

opensearch 3.1.0 Homebrew/homebrew-core#228059

Merged

neuenfeldttj pushed a commit to neuenfeldttj/OpenSearch that referenced this pull request


          [Pull-based Ingestion] Support consumer reset in Resume API (opensear…

8765e42

…ch-project#18332)

* Support shard pointer reset in resume API

Signed-off-by: Varun Bharadwaj <varunbharadwaj1995@gmail.com>

* Undo asString interface change in consumer

Signed-off-by: Varun Bharadwaj <varunbharadwaj1995@gmail.com>

* Create builder for IngestionState

Signed-off-by: Varun Bharadwaj <varunbharadwaj1995@gmail.com>

---------

Signed-off-by: Varun Bharadwaj <varunbharadwaj1995@gmail.com>Signed-off-by: TJ Neuenfeldt <tjneu@amazon.com>

neuenfeldttj pushed a commit to neuenfeldttj/OpenSearch that referenced this pull request


          [Pull-based Ingestion] Support consumer reset in Resume API (opensear…

e6481bf

…ch-project#18332)

* Support shard pointer reset in resume API

Signed-off-by: Varun Bharadwaj <varunbharadwaj1995@gmail.com>

* Undo asString interface change in consumer

Signed-off-by: Varun Bharadwaj <varunbharadwaj1995@gmail.com>

* Create builder for IngestionState

Signed-off-by: Varun Bharadwaj <varunbharadwaj1995@gmail.com>

---------

Signed-off-by: Varun Bharadwaj <varunbharadwaj1995@gmail.com>

tandonks pushed a commit to tandonks/OpenSearch that referenced this pull request


          [Pull-based Ingestion] Support consumer reset in Resume API (opensear…

86de381

…ch-project#18332)

* Support shard pointer reset in resume API

Signed-off-by: Varun Bharadwaj <varunbharadwaj1995@gmail.com>

* Undo asString interface change in consumer

Signed-off-by: Varun Bharadwaj <varunbharadwaj1995@gmail.com>

* Create builder for IngestionState

Signed-off-by: Varun Bharadwaj <varunbharadwaj1995@gmail.com>

---------

Signed-off-by: Varun Bharadwaj <varunbharadwaj1995@gmail.com>

opensearch-ci-bot mentioned this pull request

[AUTOCUT] Gradle Check Flaky Test Report for IngestFromKinesisIT #17678

Closed

opensearch-ci-bot mentioned this pull request

[AUTOCUT] Gradle Check Flaky Test Report for IngestFromKafkaIT #17215

Open

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

andrross andrross approved these changes

yupeng9 yupeng9 approved these changes

anasalkouz Awaiting requested review from anasalkouz anasalkouz is a code owner

ashking94 Awaiting requested review from ashking94 ashking94 is a code owner

Bukhtawar Awaiting requested review from Bukhtawar Bukhtawar is a code owner

CEHENKLE Awaiting requested review from CEHENKLE CEHENKLE is a code owner

cwperks Awaiting requested review from cwperks cwperks is a code owner

dbwiddis Awaiting requested review from dbwiddis dbwiddis is a code owner

gbbafna Awaiting requested review from gbbafna gbbafna is a code owner

jed326 Awaiting requested review from jed326 jed326 is a code owner

kotwanikunal Awaiting requested review from kotwanikunal kotwanikunal is a code owner

mch2 Awaiting requested review from mch2 mch2 is a code owner

msfroh Awaiting requested review from msfroh msfroh is a code owner

owaiskazi19 Awaiting requested review from owaiskazi19 owaiskazi19 is a code owner

reta Awaiting requested review from reta reta is a code owner

Rishikesh1159 Awaiting requested review from Rishikesh1159 Rishikesh1159 is a code owner

sachinpkale Awaiting requested review from sachinpkale sachinpkale is a code owner

saratvemulapalli Awaiting requested review from saratvemulapalli saratvemulapalli is a code owner

shwetathareja Awaiting requested review from shwetathareja shwetathareja is a code owner

sohami Awaiting requested review from sohami sohami is a code owner

VachaShah Awaiting requested review from VachaShah

peternied Awaiting requested review from peternied peternied is a code owner

Labels

enhancement Indexing