Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Delete pit 1 #7

Open
wants to merge 77 commits into
base: main
Choose a base branch
from
Open

Delete pit 1 #7

wants to merge 77 commits into from

Conversation

bharath-techie
Copy link
Owner

Description

[Describe what this change achieves]

Issues Resolved

[List any issues this PR will resolve]

Check List

  • New functionality includes testing.
    • All tests pass
  • New functionality has been documented.
    • New functionality has javadoc added
  • Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

bharath-techie and others added 30 commits May 19, 2022 11:08
@bharath-techie
Create PIT API (opensearch-project#2745)
4a1cbd9 
* Create Point In Time API changes

Signed-off-by: Bharathwaj G <bharath78910@gmail.com>
@bharath-techie
Delete PIT changes
25cc2b1 
Signed-off-by: Bharathwaj G <bharath78910@gmail.com>
@bharath-techie
Delete PIT changes - adding tests for invalid ids
724dffc 
Signed-off-by: Bharathwaj G <bharath78910@gmail.com>
@bharath-techie
Addressing comments
e482053 
Signed-off-by: Bharathwaj G <bharath78910@gmail.com>
@bharath-techie
Addressing comments
233b15d 
Signed-off-by: Bharathwaj G <bharath78910@gmail.com>
@bharath-techie
Addressing comments - changing delete pit response structure
28d8605 
Signed-off-by: Bharathwaj G <bharath78910@gmail.com>
@bharath-techie
Addressing comments
e06bac7 
Signed-off-by: Bharathwaj G <bharath78910@gmail.com>
@dependabot @Bukhtawar
Bump reactor-netty-core from 1.0.16 to 1.0.19 in /plugins/repository-…
22eb286 
…azure (opensearch-project#3360)

* Bump reactor-netty-core in /plugins/repository-azure

Bumps [reactor-netty-core](https://github.com/reactor/reactor-netty) from 1.0.16 to 1.0.19.
- [Release notes](https://github.com/reactor/reactor-netty/releases)
- [Commits](reactor/reactor-netty@v1.0.16...v1.0.19)

---
updated-dependencies:
- dependency-name: io.projectreactor.netty:reactor-netty-core
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* Updating SHAs

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com>
@dreamer-89 @Bukhtawar
[Type removal] _type removal from mocked responses of scroll hit tests (
f5fc840 
opensearch-project#3377)

Signed-off-by: Suraj Singh <surajrider@gmail.com>
@dreamer-89 @Bukhtawar
[Type removal] Remove _type deprecation from script and conditional p…
f14e041 
…rocessor (opensearch-project#3239)

* [Type removal] Remove _type deprecation from script and conditional processor

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Spotless check apply

Signed-off-by: Suraj Singh <surajrider@gmail.com>
@dreamer-89 @Bukhtawar
[Type removal] Remove _type from _bulk yaml test, scripts, unused con…
2ffc292 
…stants (opensearch-project#3372)

* [Type removal] Remove redundant _type deprecation checks in bulk request

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* [Type removal] bulk yaml tests validating deprecation on _type and removal from scripts

Signed-off-by: Suraj Singh <surajrider@gmail.com>
@mch2 @Bukhtawar
Fix Lucene-snapshots repo for jdk 17. (opensearch-project#3396)
833010a 
Signed-off-by: Marc Handalian <handalm@amazon.com>
@Bukhtawar
Replace internal usages of 'master' term in 'server/src/internalClust…
44ceeed 
…erTest' directory (opensearch-project#2521)

Signed-off-by: Tianli Feng <ftianli@amazon.com>
@reta @Bukhtawar
[REMOVE] Cleanup deprecated thread pool types (FIXED_AUTO_QUEUE_SIZE) (…
a70edc6 
…opensearch-project#3369)

Signed-off-by: Andriy Redko <andriy.redko@aiven.io>
@dreamer-89 @Bukhtawar
[Type removal] _type removal from tests of yaml tests (opensearch-pro…
326a1a8 
…ject#3406)

* [Type removal] _type removal from tests of yaml tests

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Fix spotless failures

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Fix assertion failures

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Fix assertion failures in DoSectionTests

Signed-off-by: Suraj Singh <surajrider@gmail.com>
@adnapibar @Bukhtawar
Add release notes for version 2.0.0 (opensearch-project#3410)
c5a3e01 
Signed-off-by: Rabi Panda <adnapibar@gmail.com>
@nknize @Bukhtawar
[Upgrade] Lucene-9.2.0-snapshot-ba8c3a8 (opensearch-project#3416)
72a65ae 
Upgrades to latest snapshot of lucene 9.2.0 in preparation for GA release.

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
@adnapibar @Bukhtawar
Fix release notes for 2.0.0-rc1 version (opensearch-project#3418)
cd82593 
This change removes some old commits from the 2.0.0-rc1 release notes. These commits were already released as part of 1.x releases.

Add back some missing type removal commits to the 2.0.0 release notes

Signed-off-by: Rabi Panda <adnapibar@gmail.com>
@nknize @Bukhtawar
Bump version 2.1 to Lucene 9.2 after upgrade (opensearch-project#3424)
8cae3a3 
Bumps Version.V_2_1_0 lucene version to 9.2 after backporting upgrage.

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
@dependabot @Bukhtawar
Bump com.gradle.enterprise from 3.10 to 3.10.1 (opensearch-project#3425)
c0f829c 
Bumps com.gradle.enterprise from 3.10 to 3.10.1.

---
updated-dependencies:
- dependency-name: com.gradle.enterprise
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
@dependabot @Bukhtawar
Bump reactor-core from 3.4.17 to 3.4.18 in /plugins/repository-azure (o…
1f1526b 
…pensearch-project#3427)

Bumps [reactor-core](https://github.com/reactor/reactor-core) from 3.4.17 to 3.4.18.
- [Release notes](https://github.com/reactor/reactor-core/releases)
- [Commits](reactor/reactor-core@v3.4.17...v3.4.18)

---
updated-dependencies:
- dependency-name: io.projectreactor:reactor-core
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com>
@dependabot @Bukhtawar
Bump gax-httpjson from 0.101.0 to 0.103.1 in /plugins/repository-gcs (o…
3a9b37a 
…pensearch-project#3426)

Bumps [gax-httpjson](https://github.com/googleapis/gax-java) from 0.101.0 to 0.103.1.
- [Release notes](https://github.com/googleapis/gax-java/releases)
- [Changelog](https://github.com/googleapis/gax-java/blob/main/CHANGELOG.md)
- [Commits](https://github.com/googleapis/gax-java/commits)

---
updated-dependencies:
- dependency-name: com.google.api:gax-httpjson
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com>
@Poojita-Raj @Bukhtawar
[segment replication]Introducing common Replication interfaces for se…
146b334 
…gment replication and recovery code paths (opensearch-project#3234)

* RecoveryState inherits from ReplicationState + RecoveryTarget inherits from ReplicationTarget

Signed-off-by: Poojita Raj <poojiraj@amazon.com>

* Refactoring: mixedClusterVersion error fix + move Stage to ReplicationState

Signed-off-by: Poojita Raj <poojiraj@amazon.com>

* pull ReplicationListener into a top level class + add javadocs + address review comments

Signed-off-by: Poojita Raj <poojiraj@amazon.com>

* fix javadoc

Signed-off-by: Poojita Raj <poojiraj@amazon.com>

* review changes

Signed-off-by: Poojita Raj <poojiraj@amazon.com>

* Refactoring the hierarchy relationship between repl and recovery

Signed-off-by: Poojita Raj <poojiraj@amazon.com>

* style fix

Signed-off-by: Poojita Raj <poojiraj@amazon.com>

* move package common under replication

Signed-off-by: Poojita Raj <poojiraj@amazon.com>

* rename to replication

Signed-off-by: Poojita Raj <poojiraj@amazon.com>

* rename and doc changes

Signed-off-by: Poojita Raj <poojiraj@amazon.com>
@dreamer-89 @Bukhtawar
[Type removal] Remove type from BulkRequestParser (opensearch-project…
f0d2033 
…#3423)

* [Type removal] Remove type handling in bulk request parser

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* [Type removal] Remove testTypesStillParsedForBulkMonitoring as it is no longer present in codebase

Signed-off-by: Suraj Singh <surajrider@gmail.com>
@Rishikesh1159 @Bukhtawar
Adding CheckpointRefreshListener to trigger when Segment replication …
87d354a 
…is turned on and Primary shard refreshes (opensearch-project#3108)

* Intial PR adding classes and tests related to checkpoint publishing

Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com>

* Putting a Draft PR with all changes in classes. Testing is still not included in this commit.

Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com>

* Wiring up index shard to new engine, spotless apply and removing unnecessary tests and logs

Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com>

* Adding Unit test for checkpointRefreshListener

Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com>

* Applying spotless check

Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com>

* Fixing import statements *

Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com>

* removing unused constructor in index shard

Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com>

* Addressing comments from last commit

Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com>

* Adding package-info.java files for two new packages

Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com>

* Adding test for null checkpoint publisher and addreesing PR comments

Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com>

* Add docs for indexshardtests and remove shard.refresh

Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com>
@mch2 @Bukhtawar
Add a new Engine implementation for replicas with segment replication…
052e960 
… enabled. (opensearch-project#3240)

* Change fastForwardProcessedSeqNo method in LocalCheckpointTracker to persisted checkpoint.

This change inverts fastForwardProcessedSeqNo to fastForwardPersistedSeqNo for use in
Segment Replication.  This is so that a Segrep Engine can match the logic of InternalEngine
where the seqNo is incremented with each operation, but only persisted in the tracker on a flush.
With Segment Replication we bump the processed number with each operation received index/delete/noOp, and
invoke this method when we receive a new set of segments to bump the persisted seqNo.

Signed-off-by: Marc Handalian <handalm@amazon.com>

* Extract Translog specific engine methods into an abstract class.

This change extracts translog specific methods to an abstract engine class so that other engine
implementations can reuse translog logic.

Signed-off-by: Marc Handalian <handalm@amazon.com>

* Add a separate Engine implementation for replicas with segment replication enabled.

This change adds a new engine intended to be used on replicas with segment replication enabled.
This engine does not wire up an IndexWriter, but still writes all operations to a translog.
The engine uses a new ReaderManager that refreshes from an externally provided SegmentInfos.

Signed-off-by: Marc Handalian <handalm@amazon.com>

* Fix spotless checks.

Signed-off-by: Marc Handalian <handalm@amazon.com>

* Fix :server:compileInternalClusterTestJava compilation.

Signed-off-by: Marc Handalian <handalm@amazon.com>

* Fix failing test naming convention check.

Signed-off-by: Marc Handalian <handalm@amazon.com>

* PR feedback.

- Removed isReadOnlyReplica from overloaded constructor and added feature flag checks.
- Updated log msg in NRTReplicationReaderManager
- cleaned up store ref counting in NRTReplicationEngine.

Signed-off-by: Marc Handalian <handalm@amazon.com>

* Fix spotless check.

Signed-off-by: Marc Handalian <handalm@amazon.com>

* Remove TranslogAwareEngine and build translog in NRTReplicationEngine.

Signed-off-by: Marc Handalian <handalm@amazon.com>

* Fix formatting

Signed-off-by: Marc Handalian <handalm@amazon.com>

* Add missing translog methods to NRTEngine.

Signed-off-by: Marc Handalian <handalm@amazon.com>

* Remove persistent seqNo check from fastForwardProcessedSeqNo.

Signed-off-by: Marc Handalian <handalm@amazon.com>

* PR feedback.

Signed-off-by: Marc Handalian <handalm@amazon.com>

* Add test specific to translog trimming.

Signed-off-by: Marc Handalian <handalm@amazon.com>

* Javadoc check.

Signed-off-by: Marc Handalian <handalm@amazon.com>

* Add failEngine calls to translog methods in NRTReplicationEngine.
Roll xlog generation on replica when a new commit point is received.

Signed-off-by: Marc Handalian <handalm@amazon.com>
@Bukhtawar
Rename master to cluster_manager in the XContent Parser of ClusterHea…
1d9e369 
…lthResponse (opensearch-project#3432)

Signed-off-by: Tianli Feng <ftianli@amazon.com>
@dependabot @Bukhtawar
Bump hadoop-minicluster in /test/fixtures/hdfs-fixture (opensearch-pr…
6251f27 
…oject#3359)

Bumps hadoop-minicluster from 3.3.2 to 3.3.3.

---
updated-dependencies:
- dependency-name: org.apache.hadoop:hadoop-minicluster
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
@dependabot @Bukhtawar
Bump avro from 1.10.2 to 1.11.0 in /plugins/repository-hdfs (opensear…
d8a64e7 
…ch-project#3358)

* Bump avro from 1.10.2 to 1.11.0 in /plugins/repository-hdfs

Bumps avro from 1.10.2 to 1.11.0.

---
updated-dependencies:
- dependency-name: org.apache.avro:avro
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* Updating SHAs

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com>
@Bukhtawar
Fix testSetAdditionalRolesCanAddDeprecatedMasterRole() by removing th…
12bf60f 
…e initial assertion (opensearch-project#3441)

Signed-off-by: Tianli Feng <ftianli@amazon.com>
opensearch-trigger-bot bot and others added 7 commits June 20, 2022 12:26
@opensearch-trigger-bot @opensearch-ci-bot @Bukhtawar
Added bwc version 2.0.2 (opensearch-project#3613)
5b3a164 
Co-authored-by: opensearch-ci-bot <opensearch-ci-bot@users.noreply.github.com>
@aabukhalil @Bukhtawar
Fix false positive query timeouts due to using cached time (opensearc…
bc3dffb 
…h-project#3454)

* Fix false positive query timeouts due to using cached time

Signed-off-by: Ahmad AbuKhalil <abukhali@amazon.com>

* delegate nanoTime call to SearchContext

Signed-off-by: Ahmad AbuKhalil <abukhali@amazon.com>

* add override to SearchContext getRelativeTimeInMillis to force non cached time

Signed-off-by: Ahmad AbuKhalil <abukhali@amazon.com>
@mch2 @Bukhtawar
Fix random gradle check failure issue 3584. (opensearch-project#3627)
75b86b9
@bharath-techie
Merge branch 'feature/point_in_time' of https://github.com/opensearch…
6200228 
…-project/OpenSearch into delete_pit
@bharath-techie
Re-adding code which was removed
c3f4730 
Signed-off-by: Bharathwaj G <bharath78910@gmail.com>
@bharath-techie
Addressing comments - adding tests
93f8db7 
Signed-off-by: Bharathwaj G <bharath78910@gmail.com>
@bharath-techie
Merge branch 'main' of https://github.com/opensearch-project/OpenSearch…
6b71c40 
… into delete_pit_1
@bharath-techie bharath-techie force-pushed the delete_pit_1 branch 2 times, most recently from 93ccc5a to 12bbcc5 Compare June 27, 2022 06:31
@bharath-techie
resolving conflicts
20e5216 
Signed-off-by: Bharathwaj G <bharath78910@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.