Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

create publication repos during join task execution #16383

Open
wants to merge 2 commits into
base: main
Choose a base branch
from
Open

create publication repos during join task execution #16383

wants to merge 2 commits into from
+272 −15

Conversation

rajiv-kv
Copy link
Contributor

@rajiv-kv rajiv-kv commented Oct 18, 2024 •
edited
Loading

Description

When a node with publication repositories are configured, cluster-manager updates the repositories metadata in cluster-state after validation.

Related Issues

Resolves #16363

Check List

  • Functionality includes testing.
  • API changes companion pull request created, if applicable.
  • Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@github-actions GitHub Actions
Copy link
Contributor

❌ Gradle check result for f54d4f3: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

@github-actions GitHub Actions
Copy link
Contributor

❌ Gradle check result for 7a56884: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

@opensearch-ci-bot opensearch-ci-bot mentioned this pull request Oct 18, 2024
Open
@github-actions GitHub Actions
Copy link
Contributor

✅ Gradle check result for c23f86e: SUCCESS

@codecov Codecov
Copy link

codecov bot commented Oct 19, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 72.05%. Comparing base (0bded88) to head (c23f86e).
Report is 4 commits behind head on main.

Additional details and impacted files 
@@             Coverage Diff              @@
##               main   #16383      +/-   ##
============================================
- Coverage     72.07%   72.05%   -0.02%     
- Complexity    64819    64853      +34     
============================================
  Files          5307     5307              
  Lines        302718   302727       +9     
  Branches      43734    43736       +2     
============================================
- Hits         218178   218134      -44     
- Misses        66638    66670      +32     
- Partials      17902    17923      +21

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@rajiv-kv rajiv-kv marked this pull request as ready for review October 19, 2024 10:03
soosinha
soosinha reviewed Oct 20, 2024
View reviewed changes
server/src/main/java/org/opensearch/cluster/coordination/JoinTaskExecutor.java
@@ -191,6 +191,15 @@ public ClusterTasksResult<Task> execute(ClusterState currentState, List<Task> jo
currentState.getMetadata().custom(RepositoriesMetadata.TYPE)
);

Optional<DiscoveryNode> remotePublicationDN = currentNodes.getNodes()
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need this remotePublicationDN separately? Can we append the condition in the remoteDN logic above ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, there can be cases where the remotePublicationDN has additional repository (eg :- routingTable) as compared to remoteDN and both the nodes are present in the cluster.

@github-actions GitHub Actions
Copy link
Contributor

❌ Gradle check result for dda859c: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

@opensearch-ci-bot opensearch-ci-bot mentioned this pull request Oct 21, 2024
Open
soosinha
soosinha reviewed Oct 21, 2024
View reviewed changes
server/src/main/java/org/opensearch/cluster/coordination/JoinTaskExecutor.java
@@ -241,7 +271,7 @@ public ClusterTasksResult<Task> execute(ClusterState currentState, List<Task> jo
}
results.success(joinTask);
}

RepositoriesMetadata repositoriesMetadata = new RepositoriesMetadata(new ArrayList<>(repositories.values()));
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If repositoriesMetadata is already being computed above why is it needed to compute again here using repositories map ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The metdata computed above is only for the individual ones (RemoteDN / RemotePublicationDN). Here we are combining the repositories from both the code paths and constructing a new metadata.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is a bit of duplication currently. Can we change to below and then use the repositoriesMetadata instead of created new repositoriesMetadata.

if (remoteDN.isEmpty() && node.isRemoteStoreNode() || remotePublicationDN.isEmpty() && node.isRemoteStatePublicationEnabled()) {
    repositoriesMetadata = remoteStoreNodeService.updateRepositoriesMetadata(
        node,
        existingrepositoriesMetadata
    );
}

Other is a node is both a remote store and remote publication node, then the logic will be executed twice.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure makes sense. updated the PR.

@github-actions GitHub Actions
Copy link
Contributor

❌ Gradle check result for 4ac77b1: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

@github-actions GitHub Actions
Copy link
Contributor

❌ Gradle check result for 0d05c92: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

@opensearch-ci-bot opensearch-ci-bot mentioned this pull request Oct 18, 2024
Open
@github-actions GitHub Actions
Copy link
Contributor

❌ Gradle check result for 75032b7: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

This was referenced Oct 22, 2024
Closed
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@soosinha soosinha soosinha left review comments

@anasalkouz anasalkouz Awaiting requested review from anasalkouz anasalkouz is a code owner

@andrross andrross Awaiting requested review from andrross andrross is a code owner

@ashking94 ashking94 Awaiting requested review from ashking94 ashking94 is a code owner

@Bukhtawar Bukhtawar Awaiting requested review from Bukhtawar Bukhtawar is a code owner

@CEHENKLE CEHENKLE Awaiting requested review from CEHENKLE CEHENKLE is a code owner

@dblock dblock Awaiting requested review from dblock dblock is a code owner

@dbwiddis dbwiddis Awaiting requested review from dbwiddis dbwiddis is a code owner

@gbbafna gbbafna Awaiting requested review from gbbafna gbbafna is a code owner

@jainankitk jainankitk Awaiting requested review from jainankitk jainankitk is a code owner

@kotwanikunal kotwanikunal Awaiting requested review from kotwanikunal kotwanikunal is a code owner

@linuxpi linuxpi Awaiting requested review from linuxpi linuxpi is a code owner

@mch2 mch2 Awaiting requested review from mch2 mch2 is a code owner

@msfroh msfroh Awaiting requested review from msfroh msfroh is a code owner

@nknize nknize Awaiting requested review from nknize nknize is a code owner

@owaiskazi19 owaiskazi19 Awaiting requested review from owaiskazi19 owaiskazi19 is a code owner

@reta reta Awaiting requested review from reta reta is a code owner

@Rishikesh1159 Rishikesh1159 Awaiting requested review from Rishikesh1159 Rishikesh1159 is a code owner

@sachinpkale sachinpkale Awaiting requested review from sachinpkale sachinpkale is a code owner

@saratvemulapalli saratvemulapalli Awaiting requested review from saratvemulapalli saratvemulapalli is a code owner

@shwetathareja shwetathareja Awaiting requested review from shwetathareja shwetathareja is a code owner

@sohami sohami Awaiting requested review from sohami sohami is a code owner

@VachaShah VachaShah Awaiting requested review from VachaShah VachaShah is a code owner

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@rajiv-kv @soosinha