Skip to content

[D&M] Remote clusters cleanup #443

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 23 commits into from
Feb 25, 2025
Merged
Show file tree
Hide file tree
Changes from 13 commits
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
51e3da7
initial cleanup for remote clusters
florent-leborgne Feb 13, 2025
25bceed
Merge branch 'main' of github.com:elastic/docs-content into dm-remote…
florent-leborgne Feb 13, 2025
41693ad
refine more
florent-leborgne Feb 13, 2025
567fab3
variables
florent-leborgne Feb 13, 2025
297a4fc
undo some variables
florent-leborgne Feb 13, 2025
6757e61
shorter nav titles
florent-leborgne Feb 17, 2025
12c0e4c
more cleanup + comments
florent-leborgne Feb 17, 2025
9c78a70
Merge branch 'main' of github.com:elastic/docs-content into dm-remote…
florent-leborgne Feb 17, 2025
921d25b
missing list item
florent-leborgne Feb 17, 2025
d17aa90
more review changes
florent-leborgne Feb 17, 2025
bb2b4ff
link
florent-leborgne Feb 17, 2025
0cfb20a
Merge branch 'main' of github.com:elastic/docs-content into dm-remote…
florent-leborgne Feb 18, 2025
05bc37c
apply tags
florent-leborgne Feb 20, 2025
a982cdf
cleanup
shainaraskas Feb 24, 2025
3176a62
Merge branch 'main' of github.com:elastic/docs-content into dm-remote…
florent-leborgne Feb 24, 2025
cb8ec6f
Merge branch 'dm-remote-clusters' of github.com:florent-leborgne/docs…
florent-leborgne Feb 24, 2025
da366d6
fix troubleshooting links
shainaraskas Feb 24, 2025
913b811
little formatting thingies
shainaraskas Feb 24, 2025
e8377d1
Merge branch 'main' into dm-remote-clusters
shainaraskas Feb 24, 2025
4f59eb7
fix"
shainaraskas Feb 24, 2025
a8c53d3
Merge branch 'dm-remote-clusters' of https://github.com/florent-lebor…
shainaraskas Feb 24, 2025
c84e9e1
Merge branch 'main' into dm-remote-clusters
shainaraskas Feb 24, 2025
1bf69c9
Merge branch 'main' into dm-remote-clusters
florent-leborgne Feb 25, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
46 changes: 43 additions & 3 deletions deploy-manage/remote-clusters.md
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

a perfect page????

Original file line number Diff line number Diff line change
@@ -1,9 +1,49 @@
# Remote clusters
---
applies_to:
stack: ga
deployment:
ece: ga
eck: ga
ess: ga
self: ga
serverless: unavailable
---

# Remote clusters [remote-clusters]

% What needs to be done: Write from scratch

% GitHub issue: https://github.com/elastic/docs-projects/issues/345

% Scope notes: "Landing page for cross cluster comms, used by CCS and CCR.
We will cover here the raw configuration at Elasticsearch level and the docs to enable remote clusters in ESS / ECE / ECK.
We can include links to the use cases of remote clusters, such as CCR and CCS."
% We will cover here the raw configuration at Elasticsearch level and the docs to enable remote clusters in ESS / ECE / ECK.
% We can include links to the use cases of remote clusters, such as CCR and CCS."

By setting up **remote clusters**, you can connect an {{es}} cluster to other {{es}} clusters. Remote clusters can be located in different data centers, geographic regions, and run on a different type of environment: {{ech}}, {{ece}}, {{eck}}, or self-managed.

Remote clusters are especially useful in two cases:

- **Cross-cluster replication**
With [cross-cluster replication](/deploy-manage/tools/cross-cluster-replication.md), or CCR, you ingest data to an index on a remote cluster. This leader index is replicated to one or more read-only follower indices on your local cluster. Creating a multi-cluster architecture with cross-cluster replication enables you to configure disaster recovery, bring data closer to your users, or establish a centralized reporting cluster to process reports locally.

- **Cross-cluster search**
[Cross-cluster search](/solutions/search/cross-cluster-search.md), or CCS, enables you to run a search request against one or more remote clusters. This capability provides each region with a global view of all clusters, allowing you to send a search request from a local cluster and return results from all connected remote clusters. For full {{ccs}} capabilities, the local and remote cluster must be on the same [subscription level](https://www.elastic.co/subscriptions).

::::{note} about terminology
In the case of remote clusters, the {{es}} cluster or deployment initiating the connection and requests is often referred to as the **local cluster**, while the {{es}} cluster or deployment receiving the requests is referred to as the **remote cluster**.
::::

## Setup

Depending on the environment the local and remote clusters are deployed on and the security model you wish to use, the exact details needed to add a remote cluster vary but generally follow the same path:

1. **Configure trust between clusters.** In the settings of the local deployment or cluster, configure the trust security model that your remote connections will use to access the remote cluster. This step involves specifying API keys or certificates retrieved from the remote clusters.

2. **Establish the connection.** In {{kib}} on the local cluster, finalize the connection by specifying each remote cluster's details.

Find the instructions with details on the supported security models and available connection modes for your specific scenario:

- [Remote clusters with {{ech}}](remote-clusters/ec-enable-ccs.md)
- [Remote clusters with {{ece}}](remote-clusters/ece-enable-ccs.md)
- [Remote clusters with {{eck}}](remote-clusters/eck-remote-clusters.md)
- [Remote clusters with self-managed installations](remote-clusters/remote-clusters-self-managed.md)
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
:::::{dropdown} Version compatibility table

* Any node can communicate with another node on the same major version. For example, 9.0 can talk to any 9.x node.
* Version compatibility is symmetric, meaning that if 7.16 can communicate with 8.0, 8.0 can also communicate with 7.16. The following table depicts version compatibility between local and remote nodes.

| | |
| --- | --- |
| | Local cluster |
| Remote cluster | 5.0–5.5 | 5.6 | 6.0–6.6 | 6.7 | 6.8 | 7.0 | 7.1–7.16 | 7.17 | 8.0–9.0 |
| 5.0–5.5 | ![Yes](https://doc-icons.s3.us-east-2.amazonaws.com/icon-yes.png "") | ![Yes](https://doc-icons.s3.us-east-2.amazonaws.com/icon-yes.png "") | ![No](https://doc-icons.s3.us-east-2.amazonaws.com/icon-no.png "") | ![No](https://doc-icons.s3.us-east-2.amazonaws.com/icon-no.png "") | ![No](https://doc-icons.s3.us-east-2.amazonaws.com/icon-no.png "") | ![No](https://doc-icons.s3.us-east-2.amazonaws.com/icon-no.png "") | ![No](https://doc-icons.s3.us-east-2.amazonaws.com/icon-no.png "") | ![No](https://doc-icons.s3.us-east-2.amazonaws.com/icon-no.png "") | ![No](https://doc-icons.s3.us-east-2.amazonaws.com/icon-no.png "") |
| 5.6 | ![Yes](https://doc-icons.s3.us-east-2.amazonaws.com/icon-yes.png "") | ![Yes](https://doc-icons.s3.us-east-2.amazonaws.com/icon-yes.png "") | ![Yes](https://doc-icons.s3.us-east-2.amazonaws.com/icon-yes.png "") | ![Yes](https://doc-icons.s3.us-east-2.amazonaws.com/icon-yes.png "") | ![Yes](https://doc-icons.s3.us-east-2.amazonaws.com/icon-yes.png "") | ![No](https://doc-icons.s3.us-east-2.amazonaws.com/icon-no.png "") | ![No](https://doc-icons.s3.us-east-2.amazonaws.com/icon-no.png "") | ![No](https://doc-icons.s3.us-east-2.amazonaws.com/icon-no.png "") | ![No](https://doc-icons.s3.us-east-2.amazonaws.com/icon-no.png "") |
| 6.0–6.6 | ![No](https://doc-icons.s3.us-east-2.amazonaws.com/icon-no.png "") | ![Yes](https://doc-icons.s3.us-east-2.amazonaws.com/icon-yes.png "") | ![Yes](https://doc-icons.s3.us-east-2.amazonaws.com/icon-yes.png "") | ![Yes](https://doc-icons.s3.us-east-2.amazonaws.com/icon-yes.png "") | ![Yes](https://doc-icons.s3.us-east-2.amazonaws.com/icon-yes.png "") | ![No](https://doc-icons.s3.us-east-2.amazonaws.com/icon-no.png "") | ![No](https://doc-icons.s3.us-east-2.amazonaws.com/icon-no.png "") | ![No](https://doc-icons.s3.us-east-2.amazonaws.com/icon-no.png "") | ![No](https://doc-icons.s3.us-east-2.amazonaws.com/icon-no.png "") |
| 6.7 | ![No](https://doc-icons.s3.us-east-2.amazonaws.com/icon-no.png "") | ![Yes](https://doc-icons.s3.us-east-2.amazonaws.com/icon-yes.png "") | ![Yes](https://doc-icons.s3.us-east-2.amazonaws.com/icon-yes.png "") | ![Yes](https://doc-icons.s3.us-east-2.amazonaws.com/icon-yes.png "") | ![Yes](https://doc-icons.s3.us-east-2.amazonaws.com/icon-yes.png "") | ![Yes](https://doc-icons.s3.us-east-2.amazonaws.com/icon-yes.png "") | ![No](https://doc-icons.s3.us-east-2.amazonaws.com/icon-no.png "") | ![No](https://doc-icons.s3.us-east-2.amazonaws.com/icon-no.png "") | ![No](https://doc-icons.s3.us-east-2.amazonaws.com/icon-no.png "") |
| 6.8 | ![No](https://doc-icons.s3.us-east-2.amazonaws.com/icon-no.png "") | ![Yes](https://doc-icons.s3.us-east-2.amazonaws.com/icon-yes.png "") | ![Yes](https://doc-icons.s3.us-east-2.amazonaws.com/icon-yes.png "") | ![Yes](https://doc-icons.s3.us-east-2.amazonaws.com/icon-yes.png "") | ![Yes](https://doc-icons.s3.us-east-2.amazonaws.com/icon-yes.png "") | ![Yes](https://doc-icons.s3.us-east-2.amazonaws.com/icon-yes.png "") | ![Yes](https://doc-icons.s3.us-east-2.amazonaws.com/icon-yes.png "") | ![Yes](https://doc-icons.s3.us-east-2.amazonaws.com/icon-yes.png "") | ![No](https://doc-icons.s3.us-east-2.amazonaws.com/icon-no.png "") |
| 7.0 | ![No](https://doc-icons.s3.us-east-2.amazonaws.com/icon-no.png "") | ![No](https://doc-icons.s3.us-east-2.amazonaws.com/icon-no.png "") | ![No](https://doc-icons.s3.us-east-2.amazonaws.com/icon-no.png "") | ![Yes](https://doc-icons.s3.us-east-2.amazonaws.com/icon-yes.png "") | ![Yes](https://doc-icons.s3.us-east-2.amazonaws.com/icon-yes.png "") | ![Yes](https://doc-icons.s3.us-east-2.amazonaws.com/icon-yes.png "") | ![Yes](https://doc-icons.s3.us-east-2.amazonaws.com/icon-yes.png "") | ![Yes](https://doc-icons.s3.us-east-2.amazonaws.com/icon-yes.png "") | ![No](https://doc-icons.s3.us-east-2.amazonaws.com/icon-no.png "") |
| 7.1–7.16 | ![No](https://doc-icons.s3.us-east-2.amazonaws.com/icon-no.png "") | ![No](https://doc-icons.s3.us-east-2.amazonaws.com/icon-no.png "") | ![No](https://doc-icons.s3.us-east-2.amazonaws.com/icon-no.png "") | ![No](https://doc-icons.s3.us-east-2.amazonaws.com/icon-no.png "") | ![Yes](https://doc-icons.s3.us-east-2.amazonaws.com/icon-yes.png "") | ![Yes](https://doc-icons.s3.us-east-2.amazonaws.com/icon-yes.png "") | ![Yes](https://doc-icons.s3.us-east-2.amazonaws.com/icon-yes.png "") | ![Yes](https://doc-icons.s3.us-east-2.amazonaws.com/icon-yes.png "") | ![No](https://doc-icons.s3.us-east-2.amazonaws.com/icon-no.png "") |
| 7.17 | ![No](https://doc-icons.s3.us-east-2.amazonaws.com/icon-no.png "") | ![No](https://doc-icons.s3.us-east-2.amazonaws.com/icon-no.png "") | ![No](https://doc-icons.s3.us-east-2.amazonaws.com/icon-no.png "") | ![No](https://doc-icons.s3.us-east-2.amazonaws.com/icon-no.png "") | ![Yes](https://doc-icons.s3.us-east-2.amazonaws.com/icon-yes.png "") | ![Yes](https://doc-icons.s3.us-east-2.amazonaws.com/icon-yes.png "") | ![Yes](https://doc-icons.s3.us-east-2.amazonaws.com/icon-yes.png "") | ![Yes](https://doc-icons.s3.us-east-2.amazonaws.com/icon-yes.png "") | ![Yes](https://doc-icons.s3.us-east-2.amazonaws.com/icon-yes.png "") |
| 8.0–9.0 | ![No](https://doc-icons.s3.us-east-2.amazonaws.com/icon-no.png "") | ![No](https://doc-icons.s3.us-east-2.amazonaws.com/icon-no.png "") | ![No](https://doc-icons.s3.us-east-2.amazonaws.com/icon-no.png "") | ![No](https://doc-icons.s3.us-east-2.amazonaws.com/icon-no.png "") | ![No](https://doc-icons.s3.us-east-2.amazonaws.com/icon-no.png "") | ![No](https://doc-icons.s3.us-east-2.amazonaws.com/icon-no.png "") | ![No](https://doc-icons.s3.us-east-2.amazonaws.com/icon-no.png "") | ![Yes](https://doc-icons.s3.us-east-2.amazonaws.com/icon-yes.png "") | ![Yes](https://doc-icons.s3.us-east-2.amazonaws.com/icon-yes.png "") |


::::{important}
Elastic only supports {{ccs}} on a subset of these configurations. See [Supported {{ccs}} configurations](../../../solutions/search/cross-cluster-search.md#ccs-supported-configurations).
::::

:::::


Original file line number Diff line number Diff line change
@@ -1,4 +1,7 @@
---
applies_to:
deployment:
ess: ga
mapped_pages:
- https://www.elastic.co/guide/en/cloud/current/ec-edit-remove-trusted-environment.html
---
Expand All @@ -12,7 +15,7 @@ From a deployment’s **Security** page, you can manage trusted environments tha
* You want to remove or update the access level granted by a cross-cluster API key.


## Remove a trusted environment [ec_remove_a_trusted_environment]
## Remove a certificate-based trusted environment [ec_remove_a_trusted_environment]

By removing a trusted environment, this deployment will no longer be able to establish remote connections using certificate trust to clusters of that environment. The remote environment will also no longer be able to connect to this deployment using certificate trust.

Expand All @@ -25,11 +28,11 @@ With this method, you can only remove trusted environments relying exclusively o
2. In the list of trusted environments, locate the one you want to remove.
3. Remove it using the corresponding `delete` icon.

:::{image} ../../images/cloud-delete-trust-environment.png
:alt: button for deleting a trusted environment
:::
:::{image} ../../images/cloud-delete-trust-environment.png
:alt: button for deleting a trusted environment
:::

4. In Kibana, go to **Stack Management** > **Remote Clusters**.
4. In {{kib}}, go to **Stack Management** > **Remote Clusters**.
5. In the list of existing remote clusters, delete the ones corresponding to the trusted environment you removed earlier.


Expand All @@ -39,14 +42,14 @@ With this method, you can only remove trusted environments relying exclusively o
2. In the list of trusted environments, locate the one you want to edit.
3. Open its details by selecting the `Edit` icon.

:::{image} ../../images/cloud-edit-trust-environment.png
:alt: button for editing a trusted environment
:::
:::{image} ../../images/cloud-edit-trust-environment.png
:alt: button for editing a trusted environment
:::

4. Edit the trust configuration for that environment:

* From the **Trust level** tab, you can add or remove trusted deployments.
* From the **Environment settings** tab, you can manage the certificates and the label of the environment.
* From the **Trust level** tab, you can add or remove trusted deployments.
* From the **Environment settings** tab, you can manage the certificates and the label of the environment.

5. Save your changes.

Expand All @@ -56,28 +59,26 @@ With this method, you can only remove trusted environments relying exclusively o
This section describes the steps to change the API key used for an existing remote connection. For example, if the previous key expired and you need to rotate it with a new one.

::::{note}
If you need to update the permissions granted by a cross-cluster API key for a remote connection, you only need to update the privileges granted by the API key directly in Kibana.
If you need to update the permissions granted by a cross-cluster API key for a remote connection, you only need to update the privileges granted by the API key directly in {{kib}}.
::::


1. On the deployment you will use as remote, use the [{{es}} API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-security-create-cross-cluster-api-key) or [Kibana](../api-keys/elasticsearch-api-keys.md) to create a cross-cluster API key with the appropriate permissions. Configure it with access to the indices you want to use for {{ccs}} or {{ccr}}.
1. On the deployment you will use as remote, use the [{{es}} API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-security-create-cross-cluster-api-key) or [{{kib}}](../api-keys/elasticsearch-api-keys.md) to create a cross-cluster API key with the appropriate permissions. Configure it with access to the indices you want to use for {{ccs}} or {{ccr}}.
2. Copy the encoded key (`encoded` in the response) to a safe location. You will need it in the next steps.
3. Go to the **Security** page of the local deployment and locate the **Remote connections** section.
4. Locate the API key currently used for connecting to the remote cluster, copy its current alias, and delete it.
5. Add the new API key by selecting **Add an API key**.

* For the **Setting name**, enter the same alias that was used for the previous key.

::::{note}
If you use a different alias, you also need to re-create the remote cluster in Kibana with a **Name** that matches the new alias.
::::
::::{note}
If you use a different alias, you also need to re-create the remote cluster in {{kib}} with a **Name** that matches the new alias.
::::

* For the **Secret**, paste the encoded cross-cluster API key.
* For the **Secret**, paste the encoded cross-cluster API key, then click **Add** to save the API key to the keystore.

1. Click **Add** to save the API key to the keystore.
6. Restart the local deployment to reload the keystore with its new setting. To do that, go to the deployment’s main page (named after your deployment’s name), locate the **Actions** menu, and select **Restart {{es}}**.<br>

6. Restart the local deployment to reload the keystore with its new setting. To do that, go to the deployment’s main page (named after your deployment’s name), locate the **Actions** menu, and select **Restart Elasticsearch**.<br>

::::{note}
If the local deployment runs on version 8.13 or greater, you no longer need to perform this step because the keystore is reloaded automatically with the new API keys.
::::
::::{note}
If the local deployment runs on version 8.14 or greater, you no longer need to perform this step because the keystore is reloaded automatically with the new API keys.
::::
21 changes: 13 additions & 8 deletions deploy-manage/remote-clusters/ec-enable-ccs-for-eck.md
Original file line number Diff line number Diff line change
@@ -1,19 +1,24 @@
---
applies_to:
deployment:
ess: ga
eck: ga
navigation_title: With {{eck}}
mapped_pages:
- https://www.elastic.co/guide/en/cloud/current/ec-enable-ccs-for-eck.html
---

# Enabling CCS/R between Elasticsearch Service and ECK [ec-enable-ccs-for-eck]
# Remote clusters between {{ech}} and ECK [ec-enable-ccs-for-eck]

These steps describe how to configure remote clusters between an {{es}} cluster in Elasticsearch Service and an {{es}} cluster running within [Elastic Cloud on Kubernetes (ECK)](https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-overview.html). Once that’s done, you’ll be able to [run CCS queries from {{es}}](https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-cross-cluster-search.html) or [set up CCR](https://www.elastic.co/guide/en/elasticsearch/reference/current/ccr-getting-started-tutorial.html).
These steps describe how to configure remote clusters between an {{es}} cluster in {{ech}} and an {{es}} cluster running within [{{eck}} (ECK)](https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-overview.html). Once that’s done, you’ll be able to [run CCS queries from {{es}}](https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-cross-cluster-search.html) or [set up CCR](https://www.elastic.co/guide/en/elasticsearch/reference/current/ccr-getting-started-tutorial.html).


## Establish trust between two clusters [ec_establish_trust_between_two_clusters]
Copy link
Collaborator

@shainaraskas shainaraskas Feb 24, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

line 81: confused about what the user is supposed to do on the target page. the establishing trust procedures above also refer to these pages.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If that's what you're referring to: Those page used to be maintained by the ECK team. Only thing I know is that doing it on ECK is very close to how it's done with self-managed, which isn't very concrete information... We can make a note to improve this later.


The first step is to establish trust between the two clusters.


### Establish trust in the Elasticsearch Service cluster [ec_establish_trust_in_the_elasticsearch_service_cluster]
### Establish trust in the {{ech}} cluster [ec_establish_trust_in_the_elasticsearch_service_cluster]

1. Save the ECK CA certificate to a file. For a cluster named `quickstart`, run:

Expand All @@ -22,7 +27,7 @@ The first step is to establish trust between the two clusters.
```


1. Update the trust settings for the Elasticsearch Service deployment. Follow the steps provided in [Access clusters of a self-managed environment](ec-remote-cluster-self-managed.md), and specifically the first three steps in **Specify the deployments trusted to be used as remote clusters** using TLS certificate as security model.
1. Update the trust settings for the {{ech}} deployment. Follow the steps provided in [Access clusters of a self-managed environment](ec-remote-cluster-self-managed.md), and specifically the first three steps in **Specify the deployments trusted to be used as remote clusters** using TLS certificate as security model.

* Use the certificate file saved in the first step.
* Select the {{ecloud}} pattern and enter `default.es.local` for the `Scope ID`.
Expand All @@ -32,7 +37,7 @@ The first step is to establish trust between the two clusters.

### Establish trust in the ECK cluster [ec_establish_trust_in_the_eck_cluster]

1. Upload the Elasticsearch Service certificate (that you downloaded in the last step of the previous section) as a Kubernetes secret.
1. Upload the {{ech}} certificate (that you downloaded in the last step of the previous section) as a Kubernetes secret.

```sh
kubectl create secret generic ce-aws-cert --from-file=<path to certificate file>
Expand Down Expand Up @@ -75,14 +80,14 @@ The first step is to establish trust between the two clusters.

## Setup CCS/R [ec_setup_ccsr]

Now that trust has been established, you can set up CCS/R from the ECK cluster to the Elasticsearch Service cluster or from the Elasticsearch Service cluster to the ECK cluster.
Now that trust has been established, you can set up CCS/R from the ECK cluster to the {{ech}} cluster or from the {{ech}} cluster to the ECK cluster.


### ECK Cluster to Elasticsearch Service cluster [ec_eck_cluster_to_elasticsearch_service_cluster]
### ECK Cluster to {{ech}} cluster [ec_eck_cluster_to_elasticsearch_service_cluster]

Configure the ECK cluster [using certificate based authentication](ec-remote-cluster-self-managed.md).


### Elasticsearch Service cluster to ECK Cluster [ec_elasticsearch_service_cluster_to_eck_cluster]
### {{ech}} cluster to ECK Cluster [ec_elasticsearch_service_cluster_to_eck_cluster]

Follow the steps outlined in the [ECK documentation](https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-remote-clusters.html#k8s_configure_the_remote_cluster_connection_through_the_elasticsearch_rest_api).
Loading
Loading