Skip to content

Add docs for check external connection #19642

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 2 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
You can test the connection from each node in the cluster to your external storage with the [`CHECK EXTERNAL CONNECTION`]({% link {{ page.version.version }}/check-external-connection.md %}) statement.
6 changes: 6 additions & 0 deletions src/current/_includes/v25.1/sidebar-data/sql.json
Original file line number Diff line number Diff line change
Expand Up @@ -160,6 +160,12 @@
"/${VERSION}/cancel-session.html"
]
},
{
"title": "<code>CHECK EXTERNAL CONNECTION</code>",
"urls": [
"/${VERSION}/check-external-connection.html"
]
},
{
"title": "<code>COMMENT ON</code>",
"urls": [
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
You can test the connection from each node in the cluster to your external storage with the [`CHECK EXTERNAL CONNECTION`]({% link {{ page.version.version }}/check-external-connection.md %}) statement.
6 changes: 6 additions & 0 deletions src/current/_includes/v25.2/sidebar-data/sql.json
Original file line number Diff line number Diff line change
Expand Up @@ -166,6 +166,12 @@
"/${VERSION}/cancel-session.html"
]
},
{
"title": "<code>CHECK EXTERNAL CONNECTION</code>",
"urls": [
"/${VERSION}/check-external-connection.html"
]
},
{
"title": "<code>COMMENT ON</code>",
"urls": [
Expand Down
4 changes: 4 additions & 0 deletions src/current/v25.1/backup-validation.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,10 @@ You can validate a backup of a [cluster]({% link {{ page.version.version }}/back

The options that give the most validation coverage will increase the runtime of the check. That is, `verify_backup_table_data` will take a longer time to validate a backup compared to `check_files` or `schema_only` alone. Despite that, each of these validation options provide a quicker way to validate a backup over running a "regular" restore.

{{site.data.alerts.callout_success}}
{% include {{ page.version.version }}/backups/external-storage-check-tip.md %}
{{site.data.alerts.end}}

## Recommendations

Cockroach Labs recommends implementing the following validation plan to test your backups:
Expand Down
4 changes: 4 additions & 0 deletions src/current/v25.1/backup.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,10 @@ To view the contents of an backup created with the `BACKUP` statement, use [`SHO
- Modifying backup files in the storage location could invalidate a backup, and therefore, prevent a restore. In v22.1 and later, **we recommend enabling [object locking]({% link {{ page.version.version }}/use-cloud-storage.md %}#immutable-storage) in your cloud storage bucket.**
- While Cockroach Labs actively tests Amazon S3, Google Cloud Storage, and Azure Storage, we **do not** test [S3-compatible services]({% link {{ page.version.version }}/use-cloud-storage.md %}) (e.g., [MinIO](https://min.io/), [Red Hat Ceph](https://docs.ceph.com/en/pacific/radosgw/s3/)).

{{site.data.alerts.callout_success}}
{% include {{ page.version.version }}/backups/external-storage-check-tip.md %}
{{site.data.alerts.end}}

## Required privileges

{% include {{ page.version.version }}/backups/updated-backup-privileges.md %}
Expand Down
93 changes: 93 additions & 0 deletions src/current/v25.1/check-external-connection.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,93 @@
---
title: CHECK EXTERNAL CONNECTION
summary: Test the connection of each node to your cloud storage location.
toc: true
---

{% include_cached new-in.html version="v25.1" %} The `CHECK EXTERNAL CONNECTION` tests the connection from each node in the cluster to an external cloud storage location. `CHECK EXTERNAL CONNECTION` will measure the time it takes each node to write a file, read it, and delete it from the specified storage location. You can adjust the number and concurrency of the test runs as well as the size of the file to write and read for each test.

{{site.data.alerts.callout_info}}
`CHECK EXTERNAL CONNECTION` supports testing the connection to [**cloud storage**]({% link {{ page.version.version }}/use-cloud-storage.md %}) locations.
{{site.data.alerts.end}}

## Synopsis

<div>
{% remote_include https://raw.githubusercontent.com/cockroachdb/generated-diagrams/{{ page.release_info.crdb_branch_name }}/grammar_svg/check_external_connection.html %}
</div>

## Parameters

Parameter | Description
----------+------------
`connection_uri` | The URI to the external storage. Specify the [provider's URI]({% link {{ page.version.version }}/use-cloud-storage.md %}) (e.g., `gs://bucket_name?AUTH...`) or a user-defined [external connection]({% link {{ page.version.version }}/create-external-connection.md %}) (e.g., `external://gcs`).

## Options

Option | Value | Description
--------+-------+------------
`concurrently` | `INT` | Run multiple connection tests concurrently. If you also set the `time` option, it will run the specified number of concurrent tests until the time has elapsed. By default, only `1` connection test will run.
`time` | `STRING` | Run the test repeatedly until the duration has elapsed.
`transfer` | `STRING` | The size of the file that is written and read during each iteration of the connection test. By default, this will transfer a `32MiB` file.

## Responses

Field | Value | Description
------|-------|------------
`node` | `INT` | The node ID.
`locality` | `STRING` | The [locality]({% link {{ page.version.version }}/cockroach-start.md %}#locality) of the node.
`ok` | `BOOL` | The success of the test run.
`error` | `STRING` | Errors encountered during the test run.
`transferred` | `STRING` | The size of the file transferred during the test.
`read_speed` | `STRING` | The speed at which the node read the test file.
`write_speed` | `STRING` | The speed at which the node wrote the test file.
`can_delete` | `BOOL` | The success of file deletion.

## Examples

Specify the connection URI to the [external storage location]({% link {{ page.version.version }}/use-cloud-storage.md %}), or a created [external connection]({% link {{ page.version.version }}/create-external-connection.md %}):

{% include_cached copy-clipboard.html %}
~~~ sql
CHECK EXTERNAL CONNECTION 'external://cloud-storage';
~~~

~~~
node | locality | ok | error | transferred | read_speed | write_speed | can_delete
-------+-------------------------------------------+----+-------+-------------+-------------+-------------+-------------
1 | cloud=gce,region=us-east1,zone=us-east1-b | t | | 32 MiB | 66.17 MiB/s | 37.52 MiB/s | t
3 | cloud=gce,region=us-east1,zone=us-east1-b | t | | 32 MiB | 41.77 MiB/s | 33.55 MiB/s | t
2 | cloud=gce,region=us-east1,zone=us-east1-b | t | | 32 MiB | 14.23 MiB/s | 37.12 MiB/s | t
~~~

To modify the testing parameters, use one or a combination of the options: `concurrently`, `time`, `transfer`. For details on each, refer to [Options](#options).

{% include_cached copy-clipboard.html %}
~~~ sql
CHECK EXTERNAL CONNECTION 'external://cloud-storage' WITH transfer = '50MiB', concurrently = 5, time = '1ms';
~~~
~~~
node | locality | ok | error | transferred | read_speed | write_speed | can_delete
-------+-------------------------------------------+----+-------+-------------+-------------+-------------+-------------
2 | cloud=gce,region=us-east1,zone=us-east1-b | t | | 50 MiB | 59.85 MiB/s | 34.99 MiB/s | t
1 | cloud=gce,region=us-east1,zone=us-east1-b | t | | 50 MiB | 58.26 MiB/s | 34.91 MiB/s | t
1 | cloud=gce,region=us-east1,zone=us-east1-b | t | | 50 MiB | 57.69 MiB/s | 32.30 MiB/s | t
2 | cloud=gce,region=us-east1,zone=us-east1-b | t | | 50 MiB | 55.51 MiB/s | 33.02 MiB/s | t
3 | cloud=gce,region=us-east1,zone=us-east1-b | t | | 50 MiB | 59.29 MiB/s | 31.45 MiB/s | t
3 | cloud=gce,region=us-east1,zone=us-east1-b | t | | 50 MiB | 55.61 MiB/s | 32.58 MiB/s | t
3 | cloud=gce,region=us-east1,zone=us-east1-b | t | | 50 MiB | 61.04 MiB/s | 29.63 MiB/s | t
1 | cloud=gce,region=us-east1,zone=us-east1-b | t | | 50 MiB | 47.69 MiB/s | 34.04 MiB/s | t
1 | cloud=gce,region=us-east1,zone=us-east1-b | t | | 50 MiB | 55.66 MiB/s | 30.39 MiB/s | t
1 | cloud=gce,region=us-east1,zone=us-east1-b | t | | 50 MiB | 57.77 MiB/s | 29.64 MiB/s | t
2 | cloud=gce,region=us-east1,zone=us-east1-b | t | | 50 MiB | 44.95 MiB/s | 34.41 MiB/s | t
2 | cloud=gce,region=us-east1,zone=us-east1-b | t | | 50 MiB | 46.77 MiB/s | 33.31 MiB/s | t
2 | cloud=gce,region=us-east1,zone=us-east1-b | t | | 50 MiB | 57.64 MiB/s | 28.96 MiB/s | t
3 | cloud=gce,region=us-east1,zone=us-east1-b | t | | 50 MiB | 58.99 MiB/s | 26.65 MiB/s | t
3 | cloud=gce,region=us-east1,zone=us-east1-b | t | | 50 MiB | 15.14 MiB/s | 33.45 MiB/s | t
~~~

## See also

- [Use Cloud Storage]({% link {{ page.version.version }}/use-cloud-storage.md %})
- [Backup and Restore Overview]({% link {{ page.version.version }}/backup-and-restore-overview.md %})
- [`CREATE EXTERNAL CONNECTION`]({% link {{ page.version.version }}/create-external-connection.md %})
4 changes: 4 additions & 0 deletions src/current/v25.1/cloud-storage-authentication.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,10 @@ Cockroach Labs supports different levels of authentication to cloud storage. Whe
We recommend using IAM roles for users to authenticate to cloud storage resources. For more detail, see the assume role and workload identity sections for [Amazon S3]({% link {{ page.version.version }}/cloud-storage-authentication.md %}#set-up-amazon-s3-assume-role) and [Google Cloud Storage](cloud-storage-authentication.html?filters=gcs#set-up-google-cloud-storage-assume-role).
{{site.data.alerts.end}}

{{site.data.alerts.callout_success}}
{% include {{ page.version.version }}/backups/external-storage-check-tip.md %}
{{site.data.alerts.end}}

<div class="filters clearfix">
<button class="filter-button" data-scope="s3">Amazon S3</button>
<button class="filter-button" data-scope="gcs">Google Cloud Storage</button>
Expand Down
4 changes: 4 additions & 0 deletions src/current/v25.1/use-cloud-storage.md
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,10 @@ You can disable the use of implicit credentials when accessing external cloud st

### Example file URLs

{{site.data.alerts.callout_success}}
{% include {{ page.version.version }}/backups/external-storage-check-tip.md %}
{{site.data.alerts.end}}

Example URLs for [`BACKUP`]({% link {{ page.version.version }}/backup.md %}), [`RESTORE`]({% link {{ page.version.version }}/restore.md %}), or [`EXPORT`]({% link {{ page.version.version }}/export.md %}) given a bucket or container name of `acme-co` and an `employees` subdirectory:

Location | Example
Expand Down
4 changes: 4 additions & 0 deletions src/current/v25.2/backup-validation.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,10 @@ You can validate a backup of a [cluster]({% link {{ page.version.version }}/back

The options that give the most validation coverage will increase the runtime of the check. That is, `verify_backup_table_data` will take a longer time to validate a backup compared to `check_files` or `schema_only` alone. Despite that, each of these validation options provide a quicker way to validate a backup over running a "regular" restore.

{{site.data.alerts.callout_success}}
{% include {{ page.version.version }}/backups/external-storage-check-tip.md %}
{{site.data.alerts.end}}

## Recommendations

Cockroach Labs recommends implementing the following validation plan to test your backups:
Expand Down
4 changes: 4 additions & 0 deletions src/current/v25.2/backup.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,10 @@ To view the contents of an backup created with the `BACKUP` statement, use [`SHO
- Modifying backup files in the storage location could invalidate a backup, and therefore, prevent a restore. In v22.1 and later, **we recommend enabling [object locking]({% link {{ page.version.version }}/use-cloud-storage.md %}#immutable-storage) in your cloud storage bucket.**
- While Cockroach Labs actively tests Amazon S3, Google Cloud Storage, and Azure Storage, we **do not** test [S3-compatible services]({% link {{ page.version.version }}/use-cloud-storage.md %}) (e.g., [MinIO](https://min.io/), [Red Hat Ceph](https://docs.ceph.com/en/pacific/radosgw/s3/)).

{{site.data.alerts.callout_success}}
{% include {{ page.version.version }}/backups/external-storage-check-tip.md %}
{{site.data.alerts.end}}

## Required privileges

{% include {{ page.version.version }}/backups/updated-backup-privileges.md %}
Expand Down
93 changes: 93 additions & 0 deletions src/current/v25.2/check-external-connection.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,93 @@
---
title: CHECK EXTERNAL CONNECTION
summary: Test the connection of each node to your cloud storage location.
toc: true
---

The `CHECK EXTERNAL CONNECTION` tests the connection from each node in the cluster to an external cloud storage location. `CHECK EXTERNAL CONNECTION` will measure the time it takes each node to write a file, read it, and delete it from the specified storage location. You can adjust the number and concurrency of the test runs as well as the size of the file to write and read for each test.

{{site.data.alerts.callout_info}}
`CHECK EXTERNAL CONNECTION` supports testing the connection to [**cloud storage**]({% link {{ page.version.version }}/use-cloud-storage.md %}) locations.
{{site.data.alerts.end}}

## Synopsis

<div>
{% remote_include https://raw.githubusercontent.com/cockroachdb/generated-diagrams/{{ page.release_info.crdb_branch_name }}/grammar_svg/check_external_connection.html %}
</div>

## Parameters

Parameter | Description
----------+------------
`connection_uri` | The URI to the external storage. Specify the [provider's URI]({% link {{ page.version.version }}/use-cloud-storage.md %}) (e.g., `gs://bucket_name?AUTH...`) or a user-defined [external connection]({% link {{ page.version.version }}/create-external-connection.md %}) (e.g., `external://gcs`).

## Options

Option | Value | Description
--------+-------+------------
`concurrently` | `INT` | Run multiple connection tests concurrently. If you also set the `time` option, it will run the specified number of concurrent tests until the time has elapsed. By default, only `1` connection test will run.
`time` | `STRING` | Run the test repeatedly until the duration has elapsed.
`transfer` | `STRING` | The size of the file that is written and read during each iteration of the connection test. By default, this will transfer a `32MiB` file.

## Responses

Field | Value | Description
------|-------|------------
`node` | `INT` | The node ID.
`locality` | `STRING` | The [locality]({% link {{ page.version.version }}/cockroach-start.md %}#locality) of the node.
`ok` | `BOOL` | The success of the test run.
`error` | `STRING` | Errors encountered during the test run.
`transferred` | `STRING` | The size of the file transferred during the test.
`read_speed` | `STRING` | The speed at which the node read the test file.
`write_speed` | `STRING` | The speed at which the node wrote the test file.
`can_delete` | `BOOL` | The success of file deletion.

## Examples

Specify the connection URI to the [external storage location]({% link {{ page.version.version }}/use-cloud-storage.md %}), or a created [external connection]({% link {{ page.version.version }}/create-external-connection.md %}):

{% include_cached copy-clipboard.html %}
~~~ sql
CHECK EXTERNAL CONNECTION 'external://cloud-storage';
~~~

~~~
node | locality | ok | error | transferred | read_speed | write_speed | can_delete
-------+-------------------------------------------+----+-------+-------------+-------------+-------------+-------------
1 | cloud=gce,region=us-east1,zone=us-east1-b | t | | 32 MiB | 66.17 MiB/s | 37.52 MiB/s | t
3 | cloud=gce,region=us-east1,zone=us-east1-b | t | | 32 MiB | 41.77 MiB/s | 33.55 MiB/s | t
2 | cloud=gce,region=us-east1,zone=us-east1-b | t | | 32 MiB | 14.23 MiB/s | 37.12 MiB/s | t
~~~

To modify the testing parameters, use one or a combination of the options: `concurrently`, `time`, `transfer`. For details on each, refer to [Options](#options).

{% include_cached copy-clipboard.html %}
~~~ sql
CHECK EXTERNAL CONNECTION 'external://cloud-storage' WITH transfer = '50MiB', concurrently = 5, time = '1ms';
~~~
~~~
node | locality | ok | error | transferred | read_speed | write_speed | can_delete
-------+-------------------------------------------+----+-------+-------------+-------------+-------------+-------------
2 | cloud=gce,region=us-east1,zone=us-east1-b | t | | 50 MiB | 59.85 MiB/s | 34.99 MiB/s | t
1 | cloud=gce,region=us-east1,zone=us-east1-b | t | | 50 MiB | 58.26 MiB/s | 34.91 MiB/s | t
1 | cloud=gce,region=us-east1,zone=us-east1-b | t | | 50 MiB | 57.69 MiB/s | 32.30 MiB/s | t
2 | cloud=gce,region=us-east1,zone=us-east1-b | t | | 50 MiB | 55.51 MiB/s | 33.02 MiB/s | t
3 | cloud=gce,region=us-east1,zone=us-east1-b | t | | 50 MiB | 59.29 MiB/s | 31.45 MiB/s | t
3 | cloud=gce,region=us-east1,zone=us-east1-b | t | | 50 MiB | 55.61 MiB/s | 32.58 MiB/s | t
3 | cloud=gce,region=us-east1,zone=us-east1-b | t | | 50 MiB | 61.04 MiB/s | 29.63 MiB/s | t
1 | cloud=gce,region=us-east1,zone=us-east1-b | t | | 50 MiB | 47.69 MiB/s | 34.04 MiB/s | t
1 | cloud=gce,region=us-east1,zone=us-east1-b | t | | 50 MiB | 55.66 MiB/s | 30.39 MiB/s | t
1 | cloud=gce,region=us-east1,zone=us-east1-b | t | | 50 MiB | 57.77 MiB/s | 29.64 MiB/s | t
2 | cloud=gce,region=us-east1,zone=us-east1-b | t | | 50 MiB | 44.95 MiB/s | 34.41 MiB/s | t
2 | cloud=gce,region=us-east1,zone=us-east1-b | t | | 50 MiB | 46.77 MiB/s | 33.31 MiB/s | t
2 | cloud=gce,region=us-east1,zone=us-east1-b | t | | 50 MiB | 57.64 MiB/s | 28.96 MiB/s | t
3 | cloud=gce,region=us-east1,zone=us-east1-b | t | | 50 MiB | 58.99 MiB/s | 26.65 MiB/s | t
3 | cloud=gce,region=us-east1,zone=us-east1-b | t | | 50 MiB | 15.14 MiB/s | 33.45 MiB/s | t
~~~

## See also

- [Use Cloud Storage]({% link {{ page.version.version }}/use-cloud-storage.md %})
- [Backup and Restore Overview]({% link {{ page.version.version }}/backup-and-restore-overview.md %})
- [`CREATE EXTERNAL CONNECTION`]({% link {{ page.version.version }}/create-external-connection.md %})
4 changes: 4 additions & 0 deletions src/current/v25.2/cloud-storage-authentication.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,10 @@ Cockroach Labs supports different levels of authentication to cloud storage. Whe
We recommend using IAM roles for users to authenticate to cloud storage resources. For more detail, see the assume role and workload identity sections for [Amazon S3]({% link {{ page.version.version }}/cloud-storage-authentication.md %}#set-up-amazon-s3-assume-role) and [Google Cloud Storage](cloud-storage-authentication.html?filters=gcs#set-up-google-cloud-storage-assume-role).
{{site.data.alerts.end}}

{{site.data.alerts.callout_success}}
{% include {{ page.version.version }}/backups/external-storage-check-tip.md %}
{{site.data.alerts.end}}

<div class="filters clearfix">
<button class="filter-button" data-scope="s3">Amazon S3</button>
<button class="filter-button" data-scope="gcs">Google Cloud Storage</button>
Expand Down
4 changes: 4 additions & 0 deletions src/current/v25.2/use-cloud-storage.md
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,10 @@ You can disable the use of implicit credentials when accessing external cloud st

### Example file URLs

{{site.data.alerts.callout_success}}
{% include {{ page.version.version }}/backups/external-storage-check-tip.md %}
{{site.data.alerts.end}}

Example URLs for [`BACKUP`]({% link {{ page.version.version }}/backup.md %}), [`RESTORE`]({% link {{ page.version.version }}/restore.md %}), or [`EXPORT`]({% link {{ page.version.version }}/export.md %}) given a bucket or container name of `acme-co` and an `employees` subdirectory:

Location | Example
Expand Down
Loading