Skip to content

Update cluster migration guide for 1.4.0 #633

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Update cluster migration guide for 1.4.0 #633

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
94 changes: 64 additions & 30 deletions docs/guides/local-to-replicated.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,45 +1,46 @@
---
title: "Migrating a single-node to a multi-node deployment"
description: "Learn how to migrate from a single node using the local loglet and metadata to multiple nodes using the replicated loglet and metadata."
description: "Learn how to migrate from a single node to a multi-node cluster using the replicated loglet and replicated metadata server."
pagination_next: null
pagination_prev: null
---

import Admonition from '@theme/Admonition';
import {Step} from "../../src/components/Stepper";

# Migrating a single-node to a multi-node deployment

By default, when starting a Restate server ({'<='} v1.2), it uses the local loglet and local metadata server.
The local loglet and local metadata server are suitable for development and single-node deployments.
However, for production deployments, it is recommended to use the replicated loglet and replicated metadata server to ensure high availability and durability.
This guide shows how to migrate an existing single-node deployment to use the replicated loglet and replicated metadata server without losing any data.
By using the replicated loglet and replicated metadata server, you can also scale your deployment to multiple nodes.
<Admonition type="warning">
You should [upgrade](/operate/upgrading) your single-node deployment to the latest Restate version before adding more nodes.
</Admonition>

This guide assumes you have a running single-node Restate server that uses the local loglet and local metadata server.
This guide shows how to scale an existing single-node deployment to a multi-node cluster. It assumes you have a running single-node Restate server that is running the replicated loglet and replicated metadata server, which are enabled by default in Restate >= 1.4.0.

<Step stepLabel="1" title="Enable the replicated metadata server">
Older versions of Restate ({'<='} 1.3.2) use the local loglet and local metadata server by default. The local loglet and local metadata server are suitable for development and single-node deployments. We recommend using the replicated loglet and replicated metadata server to ensure high availability and durability. They are also required for multi-node clusters. Starting with version 1.4.0, existing logs and metadata will be automatically migrated to the replicated equivalents.

To use the replicated metadata server, you need to change the metadata server type to `replicated` in the Restate configuration file.
Once you restart your Restate server, it will start using the replicated metadata server and automatically migrate the local metadata.
<Step stepLabel="1" title="Verify that node is running the replicated metadata server">

```toml restate.toml
[metadata-server]
type = "replicated"
Check that the metadata service is running using the [`restatectl`](/operate/clusters#controlling-clusters-with-restatectl) tool.

```shell
restatectl metadata-server list-servers
```

If you plan to extend your single-node deployment to a multi-node deployment, you also need to [configure the snapshot repository](/operate/snapshots).
This allows new nodes to join the cluster by restoring the latest snapshot.
You should see a single member node providing metadata service:

```toml restate.toml
[worker.snapshots]
destination = "s3://snapshots-bucket/cluster-prefix"
```
Metadata service
NODE STATUS VERSION LEADER MEMBERS APPLIED COMMITTED TERM LOG-LENGTH SNAP-INDEX SNAP-SIZE
N1 Member v1 N1 [N1] 2 2 2 1 1 6.7 KiB
```

If you see the node as unreachable with an error reason of "Unimplemented", verify that you are running the latest version. The older local metadata server is no longer supported in Restate 1.4.0 and newer.

</Step>

<Step stepLabel="2" title="Enable the replicated loglet">
<Step stepLabel="2" title="Verify that node is running the replicated loglet">

To be able to use the replicated loglet, make sure the node includes the `log-server` role if you configure the roles explicitly.
The default configuration does not require adjustments to be cluster-ready. However, if you have explicitly specified which roles to run, you should make sure the configuration includes the `log-server` role. Similarly, make sure that you do not explicitly set the loglet provider type to `local`. While the local logelt is still supported, the default type is `replicated` starting from 1.4.0. Without these settings, the automatic loglet type migration will not take place.

```toml restate.toml
roles = [
Expand All @@ -48,10 +49,35 @@ roles = [
"metadata-server",
"log-server"
]

[bifrost]
default-provider = "replicated"
```

Confirm that cluster configuration uses the replicated loglet as the default log provider.

```shell
restatectl config get
```

In the default configuration you should expect to see:

```
⚙️ Cluster Configuration
├ Number of partitions: 24
├ Partition replication: {node: 1}
└ Logs Provider: replicated
├ Log replication: {node: 1}
└ Nodeset size: 1
```

You can confirm that the type of logs in use by the server using the command:

```shell
restatectl logs list
```

To use the replicated loglet, you need to reconfigure the cluster configuration to use the replicated loglet as the default log provider.
You can do this by using [`restatectl`](/operate/clusters#controlling-clusters-with-restatectl).
If you have enabled the `log-server` role and left the default provider unset (or set it to `replicated`), and still do not see the cluster configuration you can change the cluster log configuration using:

```shell
restatectl config set --log-provider replicated --log-replication 1
Expand All @@ -60,14 +86,17 @@ restatectl config set --log-provider replicated --log-replication 1
This command sets the default log provider to `replicated` with a default replication of `1`.
As long as you have a single-node deployment, you must set the replication to `1`.
Otherwise, the server will become unavailable because it cannot provision the new log segments.

</Step>

<Step stepLabel="3" title="Verify the changes">
<Step stepLabel="3" title="Configure snapshot repository">

You can verify that the changes have been applied by checking the cluster status.
If you plan to extend your single-node deployment to a multi-node deployment, you also need to [configure the snapshot repository](/operate/snapshots).
This allows new nodes to join the cluster by restoring the latest snapshot.

```shell
restatectl status
```toml restate.toml
[worker.snapshots]
destination = "s3://snapshots-bucket/cluster-prefix"
```

</Step>
Expand All @@ -78,24 +107,29 @@ For other nodes to join, you need to snapshot every partition because the local
Run the following command to create a snapshot for each partition.

```shell
restatectl snapshots create
restatectl snapshots create --trim-log
```

Note that this also instructs Restate to trim the logs after partition state has been successfully published to the snapshot repository. This ensures that the logs no longer reference historic local loglets that may have existed on the original node.

</Step>

<Step stepLabel="5" title="Turn a single-node into a multi-node deployment">

To add more nodes to your cluster, you need to start new Restate servers with the same `cluster-name` and a metadata client with the address of the existing node.
If you want the new nodes to participate in the replicated metadata cluster, then they should run the `metadata-server` role and use the `replicated` metadata server.

```toml restate.toml
cluster-name = "my-cluster"

[metadata-client]
addresses = ["ADVERTISED_ADDRESS_OF_EXISTING_NODE"]
```

If you want the new nodes to participate in the replicated metadata service, then they should run the `metadata-server` role. To make nodes resilient to individual metadata service failures, specify the full list of addresses of all the nodes running the metadata service on your Restate cluster nodes.

[metadata-server]
type = "replicated"
```toml restate.toml
[metadata-client]
addresses = ["NODE_1_ADVERTISED_ADDRESS", "NODE_2_ADVERTISED_ADDRESS", "NODE_3_ADVERTISED_ADDRESS"]
```

</Step>
Expand Down
4 changes: 2 additions & 2 deletions docs/operate/upgrading.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,12 +7,12 @@ import Admonition from '@theme/Admonition';

# Upgrading

Restate follows [Semantic Versioning](https://semver.org/). The server persists compatibility markers which enable it to detect incompatible data versions. However, you should be careful to follow supported version migration paths and perform [data backups](data-backup) when performing software upgardes.
Restate follows [Semantic Versioning](https://semver.org/). The server persists compatibility markers which enable it to detect incompatible data versions. However, you should be careful to follow supported version migration paths and perform [data backups](data-backup) when performing software upgrades.

## Version compatibility

<Admonition type="note">
Consult the release notes for the specific details of any new version when planning upgrades. As with all critical infrastructure changes, we recommend that you always test the upgrade path in an isolated environment.
Consult the release notes for the specific details of any new version when planning upgrades. As with all critical infrastructure changes, we recommend that you always verify the upgrade path in an isolated test environment first.
</Admonition>

Upgrading to the latest patch version should always be possible and is recommended to benefit from the latest bug fixes and enhancements.
Expand Down
Loading