Skip to content

Commit 89c9cbe

Browse files
loicmathieuanna-gellerwrussell1999
authored
doc: backup and restore documentation (#1688)
* doc: backup and restore documentation * review and add MySQL * Update backup-and-restore.md --------- Co-authored-by: Anna Geller <anna.m.geller@gmail.com> Co-authored-by: Will Russell <will@wrussell.co.uk>
1 parent 9d4e039 commit 89c9cbe

File tree

1 file changed

+93
-0
lines changed

1 file changed

+93
-0
lines changed
Lines changed: 93 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,93 @@
1+
---
2+
title: Backup & Restore
3+
icon: /docs/icons/admin.svg
4+
---
5+
6+
Backup Kestra.
7+
8+
Kestra does not provide specific tools for backup and restore. However, Kestra uses a database and internal storage that can be backed up and restored.
9+
10+
## Backup & Restore with the JDBC Backend
11+
12+
With the JDBC backend, Kestra can be backed up and restored using the database's native backup tools.
13+
14+
### Backup & Restore for PostgreSQL
15+
16+
First, stop Kestra to ensure the database is in a stable state. Although `pg_dump` allows you to back up a running PostgreSQL database, it's always better to perform backups offline when possible.
17+
18+
Next, run the following command:
19+
20+
```shell
21+
pg_dump -h localhost -p 5432 -U <username> -d <database> -F tar -f kestra.tar
22+
```
23+
24+
To restore the backup to a new database, use `pg_restore`:
25+
26+
```shell
27+
pg_restore -h localhost -p 5432 -U <username> -d <database> kestra.tar
28+
```
29+
30+
Finally, restart Kestra.
31+
32+
### Backup & Restore for MySQL
33+
34+
First, stop Kestra to ensure the database is in a stable state. Although MySQL's `mysqldump` allows you to back up a running MySQL database, it's always better to perform backups offline when possible.
35+
36+
Next, run the following command to back up the database:
37+
38+
```shell
39+
mysqldump -h localhost -P 3306 -u <username> -p<password> <database> > kestra.sql
40+
```
41+
42+
To restore the backup to a new database, use the following command:
43+
44+
```shell
45+
mysql -h localhost -P 3306 -u <username> -p <password> <database> < kestra.sql
46+
```
47+
48+
The `< kestra.sql` part tells MySQL to read and execute the SQL statements contained in the `kestra.sql` backup file as input.
49+
50+
Finally, restart Kestra.
51+
52+
---
53+
54+
## Backup & Restore with the Elasticsearch and Kafka Backend
55+
56+
With the Elasticsearch and Kafka backend, Kestra can be backed up and restored using Elasticsearch snapshots. Kafka will be reinitialized with the information from Elasticsearch.
57+
58+
This guide assumes you have already configured a snapshot repository in Elasticsearch named `my_snapshot_repository`. Elasticsearch provides several [backup options](https://www.elastic.co/guide/en/elasticsearch/reference/current/snapshot-restore.html). We will leverage basic snapshot and restore operations using the Elasticsearch API.
59+
60+
First, create an Elasticsearch snapshot named `kestra`:
61+
62+
```
63+
PUT _snapshot/my_snapshot_repository/kestra?wait_for_completion=true
64+
```
65+
66+
Next, delete all Kestra indices and recreate them using the snapshot:
67+
68+
```
69+
POST _snapshot/my_snapshot_repository/kestra/_restore
70+
{
71+
"indices": "kestra_*"
72+
}
73+
```
74+
75+
If you need to start from a fresh Kafka instance, you can reindex Kafka from the data in Elasticsearch using the following Kestra command:
76+
77+
```shell
78+
kestra sys restore-queue
79+
```
80+
81+
Since some execution information is stored only in Kafka, not all pending executions may be restarted.
82+
83+
Finally, restart Kestra.
84+
85+
## Backup & Restore of Internal Storage
86+
87+
Kestra's internal storage can be implemented using either a local filesystem or an object storage service.
88+
89+
If you're using the local filesystem implementation, you can back up and restore the directory with standard filesystem backup tools.
90+
91+
If you're using an object storage service provided by a cloud provider, you can replicate a bucket across multiple locations. Usually, this option is sufficient for disaster recovery. Alternatively, refer to your cloud provider's documentation for instructions on external bucket backup or replication to a different region.
92+
93+
If you're running your own object storage (e.g., using MinIO from our Helm Chart), you can [leverage Restic](https://blog.min.io/back-up-restic-minio/) or set up a replication strategy that fits your needs.

0 commit comments

Comments
 (0)