Restore backup error

[gavishdewangan]

I tried restoring backup but got this error
one of restoreDataRegular go-routine return error: can't attach data parts for table 'table': table can't restore cause system.replicas entries already exists and replication in progress from another replica, log_pointer=12, log_max_index=11, absolute_delay=0, queue_size=0

When do we get this error?

[Slach]

Did you try to restore in multiple replicas and shards cluster?
look properly restore commands sequence https://github.com/Altinity/clickhouse-backup/tree/master/Examples.md#RESTORE

[gavishdewangan]

We use Kubernetes.

these are the records, I have changed some details for security reasons though

│ chi-clickhouse-server-cnxcluster-01-0-0-0 │ create_remote --env=REMOTE_STORAGE=sftp --env=SFTP_ADDRESS=tp --env=SFTP_USERNAME=username --env=SFTP_KEY=/path --env=SFTP_PORT=22 --env=SFTP_PATH=/tmp/gd/BACKUP_20250416_092621_141/clickhouse/clickhouse_20250416_092621 clickhouse_20250416_092621_0                │ 2025-04-16 09:26:52 │ 2025-04-16 09:26:56 │ success │                                                                                                                                                                                                                                                            │
│ chi-clickhouse-server-cnxcluster-01-1-0-0 │ create_remote --env=REMOTE_STORAGE=sftp --env=SFTP_ADDRESS=tp --env=SFTP_USERNAME=username --env=SFTP_KEY=/path --env=SFTP_PORT=22 --env=SFTP_PATH=/tmp/gd/BACKUP_20250416_092621_141/clickhouse/clickhouse_20250416_092621 clickhouse_20250416_092621_1                │ 2025-04-16 09:26:52 │ 2025-04-16 09:26:56 │ success │                                                                                                                                                                                                                                                            │
│ chi-clickhouse-server-cnxcluster-01-0-0-0 │ delete local clickhouse_20250416_092621_0                                                                                                                                                                                                                                           │ 2025-04-16 09:27:52 │ 2025-04-16 09:28:00 │ success │                                                                                                                                                                                                                                                            │
│ chi-clickhouse-server-cnxcluster-01-1-0-0 │ delete local clickhouse_20250416_092621_1                                                                                                                                                                                                                                           │ 2025-04-16 09:28:00 │ 2025-04-16 09:28:08 │ success │                                                                                                                                                                                                                                                            │
│ chi-clickhouse-server-cnxcluster-01-0-0-0 │ restore_remote --rm --schema --env=REMOTE_STORAGE=sftp --env=SFTP_ADDRESS=tp --env=SFTP_USERNAME=username --env=SFTP_KEY=/path --env=SFTP_PORT=22 --env=SFTP_PATH=/tmp/gd/BACKUP_20250416_092621_141/clickhouse/clickhouse_20250416_092621 clickhouse_20250416_092621_0 │ 2025-04-16 09:30:15 │ 2025-04-16 09:30:22 │ success │                                                                                                                                                                                                                                                            │
│ chi-clickhouse-server-cnxcluster-01-0-1-0 │ restore_remote --rm --schema --env=REMOTE_STORAGE=sftp --env=SFTP_ADDRESS=tp --env=SFTP_USERNAME=username --env=SFTP_KEY=/path --env=SFTP_PORT=22 --env=SFTP_PATH=/tmp/gd/BACKUP_20250416_092621_141/clickhouse/clickhouse_20250416_092621 clickhouse_20250416_092621_0 │ 2025-04-16 09:30:15 │ 2025-04-16 09:30:21 │ success │                                                                                                                                                                                                                                                            │
│ chi-clickhouse-server-cnxcluster-01-1-0-0 │ restore_remote --rm --schema --env=REMOTE_STORAGE=sftp --env=SFTP_ADDRESS=tp --env=SFTP_USERNAME=username --env=SFTP_KEY=/path --env=SFTP_PORT=22 --env=SFTP_PATH=/tmp/gd/BACKUP_20250416_092621_141/clickhouse/clickhouse_20250416_092621 clickhouse_20250416_092621_1 │ 2025-04-16 09:30:15 │ 2025-04-16 09:30:22 │ success │                                                                                                                                                                                                                                                            │
│ chi-clickhouse-server-cnxcluster-01-1-1-0 │ restore_remote --rm --schema --env=REMOTE_STORAGE=sftp --env=SFTP_ADDRESS=tp --env=SFTP_USERNAME=username --env=SFTP_KEY=/path --env=SFTP_PORT=22 --env=SFTP_PATH=/tmp/gd/BACKUP_20250416_092621_141/clickhouse/clickhouse_20250416_092621 clickhouse_20250416_092621_1 │ 2025-04-16 09:30:15 │ 2025-04-16 09:30:21 │ success │                                                                                                                                                                                                                                                            │
│ chi-clickhouse-server-cnxcluster-01-0-0-0 │ delete local clickhouse_20250416_092621_0                                                                                                                                                                                                                                           │ 2025-04-16 09:32:15 │ 2025-04-16 09:32:23 │ success │                                                                                                                                                                                                                                                            │
│ chi-clickhouse-server-cnxcluster-01-0-1-0 │ delete local clickhouse_20250416_092621_0                                                                                                                                                                                                                                           │ 2025-04-16 09:32:23 │ 2025-04-16 09:32:31 │ success │                                                                                                                                                                                                                                                            │
│ chi-clickhouse-server-cnxcluster-01-1-0-0 │ delete local clickhouse_20250416_092621_1                                                                                                                                                                                                                                           │ 2025-04-16 09:32:31 │ 2025-04-16 09:32:39 │ success │                                                                                                                                                                                                                                                            │
│ chi-clickhouse-server-cnxcluster-01-1-1-0 │ delete local clickhouse_20250416_092621_1                                                                                                                                                                                                                                           │ 2025-04-16 09:32:39 │ 2025-04-16 09:32:46 │ success │                                                                                                                                                                                                                                                            │
│ chi-clickhouse-server-cnxcluster-01-0-0-0 │ restore_remote --rm --data --env=REMOTE_STORAGE=sftp --env=SFTP_ADDRESS=tp --env=SFTP_USERNAME=username --env=SFTP_KEY=/path --env=SFTP_PORT=22 --env=SFTP_PATH=/tmp/gd/BACKUP_20250416_092621_141/clickhouse/clickhouse_20250416_092621 clickhouse_20250416_092621_0   │ 2025-04-16 09:34:47 │ 2025-04-16 09:34:55 │ success │                                                                                                                                                                                                                                                            │
│ chi-clickhouse-server-cnxcluster-01-1-0-0 │ restore_remote --rm --data --env=REMOTE_STORAGE=sftp --env=SFTP_ADDRESS=tp --env=SFTP_USERNAME=username --env=SFTP_KEY=/path --env=SFTP_PORT=22 --env=SFTP_PATH=/tmp/gd/BACKUP_20250416_092621_141/clickhouse/clickhouse_20250416_092621 clickhouse_20250416_092621_1   │ 2025-04-16 09:34:47 │ 2025-04-16 09:34:55 │ error   │ one of restoreDataRegular go-routine return error: can't attach data parts for table 'table': table can't restore cause system.replicas entries already exists and replication in progress from another replica, log_pointer=20, log_max_index=19, absolute_delay=0, queue_size=0 │

[gavishdewangan]

@Slach Any workaround for this, what is the reason we get can't attach data parts error.

[Slach]

could you share
result of
kubectl exec -n <your-namespace> chi-clickhouse-server-cnxcluster-01-0-0-0 --container=clickhouse-backup --grep Replicated /var/lib/clickhouse/backup/clickhouse_20250416_092621_0/metadata/db/table.json

where db/table is your failed table from log?

[gavishdewangan]

@Slach a question, if inserts are happening in clickhouse and at that time we trigger restore, in that case will we face this error
one of restoreDataRegular go-routine return error: can't attach data parts for table 'table': table can't restore cause system.replicas entries already exists and replication in progress from another replica, log_pointer=12, log_max_index=11, absolute_delay=0, queue_size=0

[Slach]

yes. if inserts happens between DROP TABLE + CREATE TABLE and ALTER TABLE ... ATTACH PART then you will see this error
you can use --env=CLICKHOUSE_CHECK_REPLICAS_BEFORE_ATTACH=0, but this check allow to avoid duplicate data when you use wrong restore command sequence