Race condition: deploying CHI after Keeper leaves CHI unconnected to Keeper/replication

[lvirbalas]

Hello,

I can consistently reproduce this. Not certain if it's misconfiguration or an actual bug, but thought I'll report.

How to reproduce

1. Apply ClickHouseKeeperInstallation manifest (say with 3 replicas).
2. Wait for Keeper pods to startup <- this is the main point.
3. Apply ClickHouseInstallation with replicasCount: 2 and a section for Keeper.
4. Wait for ClickHouse pods to startup.
5. Try creating a ReplicatedMergeTree table

Result (replication doesn't work)

Code: 159. DB::Exception: Received from localhost:9000. DB::Exception: Distributed DDL task /clickhouse/simple-01/task_queue/ddl/query-0000000000 is not finished on 2 of 2 hosts (0 of them are currently executing the task, 0 are inactive). They are going to execute the query in background. Was waiting for 180.652144063 seconds, which is longer than distributed_ddl_task_timeout. (TIMEOUT_EXCEEDED)

Mitigation A

1. Restart Keeper pods.
2. Wait for them to startup.
3. Try creating ReplicatedMergeTree table again. It will work this time:

Query id: 4d6da4da-d86f-4810-9f62-0557f3d5983b
  ┌─host─────────────────────┬─port─┬─status─┬─error─┬─num_hosts_remaining─┬─num_hosts_active─┐
1. │ chi-simple-01-simple-0-0 │ 9000 │      0 │       │                   1 │                1 │
2. │ chi-simple-01-simple-0-1 │ 9000 │      0 │       │                   0 │                0 │
   └──────────────────────────┴──────┴────────┴───────┴─────────────────────┴──────────────────┘

Mitigation B

1. First install CHI.
2. Then install CHK.
3. Replication will work out of the box.

Is this a known issue? Looks very much like a race condition.

[Slach]

which clickhouse-server version do you use?
could you try with 25.8 ?