Downgrade support from 3.5 to 3.4

[logicalhan]

What would you like to be added?

I would like to be able to safely downgrade from 3.5 to 3.4, and then safely reupgrade back to 3.5.

Why is this needed?

Given the vast number of data correctness issues we've unearthed in etcd 3.5 (many of them fixed by @ahrtr and @serathius), I have personal reservations about upgrading my k8s clusters to use 3.5. If there was a working rollback strategy (tested of course, as well), then I would be much more inclined to update my etcds to a more recent version.

[serathius]

I think it could be easily added to etcdutl migrate command allowing for safe offline downgrade and upgrade operations.
Code: https://github.com/etcd-io/etcd/blob/main/etcdutl/etcdutl/migrate_command.go

This would also help with kubernetes/kubernetes#117906 and cleanup of kubernetes migrate script for etcd.

[lavacat]

Please assign this to me, we already have a minimal internal patch to address this. In current form - it's a 3.4 patch that allows 3.4 to be deployed within 3.5 cluster to avoid downtime and perform a rolling downgrade.
It's done by hacking version and removing confState and term keys.
But it would be great to make it part of migrate and add more testing around it.

[serathius]

Just a note, we support for rolling update is out of scope for now. Let's start with the migrate script.

[lavacat]

Quick update - trying to get POC to work. The idea is to run etcdutl migrate --data-dir data-3.5 --target-version 3.4 and get a data dir that etcd 3.4 can be started with.
My understanding is that currently migrate only updates MetaStorageVersionName key that was added since 3.6. But it won't update ClusterClusterVersionKeyName and version in v2store.

At the moment, running into

etcdserver/membership: cluster cannot be downgraded (current version: 3.4.26 is lower than determined cluster version: 3.5).

because of v2store version.

[lavacat]

For reference, tried running etcdctl downgrade from etcd 3.6 build targeting 3.5 cluster, but it didn't work.

Related design docs
etcd Downgrades Design
etcd storage versioning

$ ./bin/etcdctl downgrade validate 3.4
Downgrade validate success, cluster version 3.5.0

$ ./bin/etcdctl downgrade enable 3.4
{"level":"warn","ts":"2023-05-30T01:12:41.770844-0700","logger":"etcd-client","caller":"v3@v3.6.0-alpha.0/retry_interceptor.go:65","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0001fc000/127.0.0.1:2379","method":"/etcdserverpb.Maintenance/Downgrade","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"}
Error: context deadline exceeded

etcd 3.5 one node cluster log

{"level":"info","ts":"2023-05-30T01:12:36.794018-0700","caller":"membership/cluster.go:890","msg":"The server is ready to downgrade","target-version":"3.4.0","server-version":"3.5.9"}
{"level":"warn","ts":"2023-05-30T01:12:36.88595-0700","caller":"etcdserver/cluster_util.go:459","msg":"remotes server has mismatching etcd version","remote-member-id":"8e9e05c52164694d","current-server-version":"3.5.0","target-version":"3.4.0"}
{"level":"warn","ts":"2023-05-30T01:12:41.77082-0700","caller":"etcdserver/v3_server.go:1047","msg":"reject downgrade request","error":"etcdserver: request timed out"}
{"level":"warn","ts":"2023-05-30T01:12:41.770895-0700","caller":"v3rpc/interceptor.go:197","msg":"request stats","start time":"2023-05-30T01:12:36.773319-0700","time spent":"4.997554901s","remote":"127.0.0.1:62022","response type":"/etcdserverpb.Maintenance/Downgrade","request count":-1,"request size":-1,"response count":-1,"response size":-1,"request content":""}
{"level":"warn","ts":"2023-05-30T01:12:41.886266-0700","caller":"etcdserver/cluster_util.go:459","msg":"remotes server has mismatching etcd version","remote-member-id":"8e9e05c52164694d","current-server-version":"3.5.0","target-version":"3.4.0"}

Going to debug this more.

[jpbetz]

Did we ever de-couple etcd version from data storage version? I vaguely recall multiple people pointing out that it is sort of silly that you can't automatically downgrade from 3.5 to 3.4 given that the file formats of the persisted data is identical, and that if we just gave data files a format version and only incremented it when we actually change how data is written to file that downgrade can be simpler.

[lavacat]

Version logic is a bit different between 3.4, 3.5 and 3.6.
In 3.4 version is first decided in decideClusterVersion based on version.Version and then saved to v2store. In Recover we rely only on version recorded in v2store. See clusterVersionFromStore. Version is also saved to backend cluster/clusterVersion but it's never read.

3.5 added clusterVersionFromBackend, but I think v2store path is still used by default. Also 3.5 added downgradeInfoFromBackend. I don't fully understand downgrade, but I think workflow is described here

3.6 is using ClusterVersionFromBackend by default. It also added meta/storageVersion key that's used in migrate.

[lavacat]

What do you think about adding a special flag to 3.4 to control version checks? See #15990 This will also allow rolling downgrade.

Another option is to snapshot using etcdctl 3.5, then stop the cluster and restore using etcdctl 3.4. Here are steps I've used to test this:
3.5 cluster

bin/etcd --name infra1 --listen-client-urls http://127.0.0.1:2379 --advertise-client-urls http://127.0.0.1:2379 --listen-peer-urls http://127.0.0.1:12380 --initial-advertise-peer-urls http://127.0.0.1:12380 --initial-cluster-token etcd-cluster-1 --initial-cluster 'infra1=http://127.0.0.1:12380,infra2=http://127.0.0.1:22380,infra3=http://127.0.0.1:32380' --initial-cluster-state new --enable-pprof --logger=zap --log-outputs=stderr
bin/etcd --name infra2 --listen-client-urls http://127.0.0.1:22379 --advertise-client-urls http://127.0.0.1:22379 --listen-peer-urls http://127.0.0.1:22380 --initial-advertise-peer-urls http://127.0.0.1:22380 --initial-cluster-token etcd-cluster-1 --initial-cluster 'infra1=http://127.0.0.1:12380,infra2=http://127.0.0.1:22380,infra3=http://127.0.0.1:32380' --initial-cluster-state new --enable-pprof --logger=zap --log-outputs=stderr
bin/etcd --name infra3 --listen-client-urls http://127.0.0.1:32379 --advertise-client-urls http://127.0.0.1:32379 --listen-peer-urls http://127.0.0.1:32380 --initial-advertise-peer-urls http://127.0.0.1:32380 --initial-cluster-token etcd-cluster-1 --initial-cluster 'infra1=http://127.0.0.1:12380,infra2=http://127.0.0.1:22380,infra3=http://127.0.0.1:32380' --initial-cluster-state new --enable-pprof --logger=zap --log-outputs=stderr

snapshot

./bin/etcdctl snapshot save snap-3.5

stop all nodes, remove infra dirs and restore:

./bin-3.4/etcdctl snapshot restore snap-3.5 --name infra1 --initial-advertise-peer-urls http://127.0.0.1:12380 --initial-cluster-token etcd-cluster-1 --initial-cluster 'infra1=http://127.0.0.1:12380,infra2=http://127.0.0.1:22380,infra3=http://127.0.0.1:32380'
./bin-3.4/etcdctl snapshot restore snap-3.5 --name infra2 --initial-advertise-peer-urls http://127.0.0.1:22380 --initial-cluster-token etcd-cluster-1 --initial-cluster 'infra1=http://127.0.0.1:12380,infra2=http://127.0.0.1:22380,infra3=http://127.0.0.1:32380'
./bin-3.4/etcdctl snapshot restore snap-3.5 --name infra3 --initial-advertise-peer-urls http://127.0.0.1:32380 --initial-cluster-token etcd-cluster-1 --initial-cluster 'infra1=http://127.0.0.1:12380,infra2=http://127.0.0.1:22380,infra3=http://127.0.0.1:32380'

then start cluster using 3.4 binary:

bin-3.4/etcd --name infra1 --listen-client-urls http://127.0.0.1:2379 --advertise-client-urls http://127.0.0.1:2379 --listen-peer-urls http://127.0.0.1:12380 --initial-advertise-peer-urls http://127.0.0.1:12380 --initial-cluster-token etcd-cluster-1 --initial-cluster 'infra1=http://127.0.0.1:12380,infra2=http://127.0.0.1:22380,infra3=http://127.0.0.1:32380' --initial-cluster-state new --enable-pprof --logger=zap --log-outputs=stderr
bin-3.4/etcd --name infra2 --listen-client-urls http://127.0.0.1:22379 --advertise-client-urls http://127.0.0.1:22379 --listen-peer-urls http://127.0.0.1:22380 --initial-advertise-peer-urls http://127.0.0.1:22380 --initial-cluster-token etcd-cluster-1 --initial-cluster 'infra1=http://127.0.0.1:12380,infra2=http://127.0.0.1:22380,infra3=http://127.0.0.1:32380' --initial-cluster-state new --enable-pprof --logger=zap --log-outputs=stderr
bin-3.4/etcd --name infra3 --listen-client-urls http://127.0.0.1:32379 --advertise-client-urls http://127.0.0.1:32379 --listen-peer-urls http://127.0.0.1:32380 --initial-advertise-peer-urls http://127.0.0.1:32380 --initial-cluster-token etcd-cluster-1 --initial-cluster 'infra1=http://127.0.0.1:12380,infra2=http://127.0.0.1:22380,infra3=http://127.0.0.1:32380' --initial-cluster-state new --enable-pprof --logger=zap --log-outputs=stderr

[lavacat]

@serathius saw your comment on PR. Duplicating my question here. migrate will only help with removing confState and term, correct? v2store will still have 3.5 version. What is the process to complete the downgrade? The only way I've found was using snapshot and it requires stopping all nodes.

I've also tried downgrade enable workflow, that still requires using snapshot but I was hoping there is no need to stop the cluster. It didn't work for me.

[serathius]

@serathius saw your comment on PR. Duplicating my #15990 (comment) here. migrate will only help with removing confState and term, correct? v2store will still have 3.5 version. What is the process to complete the downgrade? The only way I've found was using #15878 (comment) and it requires stopping all nodes.

This is exactly what we need to support downgrades. Remove the confState and term fields. This is also exactly what downgrade enable does in v3.6, but it also coordinates the change between members in live cluster. We don't want to backport the coordination logic.

To make it clear, removing confState and term field is crucial for downgrades and etcd correctness. You are right that etcd v3.4 will just start from v3.5 data. However, have you thought about what will happen with confState and term fields? Etcd v3.4 is unaware of those fields so they will remain unchanged and ignored, and then you decide to upgrade back to v3.5 and it goes BOOOM. Etcd v3.5 starts, find those fields, assumes they come from previous v3.5 run and tries to use outdated confState and term. See #13514

One thing we can add in v3.4 is a safeguard for those fields. Have etcd v3.4.27 reject db file if it finds fields from v3.5. It should make it clear to user that just loading data from v3.5 in v3.4 is unsupported and will break their cluster, maybe not immediately, but later.