master tidb fails to bootstrap a cluster several days ago

[tiancaiamao]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

Maybe not affect the released version, need furthur investigation.

I create a cluster several days ago, and today I git pull the master and restart the tidb, I find tidb fails to start.
I print the error and the error stack look likes this:

[2023/02/06 20:33:36.525 +08:00] [INFO] [client.go:702] ["[pd] tso dispatcher created"] [dc-location=global]
[2023/02/06 20:33:36.525 +08:00] [INFO] [tikv_driver.go:191] ["using API V1."]
[2023/02/06 20:33:36.526 +08:00] [INFO] [store.go:82] ["new store with retry success"]
..... create table tidb_ddl_backfill(
                id bigint not null,
                ddl_job_id bigint not null,
                ele_id bigint not null,
                ele_key blob,
                store_id bigint,
                type int,
                exec_id blob default null,
                exec_lease timestamp,
                state int,
                curr_key blob,
                start_key blob,
                end_key blob,
                start_ts bigint,
                finish_ts bigint,
                row_count bigint,
                backfill_meta longblob,
                unique key(ddl_job_id, ele_id, ele_key(20), id))
[meta:1050]table already exists
github.com/pingcap/errors.AddStack
        /home/genius/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20221009092201-b66cddb77c32/errors.go:174
github.com/pingcap/errors.(*Error).GenWithStack
        /home/genius/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20221009092201-b66cddb77c32/normalize.go:155
github.com/pingcap/tidb/meta.(*Meta).checkTableNotExists
        /home/genius/project/src/github.com/pingcap/tidb/meta/meta.go:504
github.com/pingcap/tidb/meta.(*Meta).CreateTableOrView
        /home/genius/project/src/github.com/pingcap/tidb/meta/meta.go:625
github.com/pingcap/tidb/session.createAndSplitTables
        /home/genius/project/src/github.com/pingcap/tidb/session/session.go:3138
github.com/pingcap/tidb/session.InitDDLJobTables.func1
        /home/genius/project/src/github.com/pingcap/tidb/session/session.go:3111
github.com/pingcap/tidb/kv.RunInNewTxn
        /home/genius/project/src/github.com/pingcap/tidb/kv/txn.go:129
github.com/pingcap/tidb/session.InitDDLJobTables
        /home/genius/project/src/github.com/pingcap/tidb/session/session.go:3101
github.com/pingcap/tidb/session.BootstrapSession
        /home/genius/project/src/github.com/pingcap/tidb/session/session.go:3261
main.createStoreAndDomain
        /home/genius/project/src/github.com/pingcap/tidb/tidb-server/main.go:350
main.main
        /home/genius/project/src/github.com/pingcap/tidb/tidb-server/main.go:240
runtime.main
        /home/genius/project/go/src/runtime/proc.go:250
runtime.goexit
        /home/genius/project/go/src/runtime/asm_amd64.s:1598
[2023/02/06 20:33:36.532 +08:00] [FATAL] [terror.go:300] ["unexpected error"] [error="[meta:1050]table already exists"] [stack="githu
b.com/pingcap/tidb/parser/terror.MustNil\n\t/home/genius/project/src/github.com/pingcap/tidb/parser/terror/terror.go:300\nmain.create
StoreAndDomain\n\t/home/genius/project/src/github.com/pingcap/tidb/tidb-server/main.go:354\nmain.main\n\t/home/genius/project/src/git
hub.com/pingcap/tidb/tidb-server/main.go:240\nruntime.main\n\t/home/genius/project/go/src/runtime/proc.go:250"] [stack="github.com/pi
ngcap/tidb/parser/terror.MustNil\n\t/home/genius/project/src/github.com/pingcap/tidb/parser/terror/terror.go:300\nmain.createStoreAnd
Domain\n\t/home/genius/project/src/github.com/pingcap/tidb/tidb-server/main.go:354\nmain.main\n\t/home/genius/project/src/github.com/
pingcap/tidb/tidb-server/main.go:240\nruntime.main\n\t/home/genius/project/go/src/runtime/proc.go:250"]

I worry some of the change might break the upgrade compatability so I file this issue.

2. What did you expect to see? (Required)

The master tidb binary should work with the cluster I created several days ago.

3. What did you see instead (Required)

TiDB fails to start.

4. What is your TiDB version? (Required)

ba21e73

[tiancaiamao]

It might be related to #40984
Shall we use create table IF NOT EXIST ...?
Feel free to close it if this does not really affect any released tidb. @Defined2014

[tiancaiamao]

OK, that PR had pointed out:

This will break the upgrade compatibility of the master version tidb-server, you can manually delete the mysql.tidb_ddl_backfill and mysql.tidb_ddl_backfill_history with the old version tidb-server to solve this problem

It's a known issue and just affect the master branch, so close it.

[Defined2014]

The #40984 will break the upgrade compatibility of the master version tidb-server, you can manually delete the mysql.tidb_ddl_backfill and mysql.tidb_ddl_backfill_history with the old version tidb-server to solve this problem.

Upgrade from v6.5.0 to matser works well.