mutiple lightning parall import failed for Table 'lightning_metadata.task_meta' doesn't exist

[lilinghai]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

mutiple lightnings parally import a big table and one failed the error as follows:

[2022/12/15 13:45:05.085 +08:00] [ERROR] [table_restore.go:352] ["import whole table failed"] [table=`sysbench`.`sbtest1`] [takeTime=58m25.400698161s] [error="check tasks exclusively failed: fetch task metas failed: Error 1146 (42S02): Table 'lightning_metadata.task_meta' doesn't exist"]
[2022/12/15 13:45:05.085 +08:00] [ERROR] [restore.go:1576] ["restore table failed"] [table=`sysbench`.`sbtest1`] [takeTime=58m25.415093818s] [error="[Lightning:Restore:ErrRestoreTable]restore table `sysbench`.`sbtest1` failed: check tasks exclusively failed: fetch task metas failed: Error 1146 (42S02): Table 'lightning_metadata.task_meta' doesn't exist"]
[2022/12/15 13:45:05.086 +08:00] [WARN] [util.go:183] ["check and finish schedulers failed with no retry"] [retryCnt=0] [error="fetch task meta failed: Error 1146 (42S02): Table 'lightning_metadata.task_meta' doesn't exist"]

2. What did you expect to see? (Required)

3. What did you see instead (Required)

4. What is your TiDB version? (Required)

master

[kennytm]

@niubell hello, this issue has been closed as "completed", but I don't see any PR linking to this issue. Is this actually fixed or is it "WONTFIX"? Thanks.

[lance6716]

A customer reports this issue again, we at least need to write a conclusion here. I'll check the log from the customer to locate it soon.

[lance6716]

I checked the code based on v6.5.5.

if cleanup is true, task_meta table will be dropped

tidb/br/pkg/lightning/restore/restore.go

Lines 1567 to 1572 in 71bcc44

	// clean up task metas
	if cleanup {
	logTask.Info("cleanup task metas")
	if cleanupErr := rc.taskMgr.Cleanup(context.Background()); cleanupErr != nil {
	logTask.Warn("failed to clean task metas, you may need to restore them manually", zap.Error(cleanupErr))
	}

And we need the log of every lightning instances to know the timeline. One possible cause is when a slow lightning is running between "create task_meta" and "record its task state", another fast one got true cleanup and dropped the table

[mittalrishabh]

Hi i would like to know why are we dropping this table. We can make it similar to table_meta. And also this table seems like a legacy table. I don't see much use of this table. Are you planning to remove it in 7.5

[okJiang]

And also this table seems like a legacy table. I don't see much use of this table. Are you planning to remove it in 7.5

@mittalrishabh It has not been removed. We just renamed it to task_meta_v2. https://github.com/pingcap/tidb/blob/master/br/pkg/lightning/importer/import.go