replication.strictly-match-label=true & engine=tiflash have Incompatibility problem

[Chad20N13]

Bug Report

What did you do?

1. After expanding the tiflash, the tiflash node will automatically have a layer of label information: engine: "tiflash"
   2, When the pd is configured: replication. strictly-match-label: true
   3, This causes tiflash unable to start because replication. location-labels in the default pd config does not exist: engine layer configuration.

What did you expect to see?

1、tiflash start successfully
2、replication. location-labels can automatically recognize and add a new "engine" layer;

What did you see instead?

1、tiflash start failed；
2、replication. location-labels can not automatically recognize and keep old layer;

What version of PD are you using (pd-server -V)?

Release Version: v7.5.0
Edition: Community
Git Commit Hash: ef6ba85
Git Branch: heads/refs/tags/v7.5.0
UTC Build Time: 2023-11-16 10:31:04

[rleungx]

Thanks, indeed there is a problem, I can reproduce it in my local environment:

With the following config.toml:

[replication]
strictly-match-label = true

Then run tiup playground nightly --pd.config config.toml --tiflash 2

$ tiup playground nightly --pd.config conf/config.toml --tiflash 2
Start pd instance:v8.3.0-alpha-nightly
Start tikv instance:v8.3.0-alpha-nightly
Start tidb instance:v8.3.0-alpha-nightly
Waiting for tidb instances ready
127.0.0.1:33191 ... Done
component grafana version v8.3.0-alpha-nightly is already installed
Start tiflash instance:v8.3.0-alpha-nightly
Start tiflash instance:v8.3.0-alpha-nightly
Waiting for tiflash instances ready
127.0.0.1:3930 ... ⠦
127.0.0.1:3931 ... ⠦
tiflash quit: exit status 1
[2024/08/01 20:05:09.931 +08:00] [DEBUG] [SegmentReader.cpp:45] [Stopped] [thread_id=192]
[2024/08/01 20:05:09.931 +08:00] [DEBUG] [SegmentReader.cpp:45] [Stopped] [thread_id=192]
[2024/08/01 20:05:09.931 +08:00] [DEBUG] [SegmentReader.cpp:45] [Stopped] [thread_id=192]
[2024/08/01 20:05:09.931 +08:00] [DEBUG] [SegmentReader.cpp:45] [Stopped] [thread_id=192]
[2024/08/01 20:05:09.931 +08:00] [DEBUG] [SegmentReader.cpp:45] [Stopped] [thread_id=192]
[2024/08/01 20:05:09.931 +08:00] [DEBUG] [SegmentReader.cpp:45] [Stopped] [thread_id=192]
[2024/08/01 20:05:09.931 +08:00] [DEBUG] [SegmentReader.cpp:45] [Stopped] [thread_id=192]
[2024/08/01 20:05:09.931 +08:00] [DEBUG] [SegmentReader.cpp:45] [Stopped] [thread_id=192]
[2024/08/01 20:05:09.931 +08:00] [DEBUG] [SegmentReader.cpp:45] [Stopped] [thread_id=192]
Waiting for tiflash instances ready
127.0.0.1:3930 ... ⠏
127.0.0.1:3931 ... ⠏
tiflash quit: exit status 1
[2024/08/01 20:05:10.124 +08:00] [DEBUG] [SegmentReader.cpp:45] [Stopped] [thread_id=171]
[2024/08/01 20:05:10.124 +08:00] [DEBUG] [SegmentReader.cpp:45] [Stopped] [thread_id=171]
[2024/08/01 20:05:10.124 +08:00] [DEBUG] [SegmentReader.cpp:45] [Stopped] [thread_id=171]
[2024/08/01 20:05:10.124 +08:00] [DEBUG] [SegmentReader.cpp:45] [Stopped] [thread_id=171]
[2024/08/01 20:05:10.124 +08:00] [DEBUG] [SegmentReader.cpp:45] [Stopped] [thread_id=171]
[2024/08/01 20:05:10.124 +08:00] [DEBUG] [SegmentReader.cpp:45] [Stopped] [thread_id=171]
[2024/08/01 20:05:10.124 +08:00] [DEBUG] [SegmentReader.cpp:45] [Stopped] [thread_id=171]
[2024/08/01 20:05:10.124 +08:00] [DEBUG] [SegmentReader.cpp:45] [Stopped] [thread_id=171]
[2024/08/01 20:05:10.124 +08:00] [DEBUG] [SegmentReader.cpp:45] [Stopped] [thread_id=171]
Waiting for tiflash instances ready
127.0.0.1:3930 ... Error
127.0.0.1:3931 ... Error

Here is the log:

[2024/08/01 20:05:09.899 +08:00] [FATAL] [run.rs:1372] ["failed to start node: Other(\"[components/pd_client/src/util.rs:952]: key matching the label was not found in the PD, store label key: engine \")"] [thread_id=1]