PiTR restore might fail due to EpochNotMatch

[fubinzh]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

1. There is a log backup, which is about ~1.5T, this is a log backup after Lots of log backup generated during data preparation tikv/tikv#13306
2. Try to restore it to tidb cluster with 3 TiKVs (8C 16G)

2. What did you expect to see? (Required)

Restore should succeed

3. What did you see instead (Required)

Restore failed due to

/ # /br restore point --full-backup-storage 's3://pitr-3/full0816_2' -s "s3://pitr-3/pitr3-log" --s3.endpoint "http://xxx:9000" --pd upstream-pd:2379 --restored-ts "2022-08-17 19:00:00.000 +0800"
Detail BR log in /tmp/br.log.2022-08-17T11.09.53Z
Full Restore <-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------> 100.00%
[2022/08/17 11:10:19.001 +00:00] [INFO] [collector.go:69] ["Full Restore success summary"] [total-ranges=1532] [ranges-succeed=1532] [ranges-failed=0] [split-region=1.360160676s] [restore-ranges=1024] [total-take=25.111464
319s] [restore-data-size(after-compressed)=775.7MB] [Size=775735581] [BackupTS=435351609681641473] [total-kv=15000002] [total-kv-size=1.89GB] [average-speed=75.26MB/s]
Restore Meta Files <-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------> 100.00%
Restore KV Files <---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------> 100.00%
[2022/08/17 13:28:59.043 +00:00] [INFO] [collector.go:69] ["restore log failed summary"] [error="failed to restore kv files: execute over region id:128162 start_key:\"t\\200\\000\\000\\000\\000\\000\\013\\3774_i\\200\\000\
\000\\000\\000\\377\\000\\000\\002\\004\\031\\256\\017U\\377@\\000\\000\\000\\003\\200\\000\\000\\377\\000\\0001}\\324\\000\\000\\000\\374\" end_key:\"t\\200\\000\\000\\000\\000\\000\\013\\3774_i\\200\\000\\000\\000\\000\\
377\\000\\000\\002\\004\\031\\256\\223u\\377\\300\\000\\000\\000\\003\\200\\000\\000\\377\\000\\000@+\\245\\000\\000\\000\\374\" region_epoch:<conf_ver:5 version:2007 > peers:<id:128163 store_id:1 > peers:<id:128164 store_
id:5 > peers:<id:128165 store_id:6 >  failed: EpochNotMatch current epoch of region 128162 is conf_ver: 5 version: 2008, but you sent conf_ver: 5 version: 2007; execute over region id:128210 start_key:\"t\\200\\000\\000\\0
00\\000\\000\\013\\3774_i\\200\\000\\000\\000\\000\\377\\000\\000\\004\\004\\000\\000\\000\\000\\377\\000\\000\\t6\\003\\200\\000\\000\\377\\000\\000S\\367b\\000\\000\\000\\374\" end_key:\"t\\200\\000\\000\\000\\000\\000\\
013\\3774_i\\200\\000\\000\\000\\000\\377\\000\\000\\004\\004\\000\\000\\000\\000\\377\\000\\000\\025\\311\\003\\200\\000\\000\\377\\000\\000c\\363\\232\\000\\000\\000\\374\" region_epoch:<conf_ver:5 version:2009 > peers:<
id:128211 store_id:1 > peers:<id:128212 store_id:5 > peers:<id:128213 store_id:6 >  failed: EpochNotMatch current epoch of region 128210 is conf_ver: 5 version: 2010, but you sent conf_ver: 5 version: 2009; execute over re
gion id:128274 start_key:\"t\\200\\000\\000\\000\\000\\000\\013\\3774_i\\200\\000\\000\\000\\000\\377\\000\\000\\002\\004\\031\\256\\223u\\377\\300\\000\\000\\000\\003\\200\\000\\000\\377\\000\\000@+\\245\\000\\000\\000\\3
74\" end_key:\"t\\200\\000\\000\\000\\000\\000\\013\\3774_i\\200\\000\\000\\000\\000\\377\\000\\000\\003\\004\\000\\000\\000\\000\\377\\000\\000\\000Y\\003\\200\\000\\000\\377\\000\\000?\\352 \\000\\000\\000\\374\" region_
epoch:<conf_ver:5 version:2008 > peers:<id:128275 store_id:1 > peers:<id:128276 store_id:5 > peers:<id:128277 store_id:6 > 

4. What is your TiDB version? (Required)

/ # /br -V
Release Version: v6.2.0
Git Commit Hash: be9be01
Git Branch: heads/refs/tags/v6.2.0
Go Version: go1.18.5
UTC Build Time: 2022-08-16 13:01:23
Race Enabled: false