timestamp fallback in TestMixedTSODeployment

[lhy1024]

Flaky Test

Which jobs are failing

2023-07-03T04:18:55.3685989Z === RUN   TestMixedTSODeployment
...
2023-07-03T04:19:02.8702775Z [2023/07/03 04:19:02.869 +00:00] [PANIC] [tso_dispatcher.go:790] ["[tso] timestamp fallback"] [dc-location=global] [keyspace=4294967295] [last-ts="(1688357947449, 169)"] [cur-ts="(1688357942866, 1)"] [last-tso-server=127.0.0.1:46369] [cur-tso-server=127.0.0.1:34085] [last-keyspace-group-in-request=0] [cur-keyspace-group-in-request=0] [last-keyspace-group-in-response=0] [cur-keyspace-group-in-response=0] [last-response-received-at=2023/07/03 04:19:01.744 +00:00] [cur-response-received-at=2023/07/03 04:19:02.869 +00:00]
2023-07-03T04:19:02.8704461Z [2023/07/03 04:19:02.870 +00:00] [INFO] [tso_dispatcher.go:334] ["[tso] exit tso dispatcher"] [dc-location=global]
2023-07-03T04:19:02.8737225Z panic: [tso] timestamp fallback
2023-07-03T04:19:02.8738724Z 
2023-07-03T04:19:02.8740667Z goroutine 1081029 [running]:
2023-07-03T04:19:02.8741358Z go.uber.org/zap/zapcore.CheckWriteAction.OnWrite(0x2, 0xc003b808f0, {0x545fe90?, 0x37?, 0x0?})
2023-07-03T04:19:02.8742481Z 	/home/runner/go/pkg/mod/go.uber.org/zap@v1.24.0/zapcore/entry.go:198 +0xa8
2023-07-03T04:19:02.8743297Z go.uber.org/zap/zapcore.(*CheckedEntry).Write(0xc003b808f0, {0xc0078f5b00, 0xc, 0xc})
2023-07-03T04:19:02.8744295Z 	/home/runner/go/pkg/mod/go.uber.org/zap@v1.24.0/zapcore/entry.go:264 +0x586
2023-07-03T04:19:02.8745202Z go.uber.org/zap.(*Logger).Panic(0x656b3f0?, {0x4278728, 0x18}, {0xc0078f5b00, 0xc, 0xc})
2023-07-03T04:19:02.8746200Z 	/home/runner/go/pkg/mod/go.uber.org/zap@v1.24.0/logger.go:258 +0x6e
2023-07-03T04:19:02.8748016Z github.com/pingcap/log.Panic({0x4278728, 0x18}, {0xc0078f5b00, 0xc, 0xc})
2023-07-03T04:19:02.8749007Z 	/home/runner/go/pkg/mod/github.com/pingcap/log@v1.1.1-0.20221110025148-ca232912c9f3/global.go:54 +0x74
2023-07-03T04:19:02.8750141Z github.com/tikv/pd/client.(*tsoClient).compareAndSwapTS(0xc008f51100, {0x424b08d, 0x6}, 0xc00defde80, 0x18919f9b652, 0x1)
2023-07-03T04:19:02.8750901Z 	/home/runner/work/pd/pd/client/tso_dispatcher.go:790 +0x1306
2023-07-03T04:19:02.8753345Z github.com/tikv/pd/client.(*tsoClient).processRequests(0xc008f51100, {0x5190da0, 0xc00defa4c0}, {0x424b08d, 0x6}, 0xc00bf9e9b0, {0x0, 0x0, 0x0})
2023-07-03T04:19:02.8754127Z 	/home/runner/work/pd/pd/client/tso_dispatcher.go:763 +0x625
2023-07-03T04:19:02.8755659Z github.com/tikv/pd/client.(*tsoClient).handleDispatcher(0xc008f51100, {0x51a1990, 0xc00bf9e960}, {0x424b08d, 0x6}, 0xc00bf9e9b0)
2023-07-03T04:19:02.8756469Z 	/home/runner/work/pd/pd/client/tso_dispatcher.go:477 +0x1e25
2023-07-03T04:19:02.8758319Z created by github.com/tikv/pd/client.(*tsoClient).createTSODispatcher
2023-07-03T04:19:02.8758962Z 	/home/runner/work/pd/pd/client/tso_dispatcher.go:312 +0x4f7
2023-07-03T04:19:02.9442742Z FAIL	github.com/tikv/pd/tests/integrations/tso	266.014s
2023-07-03T04:19:02.9593906Z FAIL
2023-07-03T04:19:03.0240724Z make[1]: *** [Makefile:35: ci-test-job] Error 1
2023-07-03T04:19:03.0241254Z make[1]: Leaving directory '/home/runner/work/pd/pd/tests/integrations/tso'

CI link

https://github.com/tikv/pd/actions/runs/5440496266/jobs/9893479580?pr=6720

Reason for failure (if possible)

Anything else

[binshi-bing]

This is by design, and we need to verify this flaky test and adjust the test accordingly.

There is time fallback when rollbacking, in the online manner, from TSO microservice to pd serving TSO, which is expected. The rollback plan is to stop the TSO microservice first then migrate PD from API mode to PD mode.

[lhy1024]

close by #6740