BR: PiTR unable to communicate with PD(etcd) via TLS

[db-will]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

use v6.2 br backup and restore data to new cluster

2. What did you expect to see? (Required)

should be able to restore data to new cluster successfully

3. What did you see instead (Required)

Detail BR log in /var/log/br.log Error: restore log from A_CURRENT_TIMESTAMP to A_PAST_TIMESTAMP, but the current existed log from A_PAST_TIMESTAMP to A_PAST_TIMESTAMP: [BR:Common:ErrInvalidArgument]invalid argument

A_PAST_TIMESTAMP are the same timestamp point

4. What is your TiDB version? (Required)

6.2

[db-will]

@3pointer

[3pointer]

The root cause is checkpoint didn't moving forward.

And the reason of checkpoint didn't moving forward is TiKV didn't start watch task thread.

{"level":"WARN","caller":"errors.rs:135","message":"backup stream meet error","time":"xxx","err":"during connecting to the etcd: Etcd meet error transport error: transport error","context":"failed to start watch tasks"}

we should keep retrying to start the watch threads and I think we need add some check before we start log backup via br. @joccau

[YuJuncen]

Are you using the client key with SEC1 EC format? For now, tonic supports PKCS #8 and RSA keys only.

You can use the following command to transform the SEC-1 EC key into PKCS #8 key:

openssl pkcs8 -in $key -nocrypt -topk8 -out $key.pk8

And we can try to use the new generated key then.

[3pointer]

Are you using the client key with SEC1 EC format? For now, tonic supports PKCS #8 and RSA keys only.

You can use the following command to transform the SEC-1 EC key into PKCS #8 key:

openssl pkcs8 -in $key -nocrypt -topk8 -out $key.pk8

And we can try to use the new generated key then.

please give more information about why tonic not support SEC1 EC keys

[YuJuncen]

@3pointer Generally, it seems ring doesn't support SEC1-formatted EC keys for now (briansmith/ring#1456 not yet merged), and rustls have made a workaround (Aha, which is almost exactly the openssl convert command did.), however it seems tonic doesn't utilize the workaround for now.

[3pointer]

It's more like these four DNS names *.adhoc2-tidb-peer , *.adhoc2-ticdc-peer ,*.adhoc2-tikv-peer,*.adhoc2-pd-peer cause webpki checks failed. See the issue briansmith/webpki#265.

A workaround is remove these four names out of SAN in certificate.

[joccau]

/close

[ti-chi-bot]

@joccau: Closing this issue.

In response to this:

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[3pointer]

we will get the error when use CA-bundle certificated. this affects v6.5.0

[2023/01/25 10:30:25.271 +08:00] [WARN] [errors.rs:155] ["backup stream meet error"] [verbose_err="Etcd(GRpcStatus(Status { code: Unknown, message: \"transport error\", source: Some(tonic::transport::Error(Transport, hyper::Error(Io, Custom { kind: Other, error: \"error:14094412:SSL routines:ssl3_read_bytes:sslv3 alert bad certificate:ssl/record/rec_layer_s3.c:1544:SSL alert number 42\" }))) }))"] [err="Etcd meet error grpc request error: status: Unknown, message: \"transport error\", details: [], metadata: MetadataMap { headers: {} }"] [context="failed to get backup stream task"]