grpc connection fails on firestore/auth due to ssl bad signature errors

[asenmitrev]

[REQUIRED] Step 2: Describe your environment

• Operating System version: Ubuntu 22.04.4 LTS
• Firebase SDK version: ^12.2.0
• Firebase Product: auth, firestore (auth, database, storage, etc)
• Node.js version: 20.10.0
• NPM version: 10.2.3.
• OpenSSL 3.0.2 15 Mar 2022 (Library: OpenSSL 3.0.2 15 Mar 2022)

[REQUIRED] Step 3: Describe the problem

Steps to reproduce:

What happened? How can we make the problem occur?

GRPC issues started happening randomly on a vps that was running firebase admin sdk on several different apps. The root cause seems to be a ssl error in openssl. I have included debug output from grpc running on one of the node apps. Running open ssl directly without going through node and grpc gives a successful handshake and valid certificate. I am trying to find the root cause of this, whether in grpc-js or in firebase admin sdk, as all other libraries in node do not have issues with ssl handshakes.

The issue occurs about 90% of the time, as sometimes firebase sdk is able to connect successfully, only to fail a few minutes later on subsequent requests to firestore or auth.

Important to note the same code runs successfully on other vps-s and on my local machine. This seems to be specific to this vps. Openssl and ca-certificates are at the latest version.

Relevant Code:

GRPC logs:

D 2024-06-22T08:40:15.852Z | v1.10.9 88323 | resolving_load_balancer | dns:firestore.googleapis.com:443 IDLE -> IDLE
D 2024-06-22T08:40:15.853Z | v1.10.9 88323 | connectivity_state | (1) dns:firestore.googleapis.com:443 IDLE -> IDLE
D 2024-06-22T08:40:15.853Z | v1.10.9 88323 | dns_resolver | Resolver constructed for target dns:firestore.googleapis.com:443
D 2024-06-22T08:40:15.854Z | v1.10.9 88323 | channel | (1) dns:firestore.googleapis.com:443 Channel constructed with options {
  "grpc.max_receive_message_length": -1,
  "grpc.max_send_message_length": -1,
  "grpc.initial_reconnect_backoff_ms": 1000
}
D 2024-06-22T08:40:15.854Z | v1.10.9 88323 | channel_stacktrace | (1) Channel constructed 
    at new InternalChannel (/srv/www/djolev-bot/node_modules/@grpc/grpc-js/build/src/internal-channel.js:246:23)
    at new ChannelImplementation (/srv/www/djolev-bot/node_modules/@grpc/grpc-js/build/src/channel.js:35:32)
    at new Client (/srv/www/djolev-bot/node_modules/@grpc/grpc-js/build/src/client.js:66:36)
    at new ServiceClientImpl (/srv/www/djolev-bot/node_modules/@grpc/grpc-js/build/src/make-client.js:58:5)
    at GrpcClient.createStub (/srv/www/djolev-bot/node_modules/google-gax/build/src/grpc.js:344:22)
D 2024-06-22T08:40:15.857Z | v1.10.9 88323 | channel | (1) dns:firestore.googleapis.com:443 createResolvingCall [0] method="/google.firestore.v1.Firestore/RunQuery", deadline=2024-06-22T08:45:15.856Z
D 2024-06-22T08:40:15.857Z | v1.10.9 88323 | resolving_call | [0] Created
D 2024-06-22T08:40:15.858Z | v1.10.9 88323 | resolving_call | [0] Deadline: 2024-06-22T08:45:15.856Z
D 2024-06-22T08:40:15.858Z | v1.10.9 88323 | resolving_call | [0] Deadline will be reached in 299998ms
D 2024-06-22T08:40:15.858Z | v1.10.9 88323 | resolving_call | [0] start called
D 2024-06-22T08:40:15.859Z | v1.10.9 88323 | dns_resolver | Looking up DNS hostname firestore.googleapis.com
D 2024-06-22T08:40:15.860Z | v1.10.9 88323 | resolving_load_balancer | dns:firestore.googleapis.com:443 IDLE -> CONNECTING
D 2024-06-22T08:40:15.860Z | v1.10.9 88323 | connectivity_state | (1) dns:firestore.googleapis.com:443 IDLE -> CONNECTING
D 2024-06-22T08:40:15.860Z | v1.10.9 88323 | channel | (1) dns:firestore.googleapis.com:443 callRefTimer.ref | configSelectionQueue.length=1 pickQueue.length=0
D 2024-06-22T08:40:15.867Z | v1.10.9 88323 | resolving_call | [0] write() called with message of length 89
D 2024-06-22T08:40:15.867Z | v1.10.9 88323 | resolving_call | [0] halfClose called
D 2024-06-22T08:40:15.868Z | v1.10.9 88323 | resolving_call | [0] startRead called
D 2024-06-22T08:40:15.869Z | v1.10.9 88323 | dns_resolver | Resolved addresses for target dns:firestore.googleapis.com:443: [2a00:1450:400e:802::200a:443,142.250.179.170:443]
D 2024-06-22T08:40:15.870Z | v1.10.9 88323 | subchannel | (1) [2a00:1450:400e:802::200a]:443 Subchannel constructed with options {
  "grpc.max_receive_message_length": -1,
  "grpc.max_send_message_length": -1,
  "grpc.initial_reconnect_backoff_ms": 1000
}
D 2024-06-22T08:40:15.870Z | v1.10.9 88323 | subchannel_refcount | (1) [2a00:1450:400e:802::200a]:443 refcount 0 -> 1
D 2024-06-22T08:40:15.871Z | v1.10.9 88323 | subchannel | (2) 142.250.179.170:443 Subchannel constructed with options {
  "grpc.max_receive_message_length": -1,
  "grpc.max_send_message_length": -1,
  "grpc.initial_reconnect_backoff_ms": 1000
}
D 2024-06-22T08:40:15.871Z | v1.10.9 88323 | subchannel_refcount | (2) 142.250.179.170:443 refcount 0 -> 1
D 2024-06-22T08:40:15.871Z | v1.10.9 88323 | subchannel_refcount | (1) [2a00:1450:400e:802::200a]:443 refcount 1 -> 2
D 2024-06-22T08:40:15.871Z | v1.10.9 88323 | subchannel_refcount | (2) 142.250.179.170:443 refcount 1 -> 2
D 2024-06-22T08:40:15.872Z | v1.10.9 88323 | pick_first | Start connecting to subchannel with address [2a00:1450:400e:802::200a]:443
D 2024-06-22T08:40:15.872Z | v1.10.9 88323 | pick_first | IDLE -> CONNECTING
D 2024-06-22T08:40:15.872Z | v1.10.9 88323 | resolving_load_balancer | dns:firestore.googleapis.com:443 CONNECTING -> CONNECTING
D 2024-06-22T08:40:15.872Z | v1.10.9 88323 | connectivity_state | (1) dns:firestore.googleapis.com:443 CONNECTING -> CONNECTING
D 2024-06-22T08:40:15.872Z | v1.10.9 88323 | channel | (1) dns:firestore.googleapis.com:443 callRefTimer.unref | configSelectionQueue.length=0 pickQueue.length=0
D 2024-06-22T08:40:15.873Z | v1.10.9 88323 | subchannel | (1) [2a00:1450:400e:802::200a]:443 IDLE -> CONNECTING
D 2024-06-22T08:40:15.874Z | v1.10.9 88323 | transport | dns:firestore.googleapis.com:443 creating HTTP/2 session to [2a00:1450:400e:802::200a]:443
D 2024-06-22T08:40:15.876Z | v1.10.9 88323 | channel | (1) dns:firestore.googleapis.com:443 createRetryingCall [1] method="/google.firestore.v1.Firestore/RunQuery"
D 2024-06-22T08:40:15.876Z | v1.10.9 88323 | resolving_call | [0] Created child [1]
D 2024-06-22T08:40:15.876Z | v1.10.9 88323 | retrying_call | [1] start called
D 2024-06-22T08:40:15.876Z | v1.10.9 88323 | channel | (1) dns:firestore.googleapis.com:443 createLoadBalancingCall [2] method="/google.firestore.v1.Firestore/RunQuery"
D 2024-06-22T08:40:15.877Z | v1.10.9 88323 | retrying_call | [1] Created child call [2] for attempt 1
D 2024-06-22T08:40:15.877Z | v1.10.9 88323 | load_balancing_call | [2] start called
D 2024-06-22T08:40:15.877Z | v1.10.9 88323 | load_balancing_call | [2] Pick called
D 2024-06-22T08:40:15.877Z | v1.10.9 88323 | load_balancing_call | [2] Pick result: QUEUE subchannel: null status: undefined undefined
D 2024-06-22T08:40:15.877Z | v1.10.9 88323 | channel | (1) dns:firestore.googleapis.com:443 callRefTimer.ref | configSelectionQueue.length=0 pickQueue.length=1
D 2024-06-22T08:40:15.877Z | v1.10.9 88323 | retrying_call | [1] startRead called
D 2024-06-22T08:40:15.877Z | v1.10.9 88323 | load_balancing_call | [2] startRead called
D 2024-06-22T08:40:15.878Z | v1.10.9 88323 | retrying_call | [1] write() called with message of length 94
D 2024-06-22T08:40:15.878Z | v1.10.9 88323 | load_balancing_call | [2] write() called with message of length 94
D 2024-06-22T08:40:15.878Z | v1.10.9 88323 | retrying_call | [1] halfClose called
D 2024-06-22T08:40:15.901Z | v1.10.9 88323 | transport | dns:firestore.googleapis.com:443 connection failed with error 0078E7E29D7F0000:error:0A00007B:SSL routines:tls_process_cert_verify:bad signature:../deps/openssl/openssl/ssl/statem/statem_lib.c:559:

D 2024-06-22T08:40:15.901Z | v1.10.9 88323 | subchannel | (1) [2a00:1450:400e:802::200a]:443 CONNECTING -> TRANSIENT_FAILURE
D 2024-06-22T08:40:15.902Z | v1.10.9 88323 | pick_first | Start connecting to subchannel with address 142.250.179.170:443
D 2024-06-22T08:40:15.902Z | v1.10.9 88323 | subchannel | (2) 142.250.179.170:443 IDLE -> CONNECTING
D 2024-06-22T08:40:15.902Z | v1.10.9 88323 | transport | dns:firestore.googleapis.com:443 creating HTTP/2 session to 142.250.179.170:443
D 2024-06-22T08:40:15.924Z | v1.10.9 88323 | transport | dns:firestore.googleapis.com:443 connection failed with error 0078E7E29D7F0000:error:0A00007B:SSL routines:tls_process_cert_verify:bad signature:../deps/openssl/openssl/ssl/statem/statem_lib.c:559:

D 2024-06-22T08:40:15.924Z | v1.10.9 88323 | subchannel | (2) 142.250.179.170:443 CONNECTING -> TRANSIENT_FAILURE
D 2024-06-22T08:40:15.925Z | v1.10.9 88323 | dns_resolver | resolution update delayed by "min time between resolutions" rate limit
D 2024-06-22T08:40:15.925Z | v1.10.9 88323 | pick_first | CONNECTING -> TRANSIENT_FAILURE
D 2024-06-22T08:40:15.925Z | v1.10.9 88323 | resolving_load_balancer | dns:firestore.googleapis.com:443 CONNECTING -> TRANSIENT_FAILURE
D 2024-06-22T08:40:15.925Z | v1.10.9 88323 | channel | (1) dns:firestore.googleapis.com:443 callRefTimer.unref | configSelectionQueue.length=0 pickQueue.length=0
D 2024-06-22T08:40:15.925Z | v1.10.9 88323 | load_balancing_call | [2] Pick called
D 2024-06-22T08:40:15.925Z | v1.10.9 88323 | load_balancing_call | [2] Pick result: TRANSIENT_FAILURE subchannel: null status: 14 No connection established. Last error: 0078E7E29D7F0000:error:0A00007B:SSL routines:tls_process_cert_verify:bad signature:../deps/openssl/openssl/ssl/statem/statem_lib.c:559:
 (2024-06-22T08:40:15.924Z)
D 2024-06-22T08:40:15.926Z | v1.10.9 88323 | connectivity_state | (1) dns:firestore.googleapis.com:443 CONNECTING -> TRANSIENT_FAILURE
D 2024-06-22T08:40:15.927Z | v1.10.9 88323 | load_balancing_call | [2] ended with status: code=14 details="No connection established. Last error: 0078E7E29D7F0000:error:0A00007B:SSL routines:tls_process_cert_verify:bad signature:../deps/openssl/openssl/ssl/statem/statem_lib.c:559:
 (2024-06-22T08:40:15.924Z)" start time=2024-06-22T08:40:15.877Z
D 2024-06-22T08:40:15.927Z | v1.10.9 88323 | retrying_call | [1] Received status from child [2]
D 2024-06-22T08:40:15.927Z | v1.10.9 88323 | retrying_call | [1] state=TRANSPARENT_ONLY handling status with progress PROCESSED from child [2] in state ACTIVE
D 2024-06-22T08:40:15.928Z | v1.10.9 88323 | retrying_call | [1] ended with status: code=14 details="No connection established. Last error: 0078E7E29D7F0000:error:0A00007B:SSL routines:tls_process_cert_verify:bad signature:../deps/openssl/openssl/ssl/statem/statem_lib.c:559:
 (2024-06-22T08:40:15.924Z)" start time=2024-06-22T08:40:15.876Z
D 2024-06-22T08:40:15.928Z | v1.10.9 88323 | resolving_call | [0] Received status
D 2024-06-22T08:40:15.928Z | v1.10.9 88323 | resolving_call | [0] ended with status: code=14 details="No connection established. Last error: 0078E7E29D7F0000:error:0A00007B:SSL routines:tls_process_cert_verify:bad signature:../deps/openssl/openssl/ssl/statem/statem_lib.c:559:
 (2024-06-22T08:40:15.924Z)"
D 2024-06-22T08:40:15.931Z | v1.10.9 88323 | resolving_call | [0] cancelWithStatus code: 1 details: "Cancelled on client"
D 2024-06-22T08:40:15.931Z | v1.10.9 88323 | retrying_call | [1] cancelWithStatus code: 1 details: "Cancelled on client"
D 2024-06-22T08:40:15.931Z | v1.10.9 88323 | retrying_call | [1] ended with status: code=1 details="Cancelled on client" start time=2024-06-22T08:40:15.876Z
D 2024-06-22T08:40:15.931Z | v1.10.9 88323 | load_balancing_call | [2] cancelWithStatus code: 1 details: "Cancelled on client"
D 2024-06-22T08:40:15.932Z | v1.10.9 88323 | resolving_call | [0] Received status
D 2024-06-22T08:40:16.435Z | v1.10.9 88323 | channel | (1) dns:firestore.googleapis.com:443 createResolvingCall [3] method="/google.firestore.v1.Firestore/BatchGetDocuments", deadline=2024-06-22T08:45:16.435Z
D 2024-06-22T08:40:16.435Z | v1.10.9 88323 | resolving_call | [3] Created
D 2024-06-22T08:40:16.435Z | v1.10.9 88323 | resolving_call | [3] Deadline: 2024-06-22T08:45:16.435Z
D 2024-06-22T08:40:16.435Z | v1.10.9 88323 | resolving_call | [3] Deadline will be reached in 300000ms
D 2024-06-22T08:40:16.435Z | v1.10.9 88323 | resolving_call | [3] start called
D 2024-06-22T08:40:16.436Z | v1.10.9 88323 | resolving_call | [3] write() called with message of length 106
D 2024-06-22T08:40:16.436Z | v1.10.9 88323 | resolving_call | [3] halfClose called
D 2024-06-22T08:40:16.437Z | v1.10.9 88323 | channel | (1) dns:firestore.googleapis.com:443 createRetryingCall [4] method="/google.firestore.v1.Firestore/BatchGetDocuments"
D 2024-06-22T08:40:16.437Z | v1.10.9 88323 | resolving_call | [3] Created child [4]
D 2024-06-22T08:40:16.437Z | v1.10.9 88323 | retrying_call | [4] start called
D 2024-06-22T08:40:16.437Z | v1.10.9 88323 | channel | (1) dns:firestore.googleapis.com:443 createLoadBalancingCall [5] method="/google.firestore.v1.Firestore/BatchGetDocuments"
D 2024-06-22T08:40:16.437Z | v1.10.9 88323 | retrying_call | [4] Created child call [5] for attempt 1
D 2024-06-22T08:40:16.437Z | v1.10.9 88323 | load_balancing_call | [5] start called
D 2024-06-22T08:40:16.437Z | v1.10.9 88323 | load_balancing_call | [5] Pick called
D 2024-06-22T08:40:16.437Z | v1.10.9 88323 | load_balancing_call | [5] Pick result: TRANSIENT_FAILURE subchannel: null status: 14 No connection established. Last error: 0078E7E29D7F0000:error:0A00007B:SSL routines:tls_process_cert_verify:bad signature:../deps/openssl/openssl/ssl/statem/statem_lib.c:559:
 (2024-06-22T08:40:15.924Z)
D 2024-06-22T08:40:16.437Z | v1.10.9 88323 | retrying_call | [4] write() called with message of length 111
D 2024-06-22T08:40:16.437Z | v1.10.9 88323 | load_balancing_call | [5] write() called with message of length 111
D 2024-06-22T08:40:16.437Z | v1.10.9 88323 | retrying_call | [4] halfClose called
D 2024-06-22T08:40:16.437Z | v1.10.9 88323 | resolving_call | [3] startRead called
D 2024-06-22T08:40:16.437Z | v1.10.9 88323 | retrying_call | [4] startRead called
D 2024-06-22T08:40:16.437Z | v1.10.9 88323 | load_balancing_call | [5] startRead called
D 2024-06-22T08:40:16.438Z | v1.10.9 88323 | load_balancing_call | [5] ended with status: code=14 details="No connection established. Last error: 0078E7E29D7F0000:error:0A00007B:SSL routines:tls_process_cert_verify:bad signature:../deps/openssl/openssl/ssl/statem/statem_lib.c:559:
 (2024-06-22T08:40:15.924Z)" start time=2024-06-22T08:40:16.437Z
D 2024-06-22T08:40:16.438Z | v1.10.9 88323 | retrying_call | [4] Received status from child [5]
D 2024-06-22T08:40:16.438Z | v1.10.9 88323 | retrying_call | [4] state=TRANSPARENT_ONLY handling status with progress PROCESSED from child [5] in state ACTIVE
D 2024-06-22T08:40:16.438Z | v1.10.9 88323 | retrying_call | [4] ended with status: code=14 details="No connection established. Last error: 0078E7E29D7F0000:error:0A00007B:SSL routines:tls_process_cert_verify:bad signature:../deps/openssl/openssl/ssl/statem/statem_lib.c:559:
 (2024-06-22T08:40:15.924Z)" start time=2024-06-22T08:40:16.437Z
D 2024-06-22T08:40:16.438Z | v1.10.9 88323 | resolving_call | [3] Received status
D 2024-06-22T08:40:16.438Z | v1.10.9 88323 | resolving_call | [3] ended with status: code=14 details="No connection established. Last error: 0078E7E29D7F0000:error:0A00007B:SSL routines:tls_process_cert_verify:bad signature:../deps/openssl/openssl/ssl/statem/statem_lib.c:559:

Output of openssl s_client --connect firestore.googleapis.com:443

CONNECTED(00000003)
depth=2 C = US, O = Google Trust Services LLC, CN = GTS Root R1
verify return:1
depth=1 C = US, O = Google Trust Services, CN = WR2
verify return:1
depth=0 CN = edgecert.googleapis.com
verify return:1
---
Certificate chain
 0 s:CN = edgecert.googleapis.com
   i:C = US, O = Google Trust Services, CN = WR2
   a:PKEY: id-ecPublicKey, 256 (bit); sigalg: RSA-SHA256
   v:NotBefore: Jun  3 07:51:59 2024 GMT; NotAfter: Aug 26 07:51:58 2024 GMT
 1 s:C = US, O = Google Trust Services, CN = WR2
   i:C = US, O = Google Trust Services LLC, CN = GTS Root R1
   a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
   v:NotBefore: Dec 13 09:00:00 2023 GMT; NotAfter: Feb 20 14:00:00 2029 GMT
 2 s:C = US, O = Google Trust Services LLC, CN = GTS Root R1
   i:C = BE, O = GlobalSign nv-sa, OU = Root CA, CN = GlobalSign Root CA
   a:PKEY: rsaEncryption, 4096 (bit); sigalg: RSA-SHA256
   v:NotBefore: Jun 19 00:00:42 2020 GMT; NotAfter: Jan 28 00:00:42 2028 GMT
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIGKTCCBRGgAwIBAgIQVp3lrpaEN98Sdkufg00nPjANBgkqhkiG9w0BAQsFADA7
MQswCQYDVQQGEwJVUzEeMBwGA1UEChMVR29vZ2xlIFRydXN0IFNlcnZpY2VzMQww
CgYDVQQDEwNXUjIwHhcNMjQwNjAzMDc1MTU5WhcNMjQwODI2MDc1MTU4WjAiMSAw
HgYDVQQDExdlZGdlY2VydC5nb29nbGVhcGlzLmNvbTBZMBMGByqGSM49AgEGCCqG
SM49AwEHA0IABPVEiYYKItUr+ajG0/HYFHi8pX8rl585ElUDiOccUSuXq4c4JY2I
39LG4SS6GgIRlp5ypZiXPbBLppVQYOsREzWjggQLMIIEBzAOBgNVHQ8BAf8EBAMC
B4AwEwYDVR0lBAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAdBgNVHQ4EFgQU
ZPuP0ZQ+xcjpGkHW06yG5V47IxEwHwYDVR0jBBgwFoAU3hse7XkV1D43JMMhu+w0
OW1CsjAwWAYIKwYBBQUHAQEETDBKMCEGCCsGAQUFBzABhhVodHRwOi8vby5wa2ku
Z29vZy93cjIwJQYIKwYBBQUHMAKGGWh0dHA6Ly9pLnBraS5nb29nL3dyMi5jcnQw
ggHgBgNVHREEggHXMIIB04IXZWRnZWNlcnQuZ29vZ2xlYXBpcy5jb22CFmFuZHJv
aWQuZ29vZ2xlYXBpcy5jb22CF2JpZ3RhYmxlLmdvb2dsZWFwaXMuY29tgiJjbG91
ZGVycm9ycmVwb3J0aW5nLmdvb2dsZWFwaXMuY29tghljbG91ZHRyYWNlLmdvb2ds
ZWFwaXMuY29tghhjb250YWluZXIuZ29vZ2xlYXBpcy5jb22CF2RhdGFmbG93Lmdv
b2dsZWFwaXMuY29tghhkYXRhc3RvcmUuZ29vZ2xlYXBpcy5jb22CEmRscC5nb29n
bGVhcGlzLmNvbYISZmNtLmdvb2dsZWFwaXMuY29tghhmaXJlc3RvcmUuZ29vZ2xl
YXBpcy5jb22CFmxvZ2dpbmcuZ29vZ2xlYXBpcy5jb22CGW1vbml0b3JpbmcuZ29v
Z2xlYXBpcy5jb22CFXB1YnN1Yi5nb29nbGVhcGlzLmNvbYIccnVudGltZWNvbmZp
Zy5nb29nbGVhcGlzLmNvbYIdc2VydmljZWNvbnRyb2wuZ29vZ2xlYXBpcy5jb22C
FnNwYW5uZXIuZ29vZ2xlYXBpcy5jb22CGnN0YWNrZHJpdmVyLmdvb2dsZWFwaXMu
Y29tMBMGA1UdIAQMMAowCAYGZ4EMAQIBMDYGA1UdHwQvMC0wK6ApoCeGJWh0dHA6
Ly9jLnBraS5nb29nL3dyMi83NXI0WnlBM3ZBMC5jcmwwggEFBgorBgEEAdZ5AgQC
BIH2BIHzAPEAdwB2/4g/Crb7lVHCYcz1h7o0tKTNuyncaEIKn+ZnTFo6dAAAAY/d
TF7/AAAEAwBIMEYCIQChAOvcpwYXu2ym4DJZ01vgjr0wOouDHQvj8hhLXxVTmgIh
AKIthNk5ZRdvi+Uad3i19OTu3MbhjmJ/W8EenQOUWx47AHYASLDja9qmRzQP5WoC
+p0w6xxSActW3SyB2bu/qznYhHMAAAGP3Uxe8QAABAMARzBFAiEAgEUEtoFwT0rX
iS+dOTo2rHLm33tG49AZ8qRsdTAl+AICIH66FMhAKCcnG5ub+UXzKisJjo64ZYL5
lafEuENCJ+YfMA0GCSqGSIb3DQEBCwUAA4IBAQCNhYNTOlI4RuHDPeY+u7YacbGB
5tfHH1vUzmWIcMcNZsNXbhvi1IBZ5FMH0AWpiWZhAi4aPB9tfhbCFlFxjN7qD3yJ
9KkIQq7GfXBWfaXc0FSjwKqe4KvvtBKKTtiPYjDZxBiuu8YRrUor8vXbjDUvLyAx
1OYD8jWap5q53lRv60deobXNrjqtK75SU9eq0NPYCigeb353PgZ75hlFJBxiKQrH
F6GGw9mmKY8IDamth30jz6guXYerQCbsuxNvcPuXYt8ckVFfmnR8+Sm1Mr3PG2QC
F86Y+OL9kv5IgXwabv/HcciPD3HVjJ7RupnYr608Kmv6MBnsF0kP0ndriHZc
-----END CERTIFICATE-----
subject=CN = edgecert.googleapis.com
issuer=C = US, O = Google Trust Services, CN = WR2
---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: ECDSA
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 4574 bytes and written 406 bytes
Verification: OK
---
New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
Server public key is 256 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---

[luke-rogers]

We ran into a very similar-sounding issue with our gen2 nest js application. We are using auth guards to call verifyIdToken and these were failing in the latest release 12.2.0. Reverting to 12.1.1 fixed the issue.

[asenmitrev]

We ran into a very similar-sounding issue with our gen2 nest js application. We are using auth guards to call verifyIdToken and these were failing in the latest release 12.2.0. Reverting to 12.1.1 fixed the issue.

Reproduced the issue with 12.1.1.

[christophe-g]

Possibly related, upgrading from 12.1.1 to 12.2.0 breaks unit test that call getAuth().getUser(uid) (and possibly other auth methods).

error message:

  errorInfo: {
    code: 'auth/internal-error',
    message: '//console.developers.google.com/iam-admin/iam/project?project=ida-ta and then retry. Propagation of the new permission may take a few minutes. Raw server response: "{"error":{"code":403,"message":"Caller does not have required permission to use project ida-ta. Grant the caller the roles/serviceusage.serviceUsageConsumer role, or a custom role with the serviceusage.services.use permission, by visiting https://console.developers.google.com/iam-admin/iam/project?project=ida-ta and then retry. Propagation of the new permission may take a few minutes.","errors":[{"message":"Caller does not have required permission to use project ida-ta. Grant the caller the roles/serviceusage.serviceUsageConsumer role, or a custom role with the serviceusage.services.use permission, by visiting https://console.developers.google.com/iam-admin/iam/project?project=ida-ta and then retry. Propagation of the new permission may take a few minutes.","domain":"global","reason":"forbidden"}],"status":"PERMISSION_DENIED","details":[{"@type":"type.googleapis.com/google.rpc.Help","links":[{"description":"Google developer console IAM admin","url":"https://console.developers.google.com/iam-admin/iam/project?project=ida-ta"}]},{"@type":"type.googleapis.com/google.rpc.ErrorInfo","reason":"USER_PROJECT_DENIED","domain":"googleapis.com","metadata":{"service":"identitytoolkit.googleapis.com","consumer":"projects/ida-ta"}}]}}"'
  },
  codePrefix: 'auth'

reverting back to 12.1.1.

[lahirumaramba]

The auth related issues you are seeing could be related to us adding x-goog-user-project headers in #2553
@christophe-g did you try granting the roles/serviceusage.serviceUsageConsumer IAM role to the service account in your project?

I think what @asenmitrev reported is a different issue. @asenmitrev are you able to reproduce this issue in 12.1.1?

[asenmitrev]

I also get a different error when using only auth, although the root cause still seems to be due to ssl failures. Including logs from an app using only token verification through firebase.

Error verifying Firebase token: _FirebaseAuthError: Error while making request: write EPROTO 00D8BCA1BA7F0000:error:0A00007B:SSL routines:tls_process_cert_verify:bad signature:../deps/openssl/openssl/ssl/statem/statem_lib.c:559:
. Error code: EPROTO
    at FirebaseTokenVerifier.mapJwtErrorToAuthError (/srv/www/video-ai/dist_server/index.js:141346:16)
    at /srv/www/video-app/dist_server/index.js:141325:22
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5) {
errorInfo: {
    code: 'auth/argument-error',
    message: 'Error while making request: write EPROTO 00D8BCA1BA7F0000:error:0A00007B:SSL routines:tls_process_cert_verify:bad signature:../deps/openssl/openssl/ssl/statem/statem_lib.c:559:\n' +
    '. Error code: EPROTO'
},
codePrefix: 'auth'
}

Firebase version is at 12.0.0 for the above error. Does not get fixed with 12.1.1 or 12.2.0

[asenmitrev]

The issue was with any SSL connection through node, due to an old version of node. Upgrading to an LTS version of node (in my case 20.15.1) fixes the issue. Closing this issue.