Better handling of (reverse) connection issues in federation requests

[lukas2511]

Scenario

To start this of I'll just describe the scenario I just went through:

In an encrypted group chat the messages of a few people, all with matrix.org accounts, were suddenly not able to be decrypted.

The client lets it seem like it's a local issue, and maybe another client still needs to push the keys, but since all my clients showed the same message it was clear that this is not the case.

Looking at the server log I found a lot of lines similar to this:

2023-03-17 00:33:04,249 - synapse.http.matrixfederationclient - 630 - INFO - federation_transaction_transmission_loop-711 - {PUT-O-763} [matrix.org] Got response headers: 401 Unauthorized
2023-03-17 00:33:04,249 - synapse.http.matrixfederationclient - 710 - WARNING - federation_transaction_transmission_loop-711 - {PUT-O-763} [matrix.org] Request failed: PUT matrix://matrix.org/_matrix/federation/v1/send/1679006060366: HttpResponseException('401: Unauthorized')

No further information is given. Just a 401 Unauthorized, even with DEBUG logging enabled.

I went ahead and figured out the parameters of the request to send it myself, but that only got me to another unclear error message:

{"errcode":"M_UNAUTHORIZED","error":"Failed to find any key to satisfy: _FetchKeyRequest( etc etc etc

From a few other issues I found that people were having similar issues when their TLS certificates weren't valid, but that wasn't the case here.

But that got me to think that perhaps there actually is some type of connection issue, and yea, after some digging with tcpdump I found that the incoming TCP session from the matrix.org server wasn't fully established. It seemed to be a routing issue in this case, where the connectivity between Mythic Beasts and one of my transit providers was somehow broken, and after disabling those routes everything worked perfectly.

I probably spent a few hours debugging this, because there was absolutely no indication of what was actually going wrong. That was really annoying.

Suggestion

When federation requests are sent out and a key lookup in the reverse direction fails it would be great to have additional information delivered with that error.

Something like...

{
  "errcode":"M_UNAUTHORIZED",
  "error":"Failed to find any key to satisfy: _FetchKeyRequest( etc etc etc",
  "details": "Unable to connect to $servername:$port: Connection timed out"
}

...which then also gets printed to the server logs instead of the simple "401 Unauthorized" would be great.

The error messages could be really helpful, here are some more examples:

• DNS lookup failed
• Connection refused
• Connection closed without receiving data
• TLS certificate invalid
• TLS certificate expired

That would make debugging those kind of issues a lot easier. With $servername:$port given in the details you would also see if it connects to the right server, which sometimes seems to be an issue with people hosting their servers on a subdomain.

I really hope this gets implemented as this seems like a very important debugging feature in a federated system.