We keep trying to fetch events whose keys have expired or have been invalidated

[richvdh]

Sometimes we try to fetch an event over federation (notably, if it forms part of the state or auth chain), and we try to validate its signature,

However, if we are unable to perform that validation, then we discard the event; but we might do the same again almost immediately. Possibly we need some sort of backoff.

[MadLittleMods]

Possibly we need some sort of backoff.

Tracked by #13622 and #13623

[richvdh]

Tracked by #13622...

I think #13622 is more about trying to /backfill from a given backwards extremity, than requesting individual /events? But if your point is they both need a backoff mechanism, then yes!

[MadLittleMods]

@richvdh I thought this was mainly talking about /backfill since it was mentioned in #13677. This situation does happen for backfill because we either request /state or /event when doing _get_state_ids_after_missing_prev_event() and #13622 will stop us getting into that situation over and over.

But I see how we could call /event for other things 👍

[MadLittleMods]

👍 I've added this as part of #13700 to track signature failures.

By having some backoff and processing in the background, we can save 128s that _get_state_ids_after_missing_prev_event takes for #matrix:matrix.org when backfilling messages.

To explain an exact scenario around /messages -> backfill, we call /backfill and first check the signatures of the 100 events. We see bad signature for $luA4l7QHhf_jadH3mI-AyFqho0U2Q-IXXUbGSMq6h6M and $zuOn2Rd2vsC7SUia3Hp3r6JSkSFKcc5j3QTTqW_0jDw (both member events). Then we process the 98 events remaining that have valid signatures but one of the events references $luA4l7QHhf_jadH3mI-AyFqho0U2Q-IXXUbGSMq6h6M as a prev_event. So we have to do the whole _get_state_ids_after_missing_prev_event rigmarole which pulls in those same events which fail again because the signatures are still invalid.

• backfill
  • outgoing-federation-request /backfill
  • _check_sigs_and_hash_and_fetch
    • _check_sigs_and_hash_and_fetch_one for each event received over backfill
      • ❗ $luA4l7QHhf_jadH3mI-AyFqho0U2Q-IXXUbGSMq6h6M fails with Signature on retrieved event was invalid.: unable to verify signature for sender domain xxx: 401: Failed to find any key to satisfy: _FetchKeyRequest(...)
      • ❗ $zuOn2Rd2vsC7SUia3Hp3r6JSkSFKcc5j3QTTqW_0jDw fails with Signature on retrieved event was invalid.: unable to verify signature for sender domain xxx: 401: Failed to find any key to satisfy: _FetchKeyRequest(...)
  • _process_pulled_events
    • _process_pulled_event for each validated event
      • ❗ Event $Q0iMdqtz3IJYfZQU2Xk2WjB5NDF8Gg8cFSYYyKQgKJ0 references $luA4l7QHhf_jadH3mI-AyFqho0U2Q-IXXUbGSMq6h6M as a prev_event which is missing so we try to get it
        • _get_state_ids_after_missing_prev_event
          • outgoing-federation-request /state_ids
          • ❗ get_pdu for $luA4l7QHhf_jadH3mI-AyFqho0U2Q-IXXUbGSMq6h6M which fails the signature check again
          • ❗ get_pdu for $zuOn2Rd2vsC7SUia3Hp3r6JSkSFKcc5j3QTTqW_0jDw which fails the signature check