Timeout when getting missing events from a broken server can be 5 minutes, which is way too long.

[ara4n]

#1733 is still a problem in part because even querying a single server for the missing event can take 5 minutes to time out, and the lock in #1729 meanwhile causes other federation traffic to stack up alarmingly.

An example broken request taking a while to timeout:

2016-12-30 16:44:03,169 - synapse.access.https.8448 - 59 - INFO - PUT-5559- - - 8448 - Received request: PUT /_matrix/federation/v1/send/1483048713777/
2016-12-30 16:44:03,178 - synapse.federation.transport.server - 138 - INFO - PUT-5559- Request from onedefence.com
2016-12-30 16:44:03,179 - synapse.federation.transport.server - 244 - INFO - PUT-5559- Received txn 1483048713777 from onedefence.com. (PDUs: 1, EDUs: 0, failures: 0)
2016-12-30 16:44:03,270 - synapse.federation.federation_server - 543 - INFO - PUT-5559- Acquiring lock for room '!cURbafjkfsMDVwdRDQ:matrix.org' to fetch 6 missing events: ['$1483048123119291vFtkJ:matrix.org', '$14829396373282755RweJJ:matrix.org', '$1482944175413cmBcv:cntalk.xyz', '$14829329223245920ZaaAE:matrix.org', '$14829239492mbLyx:krtdex.com']...
2016-12-30 16:44:03,271 - synapse.util.async - 188 - INFO - PUT-5559- Waiting to acquire linearizer lock for key '!cURbafjkfsMDVwdRDQ:matrix.org'
2016-12-30 16:45:24,820 - synapse.federation.federation_server - 548 - INFO - PUT-5559- Acquired lock for room '!cURbafjkfsMDVwdRDQ:matrix.org' to fetch 6 missing events
2016-12-30 16:45:25,326 - synapse.federation.federation_server - 567 - INFO - PUT-5559- Missing 1 events for room '!cURbafjkfsMDVwdRDQ:matrix.org': ['$14829239492mbLyx:krtdex.com']...
2016-12-30 16:45:25,327 - synapse.http.outbound - 124 - INFO - PUT-5559- {POST-O-9683} [onedefence.com] Sending request: POST matrix://onedefence.com/_matrix/federation/v1/get_missing_events/!cURbafjkfsMDVwdRDQ:matrix.org
2016-12-30 16:46:25,332 - synapse.http.matrixfederationclient - 183 - WARNING - PUT-5559- {POST-O-9683} Sending request failed to onedefence.com: POST matrix://onedefence.com/_matrix/federation/v1/get_missing_events/!cURbafjkfsMDVwdRDQ:matrix.org: CancelledError - CancelledError: 
2016-12-30 16:46:27,741 - synapse.handlers.typing - 79 - INFO - PUT-5559- Checking for typing timeouts
2016-12-30 16:46:28,632 - synapse.handlers.presence - 328 - INFO - PUT-5559- Handling presence timeouts
2016-12-30 16:47:26,028 - synapse.http.matrixfederationclient - 183 - WARNING - PUT-5559- {POST-O-9683} Sending request failed to onedefence.com: POST matrix://onedefence.com/_matrix/federation/v1/get_missing_events/!cURbafjkfsMDVwdRDQ:matrix.org: CancelledError - CancelledError: 
2016-12-30 16:48:27,391 - synapse.http.matrixfederationclient - 183 - WARNING - PUT-5559- {POST-O-9683} Sending request failed to onedefence.com: POST matrix://onedefence.com/_matrix/federation/v1/get_missing_events/!cURbafjkfsMDVwdRDQ:matrix.org: CancelledError - CancelledError: 
2016-12-30 16:49:29,828 - synapse.http.matrixfederationclient - 183 - WARNING - PUT-5559- {POST-O-9683} Sending request failed to onedefence.com: POST matrix://onedefence.com/_matrix/federation/v1/get_missing_events/!cURbafjkfsMDVwdRDQ:matrix.org: CancelledError - CancelledError: 
2016-12-30 16:49:29,829 - synapse.http.outbound - 209 - INFO - PUT-5559- {POST-O-9683} [onedefence.com] Result: CancelledError - CancelledError: 
2016-12-30 16:49:29,838 - synapse.access.https.8448 - 91 - INFO - PUT-5559- - - 8448 - {onedefence.com} Processed request: 326665ms (66ms, 1ms) (29ms/4) 61B 200 "PUT /_matrix/federation/v1/send/1483048713777/ HTTP/1.1" "Lotus-Domino"