Federation target which fail due to non-HTTP application layer errors are not marked as unavailable

[ara4n]

The mess in #1733 was exacerbated significantly by servers going and wasting lots of time querying dead servers for missing events. It looks like we're failing to mark servers as dead if they respond with transport level failures. Looking at logs from arasphere, examples are:

2016-12-30 11:35:23,255 - synapse.http.matrixfederationclient - 183 - WARNING -  - {GET-O-11031} Sending request failed to lefil.tsar.tm.fr: GET matrix://lefil.tsar.tm.fr/_matrix/federation/v1/event/$14168370037dJyNv:arasphere.net/: ConnectionRefusedError - ConnectionRefusedError: Connection refused
2016-12-30 11:35:23,870 - synapse.http.matrixfederationclient - 183 - WARNING -  - {GET-O-11031} Sending request failed to lefil.tsar.tm.fr: GET matrix://lefil.tsar.tm.fr/_matrix/federation/v1/event/$14168370037dJyNv:arasphere.net/: ConnectionRefusedError - ConnectionRefusedError: Connection refused
2016-12-30 11:35:24,845 - synapse.http.matrixfederationclient - 183 - WARNING -  - {GET-O-11031} Sending request failed to lefil.tsar.tm.fr: GET matrix://lefil.tsar.tm.fr/_matrix/federation/v1/event/$14168370037dJyNv:arasphere.net/: ConnectionRefusedError - ConnectionRefusedError: Connection refused
2016-12-30 11:35:27,195 - synapse.http.matrixfederationclient - 183 - WARNING -  - {GET-O-11031} Sending request failed to lefil.tsar.tm.fr: GET matrix://lefil.tsar.tm.fr/_matrix/federation/v1/event/$14168370037dJyNv:arasphere.net/: ConnectionRefusedError - ConnectionRefusedError: Connection refused

2016-12-30 11:36:30,577 - synapse.http.matrixfederationclient - 183 - WARNING -  - {GET-O-11036} Sending request failed to threespeed.org: GET matrix://threespeed.org/_matrix/federation/v1/event/$14168370037dJyNv:arasphere.net/: CancelledError - CancelledError: 
2016-12-30 11:37:31,034 - synapse.http.matrixfederationclient - 183 - WARNING -  - {GET-O-11036} Sending request failed to threespeed.org: GET matrix://threespeed.org/_matrix/federation/v1/event/$14168370037dJyNv:arasphere.net/: CancelledError - CancelledError: 
2016-12-30 11:38:31,883 - synapse.http.matrixfederationclient - 183 - WARNING -  - {GET-O-11036} Sending request failed to threespeed.org: GET matrix://threespeed.org/_matrix/federation/v1/event/$14168370037dJyNv:arasphere.net/: CancelledError - CancelledError: 
2016-12-30 11:39:33,708 - synapse.http.matrixfederationclient - 183 - WARNING -  - {GET-O-11036} Sending request failed to threespeed.org: GET matrix://threespeed.org/_matrix/federation/v1/event/$14168370037dJyNv:arasphere.net/: CancelledError - CancelledError:

2016-12-30 14:02:02,029 - synapse.http.matrixfederationclient - 183 - WARNING - PUT-33093 - {GET-O-24210} Sending request failed to vil.lv: GET matrix://vil.lv/_matrix/federation/v1/event/$14168370037dJyNv:arasphere.net/: TimeoutError - TimeoutError: 
2016-12-30 14:02:12,716 - synapse.http.matrixfederationclient - 183 - WARNING - PUT-33093 - {GET-O-24210} Sending request failed to vil.lv: GET matrix://vil.lv/_matrix/federation/v1/event/$14168370037dJyNv:arasphere.net/: TimeoutError - TimeoutError: 
2016-12-30 14:02:23,877 - synapse.http.matrixfederationclient - 183 - WARNING - PUT-33093 - {GET-O-24210} Sending request failed to vil.lv: GET matrix://vil.lv/_matrix/federation/v1/event/$14168370037dJyNv:arasphere.net/: TimeoutError - TimeoutError: 
2016-12-30 14:02:36,540 - synapse.http.matrixfederationclient - 183 - WARNING - PUT-33093 - {GET-O-24210} Sending request failed to vil.lv: GET matrix://vil.lv/_matrix/federation/v1/event/$14168370037dJyNv:arasphere.net/: TimeoutError - TimeoutError:

[ara4n]

relates to #1404

[richvdh]

Hopefully fixed by #2050