[azeventhubs] Fixing issues with connection and link robustness for Event Hubs

[richardpark-msft]

Fixing issues with connection and link robustness for Event Hubs.

There were some issues with how we do recovery, which could make things
hang or even leave us in an inconsistent state.

1. We didn't have any bound on how long a Close() could take, but since
   that requires a network hop it's possible for it to hang infinitely if
   the service doesn't respond. This now has a max bound of 60 seconds.

2. If a Close() of a link is cancelled we signal that the connection needs
   to be reset, since not doing that will leave us in an inconsistent state
   where we believe a link is active, but the service does not.

3. We now ignore errors that come back from trying to close a link (except
   for the previously mentioned cancellation). The error that comes back will
   be redundant (ie, it'll be the same error that initiated the recovery in
   the first place) or it'll be something we can't immediately troubleshoot
   and will be dealt with on the next recovery round.

There are also a number of testing quality fixes that I made to validate
this change. I've got gomock now for mocks in newer tests (and will start
to phase out the previous uses of my custom mocks). I've also added some
more tests around recovery and improved logging to log more context per
line so it's easier to correlate activity between links.

[jhendrixMSFT]

If I understand correctly, we add a timeout via context to the close operation, and if close times out (or is cancelled), then we consider the amqp.Conn to be in a bad state and recreate it. Did I get that right?

[richardpark-msft]

/azp run go - azeventhubs

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[richardpark-msft]

/azp run go - azeventhubs

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).