retry CPI calls in case of SSL_read errors as well

[anshrupani]

Recently, intermittently broken connections, most probably due to IaaS network issues have been met with SSL_read errors as follows with various CPI calls (atleast noticed for delete_vm and attach_disk calls):

ERROR -- DirectorJobRunner: Unknown CPI error 'Unknown' with message 'SSL_read: unexpected eof while reading' in 'delete_vm' CPI method

On a bit of digging, it seems that almost all languages except ruby have implemented workarounds to meet with this common OpenSSL3 error (see this issue). At the same time, the azure-cpi has implemented retries for OpenSSL::SSL::SSLError class, however, this is only true for 'SSL_connect' errors (see the previous implementation in the PR).

With this PR, to circumvent such network issues, we also retry in case of 'SSL_read' errors.

[metskem]

We have the exact same issues, during delete_vm, set_vm_metadata and more calls, and it is blocking us from doing further upgrades.

[anshrupani]

I guess I now have a more clear picture of what's happening.

Weird broken connections can occur unexpectedly anytime. There is a difference how OpenSSL 1.x dealt with such broken connections resulting in unexpected EOFs and how OpenSSL 3.x deals with such issues. It seems that OpenSSL 3.x has a much more strict behavior about unexpected EOF. In a recent PR towards the bosh director, the OpenSSL version was bumped to 3.2.0, which seems to have woken up the giant which was always there sleeping, atleast with Azure. Other languges such as Python already have workarounds to deal with this issue intrinsically by default, but unfortunately that's not the case with ruby. There are further detailed explanations regarding the issue and how people have dealt with such issues:

1. https://stackoverflow.com/questions/76183622/since-a-ruby-container-upgrade-we-expirience-a-lot-of-opensslsslsslerror
2. OpenSSL::SSL::SSLError: SSL_read: unexpected eof while reading redis/redis-rb#1106

In short, I feel retries should help in general in such cases and we should first try implementing retries. If not, people also suggest raising OpenSSL::SSL::OP_IGNORE_UNEXPECTED_EOF flag as a workaround, but that comes with other possible concerns.

[lodener]

@anshrupani I believe you are missing an additional colon between Helpers and ERROR_OPENSSL_EOF_READ.

[anshrupani]

yes you're right, thanks. we're still testing the PR on our environment though

[lodener]

This issue made using the Azure CPI near impossible for us, as we'd run into constant SSL_read errors.

When running a release with these changes, things have been working much better so far, turning the errors into retries (with the CPI log showcasing that this is happening quite a bit more than we'd like, but at least it doesn't kill the entire deployment anymore).

[jpalermo]

If not, people also suggest raising OpenSSL::SSL::OP_IGNORE_UNEXPECTED_EOF flag as a workaround, but that comes with other possible concerns.

Do you know what the concerns might be @anshrupani?

[mvach]

I just remove this whole retry logic from this fix and it is still working. Let's test it again tomorrow and then get that in.

[anshrupani]

If not, people also suggest raising OpenSSL::SSL::OP_IGNORE_UNEXPECTED_EOF flag as a workaround, but that comes with other possible concerns.

Do you know what the concerns might be @anshrupani?

Openssl.org mentions possibility of truncation attacks when this option is set and the protocol running over TLS doesn't handle such vulnerabilities. However, we should also consider that before OpenSSL3, having this option set was anyhow the case by default while we used Net::HTTP, and that other languages still have this option set by default for OpenSSL >= 3.