[TlsInterception] `OSError` when sending empty chunk to clients under `Python < 3.10`

[softhub-software-development]

So far I used proxy.py version 2.3.1 to develop a plugin. Since I switched to version 2.4.0rc2, I am getting the following
info/warning messages and TLS based access is denied. I tried with rc5 which issues the same messages on both the
Windows and macOS platform.

2022-01-11 15:13:15,840 - pid:1575 [W] handler.handle_readables:266 - Exception when receiving from client connection#29 with reason FileNotFoundError(2, 'No such file or directory')
2022-01-11 15:13:15,840 - pid:1576 [W] handler.handle_readables:266 - Exception when receiving from client connection#29 with reason FileNotFoundError(2, 'No such file or directory')
2022-01-11 15:13:15,840 - pid:1575 [I] server.access_log:406 - 127.0.0.1:50398 - CONNECT clientservices.googleapis.com:443 - 0 bytes - 50.59ms
2022-01-11 15:13:15,841 - pid:1576 [I] server.access_log:406 - 127.0.0.1:50400 - CONNECT accounts.google.com:443 - 0 bytes - 45.82ms
2022-01-11 15:13:16,290 - pid:1576 [W] handler.handle_readables:266 - Exception when receiving from client connection#29 with reason FileNotFoundError(2, 'No such file or directory')
2022-01-11 15:13:16,291 - pid:1576 [I] server.access_log:406 - 127.0.0.1:50404 - CONNECT www.google.com:443 - 0 bytes - 44.86ms

The browser states "The website is not reachable" and shows ERR_CONNECTION_CLOSED as the error message.

To my best knowledge, both configurations, v2.3.1 and 2.4.0rcX are using the same configuration and use the same
certificates but only v2.3.1 works. I am using Python 3.9.

I also notice that there are no cached server certificates in ~/.proxy/certificates when using 2.4.0rcX.

[abhinavsingh]

Thank you for reporting this. Let's dig deep. Can you also share the flags you are passing to the proxy instance.

[softhub-software-development]

I pass the following flags to proxy.py: '--hostname', '127.0.0.1', '--port', '8899', '--ca-key-file', 'ca-key.pem', '--ca-cert-file', 'ca-cert.pem', '--ca-signing-key-file', 'ca-signing-key.pem'

[abhinavsingh]

Thank you, I believe the reason we see No such file or directory is because certificate generation is failing for some reason when using v2.4.0 series. I may have to look into it. I used TLS interception last week, but certificates I have are trusted on the system.

Also, whenever you change CA certificates, you must delete ~/.proxy/certificates, otherwise, again you can run into similar issues. In, v2.4.0 we also changed how certificate serial number is generated.

To double check, can you

1. delete ~/.proxy/certificates
2. try again with v2.4.0

Also

1. delete ~/.proxy/certificates
2. Trust generated root CA on system
3. try again with v2.4.0

[abhinavsingh]

I also remember us changing default minimum SSL/TLS versions. Even that could have an impact here. But let's double check suggested workflows and then proceed to next step of investigation. Thank you.

[softhub-software-development]

To double check, can you

1. delete ~/.proxy/certificates
2. try again with v2.4.0

The folder ~/.proxy/certificates stays empty and accessing i.e. www.google.com results in "www.google.com
has closed the connection unexpectedly"

Also

1. delete ~/.proxy/certificates
2. Trust generated root CA on system
3. try again with v2.4.0

I ran "make ca-certificates" again, result is same as above. No new certificates in the cache at ~/.proxy/certificates.

[abhinavsingh]

Thank you again for highlighting this issue. I should try to delete my old certificates, un-trust them from system and try again. Looks like for some reason same workflow broken in 2.4.x series while it still works today in 2.3.x series.

We must also add an integration test for interception so that it is never found broken again. We only have unit tests, which assumes certificate generation works :(

[abhinavsingh]

I tried here with latest develop branch, and seems to work, see logs here

#979 (comment)

[softhub-software-development]

I tried here with latest develop branch, and seems to work, see logs here

#979 (comment)

I switched to the development branch and tried with Branch_5b5c77ed. (#968) and I get an error on launch:
File "...\Python39\lib\selectors.py", line 315, in _select r, w, x = select.select(r, w, w, timeout)
OSError> [WinError 10022] Invalid argument
... called from proxy/core/acceptor/threadless.py:279