Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Proxy tests flaky because of uncoordinated environment access #2468

Open
decathorpe opened this issue Oct 31, 2024 · 3 comments
Open

Proxy tests flaky because of uncoordinated environment access #2468

decathorpe opened this issue Oct 31, 2024 · 3 comments
Labels
E-easy Effort: Easy! Start here :D E-pr-welcome The feature is welcome to be added, instruction should be found in the issue.

Comments

@decathorpe
Copy link

I'm working on updating the package for reqwest on Fedora Linux to the latest version. I noticed new test failures in 0.12.9 compared to 0.12.8 in the proxy tests, and I see there were some proxy-related changes in 0.12.9, so I thought it worth reporting.

I am seeing failures in ~50% of test runs on aarch64-unknown-linux-gnu, and only on that architecture. I'm not sure why that might be the case, but our aarch64 builders are the ones that have the most multi-threading capability. The failed builds all had 12 CPU threads available, with tests running in parallel (-j12). We are running the equivalent of cargo test --release, if that matters.

I saw failures with various different Rust compiler versions (1.82, 1.81, 1.79, and 1.75) and on different OS versions (Fedora 42, 41, 40, 39, and RHEL 9 and CentOS Stream 10) so I don't think either of these play a role here.

All bad test runs I've seen were caused by one of these two failures:

failures:
---- system_http_proxy_basic_auth_parsed stdout ----
thread 'system_http_proxy_basic_auth_parsed' panicked at tests/proxy.rs:128:10:
called `Result::unwrap()` on an `Err` value: reqwest::Error { kind: Request, url: "http://hyper.rs/prox", source: hyper_util::client::legacy::Error(SendRequest, hyper::Error(IncompleteMessage)) }
failures:
---- test_using_system_proxy stdout ----
thread 'test_using_system_proxy' panicked at tests/proxy.rs:183:39:
called `Result::unwrap()` on an `Err` value: reqwest::Error { kind: Request, url: "http://not.a.real.sub.hyper.rs/prox", source: hyper_util::client::legacy::Error(SendRequest, hyper::Error(IncompleteMessage)) }
@seanmonstar
Copy link
Owner

Looks like those manipulate environment variables, so if they're run concurrently, they could clobber each other. Do you see the issues if the tests run with just 1 thread?

With v0.12.9, we removed an internal cache which made reqwest only read the environment variables once. The tests used to disable that progammatically, and also use some locks to prevent clobbering. I think those locks were lost in the fix, and probably need to be added back so concurrent tests can be run.

@decathorpe
Copy link
Author

That would indeed match what I'm seeing - I will try running tests with --test-threads=1 and report back.

@decathorpe
Copy link
Author

Sorry for the delay - I've tried running the tests single-threaded and didn't see a single failure, so I assume that your guess about clobbered environment variables was correct :) For now I'll adapt our package to always run with --test-threads=1.

@seanmonstar seanmonstar changed the title new flaky failures in proxy tests since reqwest 0.12.9 Proxy tests flaky because of uncoordinated environment access Nov 4, 2024
@seanmonstar seanmonstar added E-easy Effort: Easy! Start here :D E-pr-welcome The feature is welcome to be added, instruction should be found in the issue. labels Nov 4, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
E-easy Effort: Easy! Start here :D E-pr-welcome The feature is welcome to be added, instruction should be found in the issue.
Projects
None yet
Development

No branches or pull requests

2 participants