Local cluster test failure

[amrali]

Following the Readme instructions, specifically for running tests cargo test --all results in the following test failure.

failures:

---- local_cluster::test::test_local_cluster_start_and_exit_with_config stdout ----
thread 'local_cluster::test::test_local_cluster_start_and_exit_with_config' panicked at 'called `Result::unwrap()` on an `Err` value: Custom { kind: Other, error: StringError("Unable to get recent blockhash, too many retries") }', src/libcore/result.rs:997:5


failures:
    local_cluster::test::test_local_cluster_start_and_exit_with_config

test result: FAILED. 227 passed; 1 failed; 2 ignored; 0 measured; 0 filtered out

Running the same test in isolation cargo test test_local_cluster_start_and_exit_with_config passes.

I didn't notice that test pass when running the entire test suit a few times, so it should be reproducible.

• macOS 10.14.2
• Rust 1.33
• against master's HEAD

I think If this is a second order failure (i.e., a failure caused by another hidden failure) the test should fail on the root error instead.

[sakridge]

@amrali what is your ulimit? ulimit -n should give 65k

[sakridge]

Also make sure you're running cargo build --all

[amrali]

my FD limit is 7168, I noticed there are some stress/benchmark like tests in the test suite, is it correct to assume that cargo test --all doesn't include bench tests; to require that many FDs?

Anyway, I noticed that during tests my firewall alerted me to applications that needs to listen publicly for incoming connections. Once I turned off my firewall the tests pass.

Are there tests trying to listen on 0.0.0.0?

[sakridge]

Ah, yes you need to either disable the macos firewall or be very quick to click 'allow' when it asks which is unfortunate.

[sakridge]

my FD limit is 7168, I noticed there are some stress/benchmark like tests in the test suite, is it correct to assume that cargo test --all doesn't include bench tests; to require that many FDs?

7k may be ok, but cargo test --all does run many tests in parallel which opens many services/threads so we do see that we need to up the ulimit for running these tests in some cases.

[mvines]

source scripts/ulimit-n.sh to get the recommended ulimit -n value (used by our CI)

[amrali]

Thanks, that's helpful. I think this issue as titled is resolved. I can add a general note that developers need to check/disable their firewall before running tests to the README if that's the right place to put this.

[mvines]

Yes please. The book, around here would be perfect for this info

[amrali]

Will open a PR and edit there. Thanks.