error reusing channel for running parallel client calling the same server in `tokio::test`

[tdelabro]

Bug Report

Version

tonic = "0.12.3"
tonic-health = "0.12.3"

Platform

Darwin MacBook-Pro-64.local 21.6.0 Darwin Kernel Version 21.6.0: Wed Aug 10 14:28:23 PDT 2022; root:xnu-8020.141.5~2/RELEASE_ARM64_T6000 arm64

Description

I'm trying to run integration tests that will query the health check of a server already running.

lib.rs:

pub struct ThreadSafeSigner(OnceCell<Channel>);

#[fixture]
#[once]
pub fn signer() -> ThreadSafeSigner {
    ThreadSafeSigner(OnceCell::default())
}

impl ThreadSafeSigner {
    async fn get_signer_channel(&self) -> Result<Channel> {
        let signer_port = std::env::var("SOCKET_PORT")?;

        let channel = self
            .0
            .get_or_try_init::<anyhow::Error, _, _>(|| async {
                let address = format!("http://localhost:{}", signer_port);

                let timeout = Instant::now() + Duration::from_secs(3);
                let channel = loop {
                    if let Ok(c) = tonic::transport::Channel::builder(address.parse()?)
                        .connect()
                        .await
                    {
                        break c;
                    }
                    if Instant::now() > timeout {
                        return Err(anyhow!("timeout waiting for signer"));
                    }
                };

                Ok(channel)
            })
            .await?;

        Ok(channel.clone())
    }

    pub async fn init_health_client(&self) -> Result<HealthClient<tonic::transport::Channel>> {
        let channel = self.get_signer_channel().await?;
        let client = tonic_health::pb::health_client::HealthClient::new(channel);

        Ok(client)
    }
}

test.rs

#[rstest]
#[tokio::test]
async fn is_healthy0(signer: &ThreadSafeSigner) -> Result<()> {
    let mut client = signer.init_health_client().await?;
    let res = client
        .check(HealthCheckRequest {
            service: "signer.Signer".to_string(),
        })
        .await?;
    let serving_status = ServingStatus::try_from(res.into_inner().status)?;

    if serving_status == ServingStatus::Serving {
        Ok(())
    } else {
        Err(anyhow!(
            "invalid status, expected SERVING, got {}",
            serving_status.as_str_name()
        ))
    }
}

...

<a bunch of other similar test `is_healthy1, 2, 3, 4, etc` all calling `init_health_client` > 

I do this because the doc recommends cloning the channel and reuse it.

I get the following error:

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

     Running declare_keyset.rs (target/debug/deps/declare_keyset-aa3fe48cf645a906)

running 10 tests
test is_healthy2 ... ok
test is_healthy1 ... ok
test is_healthy6 ... ok
test is_healthy0 ... ok
test is_healthyc5 ... ok
test is_healthy7 ... ok
test is_healthy8 ... ok
Error: status: Unknown, message: "transport error", details: [], metadata: MetadataMap { headers: {} }

Caused by:
    0: transport error
    1: dispatch task is gone
    2: runtime dropped the dispatch task
Error: status: Unknown, message: "transport error"test is_healthy4 ... FAILED
, details: [], metadata: MetadataMap { headers: {} }

Caused by:
    0: transport error
    1: dispatch task is gone
    2: runtime dropped the dispatch task
test is_healthy6n ... ok
test is_healthy3 ... FAILED

failures:

failures:
    is_healthy3
    is_healthy4

test result: FAILED. 8 passed; 2 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.01s

error: test failed, to rerun pass `-p signer-tests --test declare_keyset`

Sometime it goes through, sometime 1 or 2 fail, sometimes all.

What am I doing incorrectly?