copy_bidirectional not return

[b23r0]

Version
1.17.0

Platform
Linux iZbp1cerzh0yyvcuxg4opsZ 5.4.0-77-generic #86-Ubuntu SMP Thu Jun 17 02:35:03 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

Description
I use copy_bidirectional() to copy between two tcpstreams, but after running online for a period of time, I found that a large number of TCP connections from clients are in the CLOSE_WAIT state. Through observation, it is found that some clients call in asynchronous threads after closing the connection. The copy_bidirectional() still doesn't return.

similar code:

use tokio::runtime::{Builder , Runtime};
use tokio::{self , io};
use tokio::net::{TcpStream, TcpListener};

async fn test_task(rt : &Runtime){

    let main_socket = TcpListener::bind("127.0.0.1:8000").await.unwrap();

    loop{
        let (mut s , _) = main_socket.accept().await.unwrap();
        let mut c = TcpStream::connect("127.0.0.1:8001").await.unwrap();
        
        rt.spawn(async move {
            match io::copy_bidirectional(&mut c, &mut s).await{
                Ok(_) => {},
                Err(_) => {}
            };
            println!("some task cant get here");
        });
    }
}

fn main() {

    std::thread::spawn( || {
        let rt = Builder::new_current_thread().build().unwrap();
        rt.block_on(test_task(&rt));
    }).join().unwrap();
}

[b23r0]

i dont know why sometime return but sometime not return. If this method cannot be used during the use of TcpStream as a parameter, make sure to return every time it is closed. Well I hope it's noted in the documentation, as there are tons of TCP network proxies that misuse this method. Or can you provide a new method to provide copy bidirectional function specifically for TcpStream. I think this is very useful.

[Darksonn]

It should work with tcp streams, however it wont return until the stream is closed in both directions. Perhaps the stream was only closed in one of the two directions?

[b23r0]

It should work with tcp streams, however it wont return until the stream is closed in both directions. Perhaps the stream was only closed in one of the two directions?

im sure what Ive seen in the real environment is that both sides are closed, but it still doesn't return.

[Noah-Kennedy]

@b23r0 can you try and reproduce this?

[b23r0]

@b23r0 can you try and reproduce this?

This is a problem found in the production environment. At that time, the client had about 30 to 40 concurrent connections. I could not reproduce this problem through a single or multiple normal connection and close connection tests in the development environment. This may be an issue that requires a race condition to trigger.

I need to try to simulate a large number of concurrent connections to test and reproduce the problem, but the current work may delay this plan. I may have time to do more work in about a week. In the meantime maybe you can do more testing.

[b23r0]

Currently I use select! instead of copy_bidirectional and the problem doesn't happen again.

...
        rt.spawn( async move {
            let mut buf1 = [0u8;1024];
            let mut buf2 = [0u8;1024];
            loop{
                select!{
                    a = s.read(&mut buf1) => {
                        let size = match a{
                            Ok(p) => p,
                            Err(_) => break,
                        };

                        if size == 0 {
                            break;
                        }

                        match c.write_all(&buf1[..size]).await{
                            Ok(_) => {},
                            Err(_) => break,
                        };
                    },
                    b = c.read(&mut buf2) => {
                        let size = match b{
                            Ok(p) => p,
                            Err(_) => break,
                        };

                        if size == 0 {
                            break;
                        }

                        match s.write_all(&buf2[..size]).await{
                            Ok(_) => {},
                            Err(_) => break,
                        };
                    }
                }
            }
        });
...

[Darksonn]

Okay, so the difference between that and copy_bidirectional is that you stop both directions once one of them has returned a read of size zero, whereas copy_bidirectional will continue until both directions have returned a read of size zero.

What I suspect that you're seeing is that one of the two sockets has been closed in both directions, but the other socket has not closed its write direction and isn't writing anything, so copy_bidirectional is stuck reading from that other socket, and it wont figure out that the target is closed until it actually writes to the target, which it wont because it doesn't have anything to write.

[yinheli]

@b23r0 I got same issue here, any updates?

[yinheli]

In my case there is many FIN-WAIT-2 state (with pid, not orphaned) socket is shutdown(), but not close()

https://github.com/tokio-rs/tokio/blob/master/tokio/src/net/tcp/stream.rs#L1306

Only close write, how can we shutdown both?

[yinheli]

use crate tokio-io-timeout seems problem solved.