Communication problems (GSO/IO) between lite-rpc and quic-forward-proxy on IPv6 private network #199
Description
When deploying lite-rpc and quic-forward-proxy to Fly on separate VMs, we noticed submitted TXs timing out.
After some debugging, it seems this call to write_all on the SendStream can produce this error:
sendmsg error: Os { code: 22, kind: InvalidInput, message: "Invalid argument" }, Transmit: { destination: [fdaa:0:a53d:a7b:17b:3cd9:fe0f:2]:11111, src_ip: None, enc: Some(Ect0), len: 1291, segment_size: Some(1200) }
After which the future never returns, blocking any further messages from the broadcast_receiver being processed. After restarting lite-rpc maybe one or two batches are successfully forwarded before this error occurs and no more batches are forwarded.
Adding a timeout to this future allows sending to continue, but produces the following error logs in quic-forward-proxy:
ERROR handle_client_connection: solana_lite_rpc_quic_forward_proxy::inbound::proxy_listener: failed to accept stream: timed out
thread 'tokio-runtime-worker' panicked at 'called `Result::unwrap()` on an `Err` value: Read(ConnectionLost(TimedOut))', quic-forward-proxy/src/inbound/proxy_listener.rs:106:85
ERROR solana_lite_rpc_quic_forward_proxy::inbound::proxy_listener: failed handling connection from client: error accepting stream - skip
Tests on IPv4 and local IPv6 outside of the Fly network have been successful, pointing to an incompatibility with the Fly IPv6 private network, or kernel network parameters on the VMs.