Error starting collator nodes on macbook M4

[serban300]

Is there an existing issue?

• I have searched the existing issues

Experiencing problems? Have you tried our Stack Exchange first?

• This is not a support question.

Description of bug

Issue started appearing after commit f6d18b7

It happens locally on an macbook with M4 chip. Doesn't seem to be reproducible in the CI.

When starting a zombienet locally, some of the collator nodes fail with the following error:

====================

Version: 1.18.5-1572205feae

   0: backtrace::backtrace::trace
   1: backtrace::capture::Backtrace::new
   2: sp_panic_handler::set::{{closure}}
   3: std::panicking::rust_panic_with_hook
   4: std::panicking::begin_panic_handler::{{closure}}
   5: std::sys::backtrace::__rust_end_short_backtrace
   6: _rust_begin_unwind
   7: core::panicking::panic_fmt
   8: core::panicking::panic
   9: <core::future::poll_fn::PollFn<F> as core::future::future::Future>::poll
  10: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::future::future::Future>::poll
  11: <futures_util::future::future::map::Map<Fut,F> as core::future::future::Future>::poll
  12: <sc_service::task_manager::prometheus_future::PrometheusFuture<T> as core::future::future::Future>::poll
  13: <tracing_futures::Instrumented<T> as core::future::future::Future>::poll
  14: tokio::runtime::task::core::Core<T,S>::poll
  15: tokio::runtime::task::harness::Harness<T,S>::poll
  16: tokio::runtime::scheduler::multi_thread::worker::Context::run_task
  17: tokio::runtime::scheduler::multi_thread::worker::Context::run
  18: tokio::runtime::context::scoped::Scoped<T>::set
  19: tokio::runtime::context::runtime::enter_runtime
  20: tokio::runtime::scheduler::multi_thread::worker::run
  21: tokio::runtime::task::core::Core<T,S>::poll
  22: tokio::runtime::task::harness::Harness<T,S>::poll
  23: tokio::runtime::blocking::pool::Inner::run
  24: std::sys::backtrace::__rust_begin_short_backtrace
  25: core::ops::function::FnOnce::call_once{{vtable.shim}}
  26: std::sys::pal::unix::thread::Thread::new::thread_start
  27: __pthread_cond_wait


Thread 'tokio-runtime-worker' panicked at 'SelectNextSome polled after terminated', /Users/serban-work/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.31/src/stream/stream/select_next_some.rs:32

This is a bug. Please report it at:

	https://github.com/paritytech/polkadot-sdk/issues/new

Steps to reproduce

No response

[serban300]

Example zombienet:

[relaychain]
default_command = "/Users/serban-work/workplace/sources/polkadot-sdk/target/release/polkadot"
chain = "rococo-local"

[[relaychain.nodes]]
name = "alice"
validator = true
rpc_port = 9000

[[relaychain.nodes]]
name = "bob"
validator = true

[[relaychain.nodes]]
name = "charlie"
validator = true

[[relaychain.nodes]]
name = "dave"
validator = true

[[parachains]]
id = 2000
chain = "asset-hub-rococo-local"
cumulus_based = true

	# run charlie as parachain collator
	[[parachains.collators]]
	name = "alice"
	validator = true
	rpc_port = 10000
	command = "/Users/serban-work/workplace/sources/polkadot-sdk/target/release/polkadot-parachain"
	args = ["--force-authoring"]

[[parachains]]
id = 2001
chain = "bridge-hub-rococo-local"
cumulus_based = true

	[[parachains.collators]]
	name = "bob"
	validator = true
	rpc_port = 10001
	command = "/Users/serban-work/workplace/sources/polkadot-sdk/target/release/polkadot-parachain"
	args = ["--force-authoring"]

[[parachains]]
id = 2002
chain = "bridge-hub-rococo-local"
cumulus_based = true

	[[parachains.collators]]
	name = "charlie"
	validator = true
	rpc_port = 10002
	command = "/Users/serban-work/workplace/sources/polkadot-sdk/target/release/polkadot-parachain"
	args = ["--force-authoring"]

[serban300]

The error is triggered here in the tokio-rs/mio library.

pub const AF_INET: c_int = 2;
pub const AF_INET6: c_int = 30;

But on my machine, sometimes it's receiving 0 or 64 or 128.

pub const AF_UNSPEC: c_int = 0;

But 64 and 128 don't to correspont to any AF_ definition.

[serban300]

I realized the problem was reproducing only for IPv6 addresses.

On a more detailed search looks like macOS generally supports IPv6, but its implementation might not be as flexible as some other operating systems when it comes to wildcard usage in certain contexts.

A possible fix would be to use here instead of wildcard. Opened a PR that would do this: #9061 . But I'm not super familiar with the networking logic and if this would be acceptable. @lexnv can you take a look please ?

[lexnv]

Thanks for looking into this @serban300! 🙏

Indeed, macOS might not be as widely adopted as other platforms. I think it would be better to fix this upstream, instead of switching the nodes to point to 127.0.0.1. We previously were listening by default on "any IPs" (0.0.0.0). It might become problematic, as in #9061 we'll start leasing only on the loopback address 🤔

Do we know what needs to be patched upstream to make this work seamlessly on macOS as well? 🤔

[serban300]

I didn't look into it, but I don't think this can be patched upstream. I think macOS simply doesn't support 0.0.0.0 fully for IPv6

[lexnv]

Could it be possible that the sockaddr_storage points to uninitialized / partially initialized data?

• https://github.com/tokio-rs/mio/blob/23dfe4224feb0aee8a821b5737e1d413c8cd8de9/src/sys/unix/net.rs#L191-L194

Would be interesting if we could dump the bytes of the whole storage: *const libc::sockaddr_storage on error, so we could better try to understand where the 128 value comes from 🤔 Considering that we get inconsistent data here, it might be the case that on MacOS an initialization is missed for that struct

[lexnv]

Thinking outloud, maybe this will help?

    let mut addr: MaybeUninit<libc::sockaddr_storage> = unsafe { MaybeUninit::zeroed() };

[serban300]

Could it be possible that the sockaddr_storage points to uninitialized / partially initialized data?

Yes, looks like sockaddr_storage points to uninitialized data.

Thinking outloud, maybe this will help?

let mut addr: MaybeUninit<libc::sockaddr_storage> = unsafe { MaybeUninit::zeroed() };

With this change, storage is always zeroed when the crash occurs. Probably 64 and 128 were garbage values.

(lldb) p *storage
(libc::unix::bsd::apple::sockaddr_storage) {
  ss_len = 0
  ss_family = 0
  __ss_pad1 = ([0] = 0, [1] = 0, [2] = 0, [3] = 0, [4] = 0, [5] = 0)
  __ss_align = 0
  __ss_pad2 = {
    [0] = 0
    [1] = 0
    [2] = 0
    [3] = 0
    [4] = 0
    [5] = 0
    [6] = 0
    [7] = 0
    [8] = 0
    [9] = 0
    [10] = 0
    [11] = 0
    [12] = 0
    [13] = 0
    [14] = 0
    [15] = 0
    [16] = 0
    [17] = 0
    [18] = 0
    [19] = 0
    [20] = 0
    [21] = 0
    [22] = 0
    [23] = 0
    [24] = 0
    [25] = 0
    [26] = 0
    [27] = 0
    [28] = 0
    [29] = 0
    [30] = 0
    [31] = 0
    [32] = 0
    [33] = 0
    [34] = 0
    [35] = 0
    [36] = 0
    [37] = 0
    [38] = 0
    [39] = 0
    [40] = 0
    [41] = 0
    [42] = 0
    [43] = 0
    [44] = 0
    [45] = 0
    [46] = 0
    [47] = 0
    [48] = 0
    [49] = 0
    [50] = 0
    [51] = 0
    [52] = 0
    [53] = 0
    [54] = 0
    [55] = 0
    [56] = 0
    [57] = 0
    [58] = 0
    [59] = 0
    [60] = 0
    [61] = 0
    [62] = 0
    [63] = 0
    [64] = 0
    [65] = 0
    [66] = 0
    [67] = 0
    [68] = 0
    [69] = 0
    [70] = 0
    [71] = 0
    [72] = 0
    [73] = 0
    [74] = 0
    [75] = 0
    [76] = 0
    [77] = 0
    [78] = 0
    [79] = 0
    [80] = 0
    [81] = 0
    [82] = 0
    [83] = 0
    [84] = 0
    [85] = 0
    [86] = 0
    [87] = 0
    [88] = 0
    [89] = 0
    [90] = 0
    [91] = 0
    [92] = 0
    [93] = 0
    [94] = 0
    [95] = 0
    [96] = 0
    [97] = 0
    [98] = 0
    [99] = 0
    [100] = 0
    [101] = 0
    [102] = 0
    [103] = 0
    [104] = 0
    [105] = 0
    [106] = 0
    [107] = 0
    [108] = 0
    [109] = 0
    [110] = 0
    [111] = 0
  }
}

Considering that we get inconsistent data here, it might be the case that on MacOS an initialization is missed for that struct

Would be good if this is the issue and if it could be fixed with some additional initialization. Researching. But I can't find much documentation on this.

Also accept doesn't return an error (-1). It returns an actual file descriptor.

[serban300]

Actually maybe this is not related to IPv6. The crash doesn't happen if I leave only the IPv6 listen_address. It doesn't happen if I leave only the IPv4 listen_address. It happens only when both are present.

[lexnv]

Indeed, the original hunch turned out to be correct, thanks for looking into this 🙏

I'm quite sure we need to zero out the structure on MacOS, so this is a clear bug in mio:

• nanomsg/nanomsg@5ac0c04

Since the crash happens when both addresses are enabled, it might be due to the dual stack of MacOS, the so called "IPv4 mapped to IPv6 address". Taken from https://docs.freebsd.org/en/books/developers-handbook/ipv6/index.html#_unified_tcp_and_inpcb_code:

 The platform can be configured to support IPv4 mapped address. Kernel configuration is summarized as follows:

    By default, AF_INET6 socket will grab IPv4 connections in certain condition, and can initiate connection to IPv4 destination embedded in IPv4 mapped IPv6 address.

    You can disable it on entire system with sysctl like below.

    sysctl net.inet6.ip6.mapped_addr=0

Could you give sysctl net.inet6.ip6.mapped_addr=0 a try?

Another thing we could print is the length, before and after the accept call:
https://github.com/tokio-rs/mio/blob/23dfe4224feb0aee8a821b5737e1d413c8cd8de9/src/sys/unix/tcp.rs#L112

• It might happen for whatever reason that our sockaddr_storage cannot be filled by the kernel because it doesn't have enough bytes 🤔 but less likely