Iroh aborts under high load

[rklaehn]

@ppodolsky on discord:

---

Well... I have left Iroh for the last night and it has been supposed to do intense inserts into docs, but unfortunately I have noted right now that after the last code update Iroh has started to restart sometimes. Looks like process being aborted, no any useful log lines and signs of disruption before abortion. And Iroh has restarted for 4 times in 8 hour period, I have never met these behaviour before latest commits.

I will try to watch Iroh today, but no idea where should I look at.
btw, there is also an intense spam 2024-06-28T04:28:32.076983Z INFO poll_send:get_send_addrs: new connection type typ=mixed me=yyy node=xxx, approximately every second. Mentioned node xxx is always the same

---

Possibly related to the changes we have made to tokio task handling. I have improved error logging in #2423 already.

[rklaehn]

This branch adds some brute force println! to track down why the main loop aborts: https://github.com/n0-computer/iroh/tree/pasha-debug

[rklaehn]

@pashapodolsky we just had a discussion about panic (weirdly in storage).

I noticed that we only set panic=abort in release-optimized mode. But I think that is not relevant for you since you embed iroh. Are you building with panic=abort?

Do you have RUST_BACKTRACE=1 set when you are running your thing?