[BUG] Mainnet nodes with incoming connections unexpected shutdown with Failure in Data.Map.balanceR (error, called at src/Data/Map/Internal.hs:4157:30 in containers-0.6.5.1-EiES0HFUZ8PBGNrpVjoYRF:Data.Map.Internal)

[SmaugPool]

External
External

Area
cardano-node exception leading to shutdown

Summary
All my public nodes with incoming connections (active block producer and relays) shut down unexpectedly on 2022 January 22 at 00:09:01 UTC following the following fatal exception:

janv. 22 01:09:01 de cardano-node-1.35.4[24590]: {"app":[],"at":"2023-01-22T00:09:01.24Z","data":{"immtip":{"headerHash":"fae2d44113d1fb8b99171cc64983ff5dfb92f1d360f2460e6>
janv. 22 01:09:01 de cardano-node-1.35.4[24590]: cardano-node-1.35.4: ExceptionInLinkedThread "ThreadId 165" Failure in Data.Map.balanceR
janv. 22 01:09:01 de cardano-node-1.35.4[24590]: CallStack (from HasCallStack):
janv. 22 01:09:01 de cardano-node-1.35.4[24590]:   error, called at src/Data/Map/Internal.hs:4157:30 in containers-0.6.5.1-EiES0HFUZ8PBGNrpVjoYRF:Data.Map.Internal

(time in UTC+1 above)

Some other operators confirmed this also happened to some of their nodes.

Steps to reproduce
No idea yet.

Expected behavior
No exception is expected in cardano-node.

System info (please complete the following information):

• OS Name: Ubuntu
• OS Version 20.04
• Node version

cardano-node 1.35.4 - linux-x86_64 - ghc-8.10
git rev ebc7be471b30e5931b35f9bbc236d21c375b91bb

• CLI version

cardano-cli 1.35.4 - linux-x86_64 - ghc-8.10
git rev ebc7be471b30e5931b35f9bbc236d21c375b91bb

Screenshots and attachments
Previous logs (UTC+1):

janv. 22 01:09:01 de cardano-node-1.35.4[24590]: {"app":[],"at":"2023-01-22T00:09:01.22Z","data":{"block":"77e81e9305859032fea6a8e79f5936e96723a1620608d2bd0100c2421f5ef667@82779850","kind":"TraceAddBlockEvent.AddBlockValidation.ValidCandidate"},"env":"1.35.4:ebc7b","host":"de","loc":null,"msg":"","ns":["cardano.node.ChainDB"],"pid":"24590","sev":"Info","thread":"157"}
janv. 22 01:09:01 de cardano-node-1.35.4[24590]: {"app":[],"at":"2023-01-22T00:09:01.22Z","data":{"chainLengthDelta":1,"kind":"TraceAddBlockEvent.AddedToCurrentChain","newtip":"77e81e9305859032fea6a8e79f5936e96723a1620608d2bd0100c2421f5ef667@82779850"},"env":"1.35.4:ebc7b","host":"de","loc":null,"msg":"","ns":["cardano.node.ChainDB"],"pid":"24590","sev":"Notice","thread":"157"}
janv. 22 01:09:01 de cardano-node-1.35.4[24590]: Shutting down..

Additional context
All my nodes without incoming connections (just outgoing ones) were not affected.

[rdlrt]

Many SPOs are reporting the same on SPO workgroup (node versions across 1.35.3 and 1.35.4). Only nodes that were not publicly available to connect (thus, not receiving all transactions) survived

[AndrewWestberg]

7-minute gap in my logs.

{"app":[],"at":"2023-01-22T00:09:01.59Z","data":{"kind":"DiffusionErrored","path":"ExceptionInLinkedThread \"ThreadId 214\" Failure in Data.Map.balanceR\nCallStack (from HasCallStack):\n  error, called at libraries/containers/containers/src/Data/Map/Internal.hs:4157:30 in containers-0.6.5.1:Data.Map.Internal"},"env":"1.35.4:8bf17","host":"handy","loc":null,"msg":"","ns":["cardano.node.DiffusionInitializationTracer"],"pid":"1670822","sev":"Info","thread":"16"}
{"app":[],"at":"2023-01-22T00:09:01.59Z","data":{"kind":"TraceImmutableDBEvent.DBClosed"},"env":"1.35.4:8bf17","host":"handy","loc":null,"msg":"","ns":["cardano.node.ChainDB"],"pid":"1670822","sev":"Info","thread":"16"}
{"app":[],"at":"2023-01-22T00:09:01.59Z","data":{"immtip":{"headerHash":"fae2d44113d1fb8b99171cc64983ff5dfb92f1d360f2460e69305ccaf279ceaf","kind":"BlockPoint","slot":82736539},"kind":"TraceOpenEvent.ClosedDB","tip":{"headerHash":"77e81e9305859032fea6a8e79f5936e96723a1620608d2bd0100c2421f5ef667","kind":"BlockPoint","slot":82779850}},"env":"1.35.4:8bf17","host":"handy","loc":null,"msg":"","ns":["cardano.node.ChainDB"],"pid":"1670822","sev":"Info","thread":"16"}
{"app":[],"at":"2023-01-22T00:09:08.41Z","data":{"kind":"LogMessage","message":"Byron; Shelley"},"env":"1.35.4:8bf17","host":"handy","loc":null,"msg":"","ns":["cardano.node.basicInfo.protocol"],"pid":"2149443","sev":"Notice","thread":"16"}

[SmaugPool]

Network synchronization and block height alignment from pooltool.io (https://pooltool.io/networkhealth):

[renesecur]

We saw the same. Last block to come in was 8300568. As it came in, the nodes crashed.

[nalyd88]

All our active nodes (mix of cloud and baremetal running both versions 1.35.3 and 1.35.4) went down. Only one node that had no incoming connections did not have the error.

Looking for the error message we found its from here so not a node exception.
https://hackage.haskell.org/package/containers-0.6.2.1/docs/src/Data.Map.Internal.html

[lead-pool]

Same error too - all nodes restarted...
All running 1.35.4 - combination of Bare Metal and Cloud running across multiple geographical locations

[lupsya]

Can confirm the same, all nodes restarted at the same time that was reported here.
Using both versions 1.35.3 & 1.35.4. Ubuntu 20.04

[WHITECardano]

Same here

[CryptoBlocks-pro]

Ditto here