High memory/CPU utilization for moderately sized cluster

[jacksontj]

I have a cluster of ~300 nodes and mesh is consuming ~1-2 GB of RAM. I dug into it and the memory is all being consumed by the topology gossip messages. Upon further inspection I found that the gossip messages are including all peers in the message -- which means the message sizes (and therefore the memory and CPU to generate them) scale with the cluster size.

Are there any plans to implement a more scalable topology gossip?

[bboreham]

The idea is that gossip messages are not sent very often - Weave Mesh selects log(number of connections) peers to send to - so the scalability should be good.

However various other issues in the code mean that messages are sent way more often than this ideal. #101, #106 and #107 are attempts to improve matters, though work is ongoing to understand the full set of causes.

[jacksontj]

From my use-case of protokube (kubernetes/kops#7427) I'm seeing ~2 cores of CPU usage and ~3G of RAM usage with a fully connected mesh of ~300 nodes. This seems to highlight some serious scale limitations of weaveworks/mesh -- as that isn't even a very large cluster. More importantly the utilization ramp-up was more-or-less exponential as more nodes were added.

Even after I made a custom build with #107 fixed the CPU usage dropped to 1.6 cores -- which is still way too many (all the CPU time was being spent marshaling/unmarshaling the peer list being gossiped around).

There seem to be quite a few issues, a couple: (1) no concept of "suspect" state (2) peer messages include the list of all peers it has connected-- which scaled with cluster size. There are likely more but TBH I have decided to instead spend my time swapping protokube to a more robust/reliable gossip library.

[bboreham]

We're seeing quite positive results from deferring gossip updates - #117 and #118.
Would you like to try those in your build?

[bboreham]

peer messages include the list of all peers it has connected

It's worse than that - the topology message lists all the connections of all peers. In other words in a fully-connected cluster it's O(N^2).

However, for 300 nodes that might be 8MB per message, which needs something else to get to 1-2GB.

We found that the connections would each read a message then block on the Peers lock to apply the update. So with 200 connections that's 1.6GB.

Changing the "everyone sends everything" behaviour is quite a big change, so ahead of that I felt that just slowing down the initial connections would help - #124 . After initial connection the updates only go to logN peers so we shouldn't get the massive spikes.

[bboreham]

I'm going to close this issue now 0.4 is released - if you want to come back to the discussion please do.

[cgicgi]

Hi there,
this comment is meant more for documentation purposes in case anybody might see similar issues.

We're running an AWS kubernetes cluster with kops for several years now, that heavily relies on autoscaling. Base load runs on ~ 50 nodes, scales up and down some dozen nodes daily. Recently we experienced an enourmous spike in internal network traffic along with similar effects as described by @jacksontj . protokube running on 160% CPU load. Pods getting killed due to "high disk pressure". Pods not starting quickly enough to get ready.
Situation escalated to 25TB of daily network traffic (which costs some serious money) before we were able to mitigate it.

In the logs we found tons of suspicious "->[172.20.51.167:3999] error during connection attempt: dial tcp 0.0.0.0:0->172.20.51.167:3999: connect: no route to host" from connection_maker.go:376, where the IP of the node didn't exist any more (we don't know how long, but most likely more than 7 days).

So our assessment is as follows: For some reason, inactive nodes were not deleted from mesh's collective memory. At some point, this reached a tipping point where communication between the nodes blocked the network. Restarting protokube on single nodes obviously did not help, as the knowledge about the inactive nodes was instantly shared and kept alive by other instances. Shutting down all protokube instances on all nodes might have been a solution, but we did not come up with that in time.

Unfortunately, we don't have a full picture yet, as the cluster became unstable and we decided to replace it before spending even more money on traffic costs. We have set up a lot of new alerts to detect such a situation earlier and being able to gather better debug data for further analysis.