High CPU usage after a few hours

[LLeny]

Background

High CPU usage since 0.7.1.
The node runs perfectly for a few hours and at one point the CPU usage of lnd reaches 100%, I've let it for a couple of days at 100% but it never goes down. Restarting lnd fixes the issue for a few hours. This behavior wasn't seen with 0.6.
bitcoind CPU usage stays low.

Your environment

• version of lnd
  lnd version 0.7.1-beta commit=v0.7.1-beta
• which operating system (uname -a on *Nix)
  Linux 4.19.0-5-amd64 Fix name typo in README #1 SMP Debian 4.19.37-5 (2019-06-19) x86_64 GNU/Linux
• version of btcd, bitcoind, or other backend
  Bitcoin Core Daemon version v0.18.0
• any other relevant environment details

CPU profile when it happens :

[halseth]

Thanks for the report!

cc @wpaulino

[wpaulino]

Do the logs show anything relevant while this is happening?

[LLeny]

I couldn't see anything unusual in the logs, the node appears to still be working as expected, peers are connected, channels are active.

[wpaulino]

Are you experiencing this reliably? Do you see bitcoind falling behind at any point and then catching up?

[LLeny]

Got it after a few day this time.
Bitcoind blocks are up to date, lnd is also in sync with bitcoind.
Bitcoind is not taking any CPU.
I can keep it in this state for a few days if you think about something to look for.

[wpaulino]

Could you also obtain a heap profile while experiencing the issue?

[LLeny]

[cfromknecht]

@LLeny curiously, what kind of device is this running on?

i'd guess by your description that you have a peer that requests a lot of graph data, which can a lot of allocations/deserializations as the objects are created and destroyed and gc pressure. we do keep an in-memory cache of these responses in order to avoid deserializing them every time a peer requests them, but depending on your peers or hardware, it may not be enough.

if you're machine has enough ram, you can consider increasing the caches.channel-cache-size to 35000, rather than the default of 20000. you'll notice higher overall memory usage, but less cpu (gc + deserialization).

either way you will probably want to identify which peer(s) is causing this. the logs should show something like:

... applying new update horizon: start=2019-08-04 23:38:09 +0000 UTC, end=2155-09-11 06:06:24 +0000 UTC, backlog_size=92088

thing to look for is peers that request a large backlog_size, e.g. 50k+. if you don't have a channel with them you can try banning their IP address. if you do, but the channel isn't sentimental, you can try making a channel with someone else.

this recent pr #3359 offers a new configuration option called ignore-historical-filters which will be available in 0.8. the feature is designed to help less beefy machines avoid the heavier resource costs of syncing other peers on the network. if you're comfortable, another possible solution is to run master and use ignore-historical-filters.

as noted in the PR, there are plans to remove this behavior entirely as it is quite heavy, but there are currently nodes on the network that depend on this for synchronizing the graph. you can follow the progress of its removal in this spec PR

[LLeny]

@cfromknecht thank you, lnd has 1 core of a x86 @ 2.4GHz, 2GB of RAM.
I indeed found a few update horizon backlog_size between 100k and 120k, but I couldn't identify a clear correlation with the constant CPU usage. When it happens it stays at 100% for days until the next restart.
I'll check in more depth during the next occurrence.

[Talkless]

I've noticed cpu spikes too since latest beta release:

At the same time, bitcoind (0.18.1) memory starts growing, so I assume it's due to lnd misbehavior:

And this is profile output of curl http://localhost:9000/debug/pprof/goroutine?debug=1 > lnd_profile_output.txt:

lnd_profile_output.txt

[begetan]

I came here via the search for the similar issue.

I am running lnd connected to remote bitcoind backend. I've discovered, that after missing the connect to the btc backend lnd daemon goes to eat 99% of the cpu (single - core instance)
It may be some not recovered network calls going to infinite cycle after connectivity issue.

@LLeny could you tell please how do you connect to bitcoin backend?
In your profile I see gozmq calls, so it is definitively related to the bitcoind zero mq communication.

Step to reproduce my issue. It may be the same issue from this topic.

1. Run lnd connected remotely to Bitcoind back-end
2. Interrupt the connection between lnd and bictoind
3. lncli getinfo - shows no response
4. Restore back the connection
5. lncli getinfo - shows response but lnd eats 99% of the cpu core.
6. The restart of the daemon fixes this behavior.

[wpaulino]

@Talkless thanks for the report, though the profile you've provided is for goroutines. We'll need a CPU one instead: http://localhost:9000/debug/pprof/profile.

@begetan have you confirmed whether you're experiencing the same gossip issue pointed out in #3370 (comment)?

[Talkless]

@Talkless thanks for the report, though the profile you've provided is for goroutines. We'll need a CPU one instead: http://localhost:9000/debug/pprof/profile.

Oh, I've blindly copied that line from https://github.com/lightningnetwork/lnd/blob/6afee3d0997f1661710390b39746c61f38a1c41f/docs/debugging_lnd.md#capturing-pprof-data-with-lnd . Maybe it's worth to change it there..?

[Talkless]

Curl fetch some binary output, so this is go tool pprof -text output, not sure if that's what you need:

text.txt