`LimitNOFILE` env var in systemd should be higher

[jbowen93]

This is the page they should update: https://docs.celestia.org/nodes/systemd

What Happens?

• badgerdb uses extensive files to run
  • Badger keeps the file descriptors open for SST and vlog files.
  • badger doesn't release the file descriptors until the db (or the file) is closed.
• Celestia Docs for systemd set the ulimit at 4096 LimitNOTFILE=4096
• Badger themselves say it needs a lot of open files to run properly in production (>1million)
  • https://dgraph.io/docs/deploy/troubleshooting/#too-many-open-files-errors
• Even if you set DefaultLimitNoFILE to a large number for ALL processes, by setting the value in systemd it will use the systemd limit

From our bridge node, current open files in badgerDB - 1330736

[jcstein]

thanks for the issue. @Wondertan / @ramin what should we recommend for all node types? it seems that recommendation is that badger will be >1million for all full, light and bridge nodes?

[jcstein]

I think we will also need to increase the value for celestia-app and will tag in @rootulp to help with that

[rootulp]

It looks like celestia-core uses the goleveldb backend by default so I think this is only applicable if a consensus node operator configures their DB backend to badgerdb. badgerdb docs state

A soft limit of 1,048,576 open files is the recommended minimum to use Dgraph in production, but you can try increasing this soft limit if you continue to see this error. To learn more, see the ulimit documentation for your operating system.

and based on the OP, we should bump to a value even higher than that for bridge nodes. For consensus nodes, I'll run a mainnet consensus node with the badgerdb backend and try to collect data on open files and report back.

Ref:

• https://github.com/celestiaorg/celestia-core/blob/40ee792bb2f878dfaba02f352c4951386eb59883/config/toml.go#L95-L114
• https://dgraph.io/docs/deploy/troubleshooting/#too-many-open-files-errors

[jcstein]

Hey @rootulp I was mistaken in tagging you here. I don't think you'll need to test this, I didn't realize the DB was different.

[jcstein]

Maybe a better question if the current limit for celestia-app is accurate @rootulp ?

[rootulp]

Consensus node on mainnet:

Database	Number of open files
goleveldb	12,673
badgerdb	-

I can't get a consensus node to sync on mainnet with badgerdb because it remains stuck after:

root@rootul-mainnet-consensus:~/celestia-app# ./build/celestia-appd start
6:53PM INF starting node with ABCI Tendermint in-process

but I think we should bump from 4096 to something greater than 12,673 to support the default db (goleveldb).

[jcstein]

thank you @rootulp for the testing and analysis - do you think that using 13- or 14,000 is enough?

[rootulp]

This line makes me think that goleveldb isn't actually consuming that many open files. Regardless, I'm not aware of any downsides to bumping this value so my recommendation would be to bump to the next power of 2 larger than 12,673 (e.g. 16,384) for consensus nodes to leave room for some buffer.

Note: if we're going to bump the bridge node recommendation to something greater than 1 million then 16,384 seems small in comparison.

[jcstein]

Checking in here @jbowen93 - have you had this go higher than what you reported? What do you think a safe value would be to include a buffer?

[jcstein]

i'm leaning toward of using 1400000 for all DA nodes, unless you all think it is advisable to go to the next power of 2, which in this case is 2097152?

[jbowen93]

I don't know off the top of my head if I've seen it go higher but I think 1,400,000 is probably fine.