Description
Request Type
Bug
Work Environment
Question | Answer |
---|---|
OS version (server) | Debian 9 |
OS version (client) | Windows 10 |
TheHive version / git hash | 4.0.3-1 |
Package Type | DEB |
Browser type & version | Chromium and Firefox latest |
Problem Description
When sending a webhook message to a webhook listener (Nodered, Shuffle, ...) TheHive opens "files" (what linux calls files but its marked as "pipe" using lsof -u thehive
).
These files are opened but never closed, and by the end of the day we reach the hard limit of 65536 files as our instance has a lot of traffic.
We reproduced this issue in clean virtual machines, with different distributions (Debian 9, Debian 10 and Oracle Linux 8) and different JRE versions (8 LTS, 11 LTS, 15) from Adopt OpenJDK. We got the same behaviour.
Each distribution was up-to-date as of 2021-01-10.
Steps to Reproduce
- Install TheHive 4
- Setup webhook
- Do any action that will send a webhook message
- Watch the opened file count increase as TheHive sends webhook message with the command
lsof -u thehive | wc -l
(assuming TheHive runs as user thehive)
Complementary information
This is the exception that appears in the logs when reaching the opened file hard limit.
2021-01-07 10:50:21,306 [ERROR] from akka.io.TcpListener in application-akka.actor.default-dispatcher-5342 [|] Accept error: could not accept new connection
java.io.IOException: Trop de fichiers ouverts
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:421)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:249)
at akka.io.TcpListener.acceptAllPending(TcpListener.scala:128)
at akka.io.TcpListener$$anonfun$bound$1.applyOrElse(TcpListener.scala:96)
at akka.actor.Actor.aroundReceive(Actor.scala:537)
at akka.actor.Actor.aroundReceive$(Actor.scala:535)
at akka.io.TcpListener.aroundReceive(TcpListener.scala:36)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:577)
at akka.actor.ActorCell.invoke(ActorCell.scala:547)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:270)
at akka.dispatch.Mailbox.run(Mailbox.scala:231)
at akka.dispatch.Mailbox.exec(Mailbox.scala:243)
at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175)
You may find the logs with level debug when sending a webhook here : https://gist.github.com/mphbig/bac95e62e77e61e1149f005f74654ed1