Skip to content

[Bug] Webhooks opening infinite amount of files #1743

Closed
@mphbig

Description

@mphbig

Request Type

Bug

Work Environment

Question Answer
OS version (server) Debian 9
OS version (client) Windows 10
TheHive version / git hash 4.0.3-1
Package Type DEB
Browser type & version Chromium and Firefox latest

Problem Description

When sending a webhook message to a webhook listener (Nodered, Shuffle, ...) TheHive opens "files" (what linux calls files but its marked as "pipe" using lsof -u thehive).
These files are opened but never closed, and by the end of the day we reach the hard limit of 65536 files as our instance has a lot of traffic.

We reproduced this issue in clean virtual machines, with different distributions (Debian 9, Debian 10 and Oracle Linux 8) and different JRE versions (8 LTS, 11 LTS, 15) from Adopt OpenJDK. We got the same behaviour.
Each distribution was up-to-date as of 2021-01-10.

Steps to Reproduce

  1. Install TheHive 4
  2. Setup webhook
  3. Do any action that will send a webhook message
  4. Watch the opened file count increase as TheHive sends webhook message with the command lsof -u thehive | wc -l (assuming TheHive runs as user thehive)

Complementary information

This is the exception that appears in the logs when reaching the opened file hard limit.

2021-01-07 10:50:21,306 [ERROR] from akka.io.TcpListener in application-akka.actor.default-dispatcher-5342 [|] Accept error: could not accept new connection
java.io.IOException: Trop de fichiers ouverts
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:421)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:249)
        at akka.io.TcpListener.acceptAllPending(TcpListener.scala:128)
        at akka.io.TcpListener$$anonfun$bound$1.applyOrElse(TcpListener.scala:96)
        at akka.actor.Actor.aroundReceive(Actor.scala:537)
        at akka.actor.Actor.aroundReceive$(Actor.scala:535)
        at akka.io.TcpListener.aroundReceive(TcpListener.scala:36)
        at akka.actor.ActorCell.receiveMessage(ActorCell.scala:577)
        at akka.actor.ActorCell.invoke(ActorCell.scala:547)
        at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:270)
        at akka.dispatch.Mailbox.run(Mailbox.scala:231)
        at akka.dispatch.Mailbox.exec(Mailbox.scala:243)
        at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
        at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
        at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
        at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175)

You may find the logs with level debug when sending a webhook here : https://gist.github.com/mphbig/bac95e62e77e61e1149f005f74654ed1

Metadata

Metadata

Assignees

Labels

TheHive4TheHive4 related issuesbug

Type

No type

Projects

No projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions