[Bug] (Still) slow loading of list-tags endpoint with 4.1.2

[crackytsi]

Request Type

Bug

Work Environment

Question	Answer
OS version (server)	Debian
OS version (client)	10
TheHive version / git hash	4.1.2
Package Type	DEB

Problem Description

Sorry for popping up this toppic once again :-(

Starting the TheHive page or directly jumping to a case using a link (still) takes several time.
By analyzing this, the root-cause seems to be related to URL:

/api/v1/query?name=list-tags

On my browser it takes about 16 seconds (sec not ms!) to download the file, but the file-size is only 993 Bytes according to browser developer-console.

Meanwhile TheHive application.log shows:
2021-03-30 06:03:17,902 [INFO] from org.thp.scalligraph.AccessLogFilter in application-akka.actor.default-dispatcher-13 [00000520|355b8639] 10.32.30.10 POST /api/v1/query?name=list-tags took 29ms and returned 200
Screenshot of developer console can be seen at the old case: #1869
I can also see that the load on cassandra is very high, I have a system with 8 cores, much more powerfull than the system on TH 3.4 that handled the same cases.

Possible Solutions

Caching? Indexing of Tags? I don't know :-(

[nadouani]

@crackytsi can you please share the POST request body for the /api/v1/query?name=list-tags call that takes 12s?

[crackytsi]

[crackytsi]

[crackytsi]

[crackytsi]

Sure :) Attached some data, if you need more, let me know ;)

[mamoedo]

Same problem here!

[mamoedo]

I hope this helps, my instance is not usable right now:

The same queries in Kibana takes miliseconds

[nadouani]

@mamoedo

• what type of index you are using (Elasticsearch or Lucene)
• what's the status of your index (just asking)
• are using a clustered installation of TheHive
• what are the filters you are applying in this screenshot
• what's the behavior of the alert list without filters?

[mamoedo]

@mamoedo

* what type of index you are using (Elasticsearch or Lucene)

* what's the status of your index (just asking)

* are using a clustered installation of TheHive

* what are the filters you are applying in this screenshot

* what's the behavior of the alert list without filters?

• I'm using ES
• Yellow status. I don't know why it's yellow because it was created by TheHive, and other index like Kibana's are green.
• I'm not using TheHive clustered
• In the alerts screenshot I was using only the "imported" filter
• When I click Alerts it takes a while to load the total number (YYY) of alerts "List of alerts (XX of YYY)", then it loads the alert list at the same time that it loads the number (XX) of filtered alerts.

If I go as admin to the Platform Status, the section "Data index status" is stuck in "Loading index status..." forever.

I'm trying to give as many resources as I can to the Cassandra and ES machines but I'm afraid that it's an implementation issue, because from Kibana the queries are super fast, faster than in TH3.