[Bug] Duplicated entities after "db.janusgraph.forceDropAndRebuildIndex: true" with Elasticsearch index

[jpferrero]

Request Type

Bug

Work Environment

Question	Answer
OS version (server)	CentOS
OS version (client)	7
Virtualized Env.	True
Dedicated RAM	16 GB
vCPU	8
TheHive version / git hash	4.17
Package Type	RPM
Database	Cassandra
Index type	Elasticsearch 7.16
Attachments storage	Local

If I set the parameter 'db.janusgraph.forceDropAndRebuildIndex: true' when using Elasticsearch as index backend, after a while lots of entities are marked as duplicated and data from observables are lost. This is the log of the reindex

2022-02-03 21:10:36,751 [INFO] from org.thp.scalligraph.models.Database in application-akka.actor.default-dispatcher-13 [|mgmt-296117d3] Reindex job 178b1e59 is running
2022-02-03 21:10:37,182 [INFO] from org.janusgraph.graphdb.database.management.ManagementSystem in Thread-61 [|] Index update job successful for [global5]
2022-02-03 21:10:37,752 [INFO] from org.thp.scalligraph.models.Database in application-akka.actor.default-dispatcher-13 [|mgmt-296117d3] Reindex job 178b1e59 is finished
2022-02-03 21:10:37,891 [INFO] from akka.cluster.singleton.ClusterSingletonManager in application-akka.actor.default-dispatcher-13 [|] Singleton manager starting singleton actor [akka://application/user/integrityCheckSingletonManager/singleton]
2022-02-03 21:10:37,891 [INFO] from akka.cluster.singleton.ClusterSingletonManager in application-akka.actor.default-dispatcher-4 [|] ClusterSingletonManager state change [Start -> Oldest]
2022-02-03 21:10:37,930 [INFO] from akka.cluster.singleton.ClusterSingletonManager in application-akka.actor.default-dispatcher-18 [|] Singleton manager starting singleton actor [akka://application/system/singletonManagerCaseNumberLeader/CaseNumberLeader]
2022-02-03 21:10:37,930 [INFO] from akka.cluster.singleton.ClusterSingletonManager in application-akka.actor.default-dispatcher-18 [|] ClusterSingletonManager state change [Start -> Oldest]
2022-02-03 21:10:38,580 [INFO] from akka.cluster.singleton.ClusterSingletonManager in application-akka.actor.default-dispatcher-4 [|] Singleton manager starting singleton actor [akka://application/user/flowSingletonManager/singleton]
2022-02-03 21:10:38,580 [INFO] from akka.cluster.singleton.ClusterSingletonManager in application-akka.actor.default-dispatcher-21 [|] ClusterSingletonManager state change [Start -> Oldest]
2022-02-03 21:10:38,619 [INFO] from play.api.Play in main [|] Application started (Prod) (no global state)
2022-02-03 21:10:38,942 [INFO] from play.core.server.AkkaHttpServer in main [|] Listening for HTTP on /0:0:0:0:0:0:0:0:8000
2022-02-03 21:10:38,943 [INFO] from akka.cluster.singleton.ClusterSingletonProxy in application-akka.actor.default-dispatcher-21 [|] Singleton identified at [akka://application/system/singletonManagerCaseNumberLeader/CaseNumberLeader]
2022-02-03 21:10:39,593 [INFO] from akka.cluster.singleton.ClusterSingletonProxy in application-akka.actor.default-dispatcher-21 [|] Singleton identified at [akka://application/user/flowSingletonManager/singleton]
2022-02-03 21:10:51,487 [INFO] from akka.cluster.singleton.ClusterSingletonProxy in application-akka.actor.default-dispatcher-4 [|] Singleton identified at [akka://application/user/integrityCheckSingletonManager/singleton]
2022-02-03 21:10:51,487 [INFO] from akka.cluster.singleton.ClusterSingletonProxy in application-akka.actor.default-dispatcher-4 [|] Singleton identified at [akka://application/user/integrityCheckSingletonManager/singleton]
2022-02-03 21:10:51,487 [INFO] from akka.cluster.singleton.ClusterSingletonProxy in application-akka.actor.default-dispatcher-4 [|] Singleton identified at [akka://application/user/integrityCheckSingletonManager/singleton]
2022-02-03 21:10:51,487 [INFO] from akka.cluster.singleton.ClusterSingletonProxy in application-akka.actor.default-dispatcher-4 [|] Singleton identified at [akka://application/user/integrityCheckSingletonManager/singleton]
2022-02-03 21:10:51,488 [INFO] from akka.cluster.singleton.ClusterSingletonProxy in application-akka.actor.default-dispatcher-4 [|] Singleton identified at [akka://application/user/integrityCheckSingletonManager/singleton]
2022-02-03 21:10:51,489 [INFO] from akka.cluster.singleton.ClusterSingletonProxy in application-akka.actor.default-dispatcher-4 [|] Singleton identified at [akka://application/user/integrityCheckSingletonManager/singleton]
2022-02-03 21:10:51,489 [INFO] from akka.cluster.singleton.ClusterSingletonProxy in application-akka.actor.default-dispatcher-4 [|] Singleton identified at [akka://application/user/integrityCheckSingletonManager/singleton]
2022-02-03 21:10:51,489 [INFO] from akka.cluster.singleton.ClusterSingletonProxy in application-akka.actor.default-dispatcher-4 [|] Singleton identified at [akka://application/user/integrityCheckSingletonManager/singleton]
2022-02-03 21:10:51,489 [INFO] from akka.cluster.singleton.ClusterSingletonProxy in application-akka.actor.default-dispatcher-4 [|] Singleton identified at [akka://application/user/integrityCheckSingletonManager/singleton]
2022-02-03 21:10:51,490 [INFO] from akka.cluster.singleton.ClusterSingletonProxy in application-akka.actor.default-dispatcher-4 [|] Singleton identified at [akka://application/user/integrityCheckSingletonManager/singleton]
2022-02-03 21:10:51,490 [INFO] from akka.cluster.singleton.ClusterSingletonProxy in application-akka.actor.default-dispatcher-4 [|] Singleton identified at [akka://application/user/integrityCheckSingletonManager/singleton]
2022-02-03 21:10:51,490 [INFO] from akka.cluster.singleton.ClusterSingletonProxy in application-akka.actor.default-dispatcher-4 [|] Singleton identified at [akka://application/user/integrityCheckSingletonManager/singleton]
2022-02-03 21:10:51,490 [INFO] from akka.cluster.singleton.ClusterSingletonProxy in application-akka.actor.default-dispatcher-4 [|] Singleton identified at [akka://application/user/integrityCheckSingletonManager/singleton]
2022-02-03 21:10:51,636 [INFO] from org.thp.thehive.services.ObservableTypeIntegrityCheckOps in pool-12-thread-1 [|774df931] Found duplicate entities:

• ObservableType(domain,false)
• ObservableType(domain,false)
  2022-02-03 21:11:32,575 [INFO] from org.thp.scalligraph.AccessLogFilter in application-akka.actor.default-dispatcher-13 [00000001|] 127.0.0.1 GET / took 14ms and returned 308 0 bytes
  2022-02-03 21:11:32,648 [INFO] from org.thp.scalligraph.AccessLogFilter in application-akka.actor.default-dispatcher-4 [00000002|] 127.0.0.1 GET /index.html took 38ms and returned 200 1191 bytes
  (.....)
  2022-02-03 21:25:26,943 [INFO] from org.thp.thehive.services.DataIntegrityCheckOps in pool-9-thread-1 [|0956f1f9] Found duplicate entities:
• Data(xxxxxxx,None)
• Data(xxxxxxx,None)
  2022-02-03 21:25:28,008 [INFO] from org.thp.thehive.services.DataIntegrityCheckOps in pool-9-thread-1 [|3eefbb8a] Found duplicate entities:
• Data(xxxxxx,None)
• Data(xxxxxx,None)
  2022-02-03 21:25:28,505 [INFO] from org.thp.thehive.services.DataIntegrityCheckOps in pool-9-thread-1 [|04b74a3b] Found duplicate entities:

Steps to Reproduce

1. Configure Elasticsearch index
2. Force reindex with parameter 'db.janusgraph.forceDropAndRebuildIndex: true'
3. The observables that are considered duplicated disappear from the alert/case:
   

[To-om]

fixed by #2334