[Bug] Analyzer reports dissapear in 4.1.4 (observable already exists error)

[androssfox]

Request Type

Bug

Work Environment

Question	Answer
OS version (server)	Ubuntu
OS version (client)	Ubuntu
Virtualized Env.	True
Dedicated RAM	48 GB
vCPU	12
TheHive version / git hash	4.1.4
Package Type	DEB
Database	Cassandra
Index type	Lucene
Attachments storage	HDFS

Problem Description

After running analyzers on an observable (and all finishing successfully), when loading the Observable page, the analyzer reports suddently dissapear and show "none", as if never run. The Hive log shows this error multiple times:

2021-04-19 07:36:10,082 [ERROR] from org.thp.scalligraph.utils.Retry in application-akka.actor.default-dispatcher-13 [|10e08eec] uncaught error, not retrying
org.thp.scalligraph.CreateError: Observable already exists
        at org.thp.thehive.services.ObservableSrv.create(ObservableSrv.scala:95)
        at org.thp.thehive.connector.cortex.services.JobSrv.$anonfun$importCortexArtifacts$5(JobSrv.scala:229)
        at scala.util.Success.flatMap(Try.scala:251)
        at org.thp.thehive.connector.cortex.services.JobSrv.$anonfun$importCortexArtifacts$4(JobSrv.scala:228)
        at org.thp.scalligraph.janus.JanusDatabase.$anonfun$tryTransaction$7(JanusDatabase.scala:241)
        at scala.util.Try$.apply(Try.scala:213)
        at org.thp.scalligraph.janus.JanusDatabase.$anonfun$tryTransaction$6(JanusDatabase.scala:241)
        at scala.util.Try$.apply(Try.scala:213)
        at org.thp.scalligraph.utils.DelayRetry.withTry(Retry.scala:93)
        at org.thp.scalligraph.janus.JanusDatabase.tryTransaction(JanusDatabase.scala:238)
        at org.thp.thehive.connector.cortex.services.JobSrv.$anonfun$importCortexArtifacts$2(JobSrv.scala:226)
        at scala.concurrent.Future$.$anonfun$traverse$1(Future.scala:850)
        at scala.collection.LinearSeqOptimized.foldLeft(LinearSeqOptimized.scala:126)
        at scala.collection.LinearSeqOptimized.foldLeft$(LinearSeqOptimized.scala:122)
        at scala.collection.immutable.List.foldLeft(List.scala:91)
        at scala.concurrent.Future$.traverse(Future.scala:850)
        at org.thp.thehive.connector.cortex.services.JobSrv.importCortexArtifacts(JobSrv.scala:220)
        at org.thp.thehive.connector.cortex.services.JobSrv.$anonfun$finished$5(JobSrv.scala:155)
        at scala.concurrent.Future.$anonfun$flatMap$1(Future.scala:307)
        at scala.concurrent.impl.Promise.$anonfun$transformWith$1(Promise.scala:41)
        at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:64)
        at org.thp.scalligraph.ContextPropagatingDispatcher$$anon$1.$anonfun$execute$2(ContextPropagatingDisptacher.scala:57)
        at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
        at org.thp.scalligraph.DiagnosticContext$.$anonfun$withDiagnosticContext$2(ContextPropagatingDisptacher.scala:93)
        at org.thp.scalligraph.DiagnosticContext$.saveDiagnosticContext(ContextPropagatingDisptacher.scala:108)
        at org.thp.scalligraph.DiagnosticContext$.withDiagnosticContext(ContextPropagatingDisptacher.scala:91)
        at org.thp.scalligraph.DiagnosticContext$$anon$2.withContext(ContextPropagatingDisptacher.scala:76)
        at org.thp.scalligraph.ContextPropagatingDispatcher$$anon$1.$anonfun$execute$1(ContextPropagatingDisptacher.scala:57)
        at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:48)
        at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:48)
        at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
        at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
        at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
        at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175)
2021-04-19 07:36:10,083 [ERROR] from org.thp.scalligraph.models.Database in application-akka.actor.default-dispatcher-13 [|10e08eec] Exception raised, rollback (Observable already exists)

This error happens mainly with file analyzers, such as EMLparser.

As no error occurs in Cortex, we believe it is an issue in TheHive.

Steps to Reproduce

1. Run analyzers on file observable, and ensure that the job in cortex finished sucessfully.
2. Open the observable page to see reports.
3. Error will happen and no reports will be shown, The Hive logs shows the same error multiple times.

Complementary information

We started observing this issue after upgrading The Hive 4.0.5 to 4.1.4.

[mphbig]

Hello,

I think this issue is the same as this one : #1905

[kiz1]

+1

[joseluratm]

we have observed the same bug in thehive4.1.4 gives us the errors:
An error occurs (java.lang. IllegalArgumentException: Neither sideEffects, map, nor path has a ff5e8828-ccc5-470e-8bae-8bae-8ba19b4f8f1f-key: WherePredicateStep(eq(ff5e8828-ccc5-470e-8bae-8ba19b4f8f1f)), retrying (1).

and later in other reports:
[�[33mwarn�[0m] o.t.s.u.Retry [00005425|40ec53c3] An error occurs (java.lang.IllegalArgumentException: The provided traverser does not map to a value: v[474452048]->[CoalesceStep([[JanusGraphVertexStep(IN,[AlertObservable],vertex), ProjectStep([2f573d25-3357-4617-80e5-f4e1d7a11e8e, 50b59f2b-9645-45d8-a84e-d63d53cbf923, 96cb0543-c9f4-4c58-b440-bc1a5f018038, 6948af53-9428-48f0-8876-63c4d64f7f20, 67e1ecef-6c8f-4a15-93d2-6b7f07d0e352],[identity, [JanusGraphVertexStep(OUT,[AlertCustomField],edge), ProjectStep([232573d1-b328-4572-a9dc-0f122f867862, 7cc67fa8-c761-4051-b189-719f338fdbdf],[identity, [EdgeVertexStep(IN)]]), FoldStep], [JanusGraphVertexStep(OUT,[AlertCase],vertex), RangeGlobalStep(0,1), IdStep, FoldStep], [JanusGraphVertexStep(OUT,[AlertCaseTemplate],vertex), JanusGraphPropertiesStep([name],value), RangeGlobalStep(0,1), FoldStep], [JanusGraphVertexStep(OUT,[AlertObservable],edge), CountGlobalStep]]), ProjectStep([coalesceIndex, coalesceValue],[[ConstantStep(0)], identity])], [JanusGraphVertexStep(IN,[ShareObservable],vertex), JanusGraphVertexStep(OUT,[ShareCase],vertex), NoOpBarrierStep(2500), ProjectStep([c4d68345-bad4-496b-b20b-4f764f71e722, 2de1f5d3-6cbc-4514-858a-c0314cab7333],[identity, [JanusGraphVertexStep(OUT,[CaseCustomField],edge), ProjectStep([7ca0107f-5595-4b10-804b-99846ffef96d, 6e482b3b-1819-49ff-bee6-ba4dd310bfa0],[identity, [EdgeVertexStep(IN)]]), FoldStep]]), ProjectStep([coalesceIndex, coalesceValue],[[ConstantStep(1)], identity])]])]), retrying (1)

Is happening to us with hashes when we consult a case report.
We are using elasticsearch as a engine.

[danniranderis]

We see the same issue for version 4.1.3 on Ubuntu 20.04.

When accessing observables page we get:

2021-04-20 11:07:12,879 [WARN] from org.thp.scalligraph.utils.Retry in application-akka.actor.default-dispatcher-26 [|] An error occurs (java.lang.IllegalArgumentException: Neither the sideEffects, map, nor path has a 7b9c26f3-2a9c-403c-9c5f-11ec81329cce-key: WherePredicateStep(eq(7b9c26f3-2a9c-403c-9c5f-11ec81329cce))), retrying (1)

After trying to run analyzer again we briefly see the analyzer showing the last time the analyzer ran, but cannot open the report - when the analyzer is done, it shows again as not being run at all. For this we see:

2021-04-20 11:08:52,130 [ERROR] from org.thp.scalligraph.models.Database in application-akka.actor.default-dispatcher-5 [|28ca54e8] Exception raised, rollback (Observable already exists)
2021-04-20 11:08:52,182 [ERROR] from org.thp.scalligraph.utils.Retry in application-akka.actor.default-dispatcher-5 [|45369882] uncaught error, not retrying
org.thp.scalligraph.CreateError: Observable already exists
        at org.thp.thehive.services.ObservableSrv.create(ObservableSrv.scala:95)
        at org.thp.thehive.connector.cortex.services.JobSrv.$anonfun$importCortexArtifacts$5(JobSrv.scala:229)
        at scala.util.Success.flatMap(Try.scala:251)
        at org.thp.thehive.connector.cortex.services.JobSrv.$anonfun$importCortexArtifacts$4(JobSrv.scala:228)
        at org.thp.scalligraph.janus.JanusDatabase.$anonfun$tryTransaction$7(JanusDatabase.scala:241)
        at scala.util.Try$.apply(Try.scala:213)
        at org.thp.scalligraph.janus.JanusDatabase.$anonfun$tryTransaction$6(JanusDatabase.scala:241)
        at scala.util.Try$.apply(Try.scala:213)
        at org.thp.scalligraph.utils.DelayRetry.withTry(Retry.scala:93)
        at org.thp.scalligraph.janus.JanusDatabase.tryTransaction(JanusDatabase.scala:238)
        at org.thp.thehive.connector.cortex.services.JobSrv.$anonfun$importCortexArtifacts$2(JobSrv.scala:226)
        at scala.concurrent.Future$.$anonfun$traverse$1(Future.scala:850)
        at scala.collection.LinearSeqOptimized.foldLeft(LinearSeqOptimized.scala:126)
        at scala.collection.LinearSeqOptimized.foldLeft$(LinearSeqOptimized.scala:122)
        at scala.collection.immutable.List.foldLeft(List.scala:91)
        at scala.concurrent.Future$.traverse(Future.scala:850)
        at org.thp.thehive.connector.cortex.services.JobSrv.importCortexArtifacts(JobSrv.scala:220)
        at org.thp.thehive.connector.cortex.services.JobSrv.$anonfun$finished$5(JobSrv.scala:155)
        at scala.concurrent.Future.$anonfun$flatMap$1(Future.scala:307)
        at scala.concurrent.impl.Promise.$anonfun$transformWith$1(Promise.scala:41)
        at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:64)
        at org.thp.scalligraph.ContextPropagatingDispatcher$$anon$1.$anonfun$execute$2(ContextPropagatingDisptacher.scala:57)
        at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
        at org.thp.scalligraph.DiagnosticContext$.$anonfun$withDiagnosticContext$2(ContextPropagatingDisptacher.scala:93)
        at org.thp.scalligraph.DiagnosticContext$.saveDiagnosticContext(ContextPropagatingDisptacher.scala:108)
        at org.thp.scalligraph.DiagnosticContext$.withDiagnosticContext(ContextPropagatingDisptacher.scala:91)
        at org.thp.scalligraph.DiagnosticContext$$anon$2.withContext(ContextPropagatingDisptacher.scala:76)
        at org.thp.scalligraph.ContextPropagatingDispatcher$$anon$1.$anonfun$execute$1(ContextPropagatingDisptacher.scala:57)
        at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:48)
        at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:48)
        at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
        at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
        at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
        at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175)

[torsolaso]

more (i think) relevant logs.

POST request that give the error:

[INFO]  [�[37minfo�[0m] o.t.s.AccessLogFilter [00005731|] 169.254.130.1 POST /api/v1/query?name=observable-jobs-~147857520 took 3ms and returned 200
[INFO]  [�[37minfo�[0m] o.t.s.AccessLogFilter [00005732|] 169.254.130.1 POST /api/v1/query?name=observable-jobs-~147857520.count took 1ms and returned 200 1 bytes
[INFO]  [�[33mwarn�[0m] o.t.s.u.Retry [|] An error occurs (java.lang.IllegalArgumentException: Neither the sideEffects, map, nor path has a 395a1fea-2f16-496c-8873-02c149dc2008-key: WherePredicateStep(eq(395a1fea-2f16-496c-8873-02c149dc2008))), retrying (1)
[INFO]  [�[33mwarn�[0m] o.t.s.u.Retry [|] An error occurs (java.lang.IllegalArgumentException: Neither the sideEffects, map, nor path has a 395a1fea-2f16-496c-8873-02c149dc2008-key: WherePredicateStep(eq(395a1fea-2f16-496c-8873-02c149dc2008))), retrying (2)

[torsolaso]

i observe that taxonomy continue to be attached to the observable on THv4

[LaZyDK]

We are seeing the same issues as explained above. Observable analysis reports are not shown.

[m5050]

also have the same issue here after upgrading Thehive 4.0.5 to 4.1.0

[mphbig]

still having the issue after upgrading to 4.1.4, is anything planned to fix this ?

do you (TheHive devs) need help figuring out what is happening ?

[To-om]

This commit fixes the error The provided traverser does not map to a value