[Bug] Analyzer reports dissapear in 4.1.5 (observable already exists error)

[ch0wm3in]

Request Type

Bug

Work Environment

Question	Answer
OS version (server)	Ubuntu
OS version (client)	20.04
Virtualized Env.	True
Dedicated RAM	8 GB
vCPU	4
TheHive version / git hash	4.1.5, 4777b29
Package Type	DEB,
Database	Cassandra
Index type	Lucene
Attachments storage	Local
Browser type & version	Chrome 91.0.4472.77 64bit

Problem Description

After running analyzers on an observable (and all finishing successfully), when loading the Observable page, the analyzer reports suddently dissapear and show "none", as if never run. The "emlattachment" tag is shown, sometimes you can view the report, when running it again, but then it is not viewable once refreshed. The Hive log shows this error multiple times:

2021-06-04 10:55:52,291 [ERROR] from org.thp.scalligraph.utils.Retry in application-akka.actor.default-dispatcher-17 [|4e2225d8] uncaught error, not retrying
org.thp.scalligraph.CreateError: Observable already exists
        at org.thp.thehive.services.ObservableSrv.create(ObservableSrv.scala:95)
        at org.thp.thehive.connector.cortex.services.JobSrv.$anonfun$importCortexArtifacts$7(JobSrv.scala:232)
        at scala.util.Success.flatMap(Try.scala:251)
        at org.thp.thehive.connector.cortex.services.JobSrv.$anonfun$importCortexArtifacts$6(JobSrv.scala:231)
        at org.thp.scalligraph.janus.JanusDatabase.$anonfun$tryTransaction$7(JanusDatabase.scala:241)
        at scala.util.Try$.apply(Try.scala:213)
        at org.thp.scalligraph.janus.JanusDatabase.$anonfun$tryTransaction$6(JanusDatabase.scala:241)
        at scala.util.Try$.apply(Try.scala:213)
        at org.thp.scalligraph.utils.DelayRetry.withTry(Retry.scala:93)
        at org.thp.scalligraph.janus.JanusDatabase.tryTransaction(JanusDatabase.scala:238)
        at org.thp.thehive.connector.cortex.services.JobSrv.$anonfun$importCortexArtifacts$5(JobSrv.scala:229)
        at scala.util.Success.fold(Try.scala:271)
        at org.thp.thehive.connector.cortex.services.JobSrv.$anonfun$importCortexArtifacts$2(JobSrv.scala:224)
        at scala.concurrent.Future$.$anonfun$traverse$1(Future.scala:850)
        at scala.collection.LinearSeqOptimized.foldLeft(LinearSeqOptimized.scala:126)
        at scala.collection.LinearSeqOptimized.foldLeft$(LinearSeqOptimized.scala:122)
        at scala.collection.immutable.List.foldLeft(List.scala:91)
        at scala.concurrent.Future$.traverse(Future.scala:850)
        at org.thp.thehive.connector.cortex.services.JobSrv.importCortexArtifacts(JobSrv.scala:220)
        at org.thp.thehive.connector.cortex.services.JobSrv.$anonfun$finished$5(JobSrv.scala:155)
        at scala.concurrent.Future.$anonfun$flatMap$1(Future.scala:307)
        at scala.concurrent.impl.Promise.$anonfun$transformWith$1(Promise.scala:41)
        at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:64)
        at org.thp.scalligraph.ContextPropagatingDispatcher$$anon$1.$anonfun$execute$2(ContextPropagatingDisptacher.scala:57)
        at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
        at org.thp.scalligraph.DiagnosticContext$.$anonfun$withDiagnosticContext$2(ContextPropagatingDisptacher.scala:93)
        at org.thp.scalligraph.DiagnosticContext$.saveDiagnosticContext(ContextPropagatingDisptacher.scala:108)
        at org.thp.scalligraph.DiagnosticContext$.withDiagnosticContext(ContextPropagatingDisptacher.scala:91)
        at org.thp.scalligraph.DiagnosticContext$$anon$2.withContext(ContextPropagatingDisptacher.scala:76)
        at org.thp.scalligraph.ContextPropagatingDispatcher$$anon$1.$anonfun$execute$1(ContextPropagatingDisptacher.scala:57)
        at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:48)
        at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:48)
        at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
        at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
        at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
        at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175)

Steps to Reproduce

1. Run analyzers on file observable, and ensure that the job in cortex finished sucessfully.
2. Open the observable page to see reports.
3. Error will happen and no reports will be shown, The Hive logs shows the same error multiple times.

Complementary information

Related to #1982

[ch0wm3in]

This seems to only happen for emlparser analyzer and not ie. fileinfo analyzer.

[ThomasHeimann242]

Not only with the emlparser. It happens also with urlscan and joesandbox. I testet emlparser 1.2 and 1.3 -> same problem.
Always the correct result is available in Cortex, but not in TheHive.

[ch0wm3in]

Ok, i've investigated this a little closer.
It seems that when an analyzer tries to report "would be" observables under the report
It requires uniqueness on observables/artifacts already in case, i tested this with a custom Office365 safelink analyzer.

So if you comment out the "def artifacts()" in emlparser or any other analyzer where this issue persists it works.

So eg. if you already have an domain artifact in case with data: test.org, and EmlParser extracts domain: test.org, the report will fail.