[Bug] TheHive4 fails to start after reboot

[H2Cyber]

TheHive4 fails to start after reboot

Request Type

Bug

Work Environment

Question	Answer
OS version (server)	Ubuntu Server 20.04
TheHive version / git hash	4.0.2
Package Type	DEB

Problem Description

TheHive service fails to start after system reboot.

Steps to Reproduce

1. Install using DEB packages using the following steps
2. Reboot the system
3. Observe the following error in the logs:

2020-12-07 21:08:24,069 [ERROR] from akka.actor.OneForOneStrategy in application-akka.actor.default-dispatcher-13 [|] Cannot access element because its enclosing transaction is closed and unbound
akka.actor.ActorInitializationException: akka://application/user/integrityCheckSingletonManager/singleton: exception during creation
        at akka.actor.ActorInitializationException$.apply(Actor.scala:196)
        at akka.actor.ActorCell.create(ActorCell.scala:661)
        at akka.actor.ActorCell.invokeAll$1(ActorCell.scala:513)
        at akka.actor.ActorCell.systemInvoke(ActorCell.scala:535)
        at akka.dispatch.Mailbox.processAllSystemMessages(Mailbox.scala:295)
        at akka.dispatch.Mailbox.run(Mailbox.scala:230)
        at akka.dispatch.Mailbox.exec(Mailbox.scala:243)
        at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
        at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
        at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
        at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175)
Caused by: java.lang.IllegalStateException: Cannot access element because its enclosing transaction is closed and unbound
        at org.janusgraph.graphdb.transaction.StandardJanusGraphTx.getNextTx(StandardJanusGraphTx.java:305)
        at org.janusgraph.graphdb.vertices.AbstractVertex.it(AbstractVertex.java:53)
        at org.janusgraph.graphdb.vertices.AbstractVertex.it(AbstractVertex.java:37)
        at org.janusgraph.graphdb.internal.AbstractElement.isLoaded(AbstractElement.java:136)
        at org.janusgraph.graphdb.types.vertices.JanusGraphSchemaVertex.name(JanusGraphSchemaVertex.java:49)
        at org.janusgraph.graphdb.vertices.AbstractVertex.label(AbstractVertex.java:121)
        at org.janusgraph.graphdb.types.system.ImplicitKey.computeProperty(ImplicitKey.java:83)
        at org.janusgraph.graphdb.query.vertex.BasicVertexCentricQueryBuilder.executeImplicitKeyQuery(BasicVertexCentricQueryBuilder.java:211)
        at org.janusgraph.graphdb.query.vertex.VertexCentricQueryBuilder.properties(VertexCentricQueryBuilder.java:99)
        at org.janusgraph.graphdb.util.ElementHelper.getValues(ElementHelper.java:41)
        at org.janusgraph.graphdb.query.condition.PredicateCondition.evaluate(PredicateCondition.java:68)
        at org.janusgraph.graphdb.query.condition.And.evaluate(And.java:55)
        at org.janusgraph.graphdb.query.graph.GraphCentricQuery.matches(GraphCentricQuery.java:153)
        at org.janusgraph.graphdb.query.QueryProcessor.lambda$getFilterIterator$2(QueryProcessor.java:133)
        at com.google.common.collect.Iterators$5.computeNext(Iterators.java:637)
        at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:141)
        at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:136)
        at org.janusgraph.graphdb.query.ResultSetIterator.nextInternal(ResultSetIterator.java:54)
        at org.janusgraph.graphdb.query.ResultSetIterator.next(ResultSetIterator.java:67)
        at org.janusgraph.graphdb.query.ResultSetIterator.next(ResultSetIterator.java:28)
        at com.google.common.collect.Iterators$5.computeNext(Iterators.java:636)
        at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:141)
        at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:136)
        at com.google.common.collect.Iterators$ConcatenatedIterator.hasNext(Iterators.java:1324)
        at java.util.Spliterators$IteratorSpliterator.tryAdvance(Spliterators.java:1811)
        at java.util.Spliterators$1Adapter.hasNext(Spliterators.java:681)
        at org.janusgraph.graphdb.util.MultiDistinctOrderedIterator.hasNext(MultiDistinctOrderedIterator.java:75)
        at org.apache.tinkerpop.gremlin.process.traversal.step.map.GraphStep.processNextStart(GraphStep.java:149)
        at org.apache.tinkerpop.gremlin.process.traversal.step.util.AbstractStep.hasNext(AbstractStep.java:143)
        at org.apache.tinkerpop.gremlin.process.traversal.util.DefaultTraversal.hasNext(DefaultTraversal.java:197)
        at scala.collection.convert.Wrappers$JIteratorWrapper.hasNext(Wrappers.scala:43)
        at org.thp.scalligraph.traversal.TraversalOps$TraversalOpsDefs$$anon$1.getNextValue(TraversalOps.scala:44)
        at org.thp.scalligraph.traversal.TraversalOps$TraversalOpsDefs$$anon$1.hasNext(TraversalOps.scala:56)
        at scala.collection.Iterator.foreach(Iterator.scala:943)
        at scala.collection.Iterator.foreach$(Iterator.scala:943)
        at scala.collection.AbstractIterator.foreach(Iterator.scala:1431)
        at org.thp.scalligraph.services.IntegrityCheckOps.$anonfun$getDuplicates$4(IntegrityCheckOps.scala:43)
        at org.thp.scalligraph.janus.JanusDatabase.roTransaction(JanusDatabase.scala:153)
        at org.thp.scalligraph.services.IntegrityCheckOps.getDuplicates(IntegrityCheckOps.scala:37)
        at org.thp.scalligraph.services.IntegrityCheckOps.getDuplicates$(IntegrityCheckOps.scala:30)
        at org.thp.thehive.services.UserIntegrityCheckOps.getDuplicates(UserSrv.scala:340)
        at org.thp.scalligraph.services.IntegrityCheckOps.$anonfun$duplicateEntities$2(IntegrityCheckOps.scala:231)
        at scala.Option.fold(Option.scala:251)
        at org.thp.scalligraph.services.IntegrityCheckOps.duplicateEntities(IntegrityCheckOps.scala:231)
        at org.thp.scalligraph.services.IntegrityCheckOps.duplicateEntities$(IntegrityCheckOps.scala:231)
        at org.thp.thehive.services.UserIntegrityCheckOps.duplicateEntities(UserSrv.scala:340)
        at org.thp.scalligraph.services.IntegrityCheckOps.check(IntegrityCheckOps.scala:234)
        at org.thp.scalligraph.services.IntegrityCheckOps.check$(IntegrityCheckOps.scala:233)
        at org.thp.thehive.services.UserIntegrityCheckOps.check(UserSrv.scala:368)
        at org.thp.thehive.services.IntegrityCheckActor.$anonfun$preStart$3(IntegrityCheckActor.scala:61)
        at scala.collection.immutable.HashSet$HashSet1.foreach(HashSet.scala:330)
        at scala.collection.immutable.HashSet$HashTrieSet.foreach(HashSet.scala:1104)
        at org.thp.thehive.services.IntegrityCheckActor.preStart(IntegrityCheckActor.scala:60)
        at akka.actor.Actor.aroundPreStart(Actor.scala:548)
        at akka.actor.Actor.aroundPreStart$(Actor.scala:548)
        at org.thp.thehive.services.IntegrityCheckActor.aroundPreStart(IntegrityCheckActor.scala:25)
        at akka.actor.ActorCell.create(ActorCell.scala:641)
        ... 9 common frames omitted
2020-12-07 21:08:24,069 [INFO] from akka.cluster.singleton.ClusterSingletonManager in application-akka.actor.default-dispatcher-13 [|] Singleton actor [akka://application/user/integrityCheckSingletonManager/singleton] was terminated

[H2Cyber]

By the way, starting TheHive service using systemctl fixes the issue, up until the machine is rebooted again.
Let me know if I can provide anything to help troobleshoot this further ?

[nadouani]

Hello @aim4r we are able te reproduce the issue in some case, so we will investigate it.

[H2Cyber]

Hi @nadouani and thanks for the follow up.
I had a similar issue with Cortex, which occasionally fails to start up properly following a system reboot.
I was able to get both startup issues (TheHive and Cortex) fixed by changing the process boot order as follows:

sudo systemctl edit thehive.service
	After=cassandra.service
	Requires=cassandra.service
sudo systemctl reload thehive.service
sudo systemctl edit cortex.service
	After=elasticsearch.service
	Requires=elasticsearch.service
systemctl reload cortex.service

[nadouani]

Hello @aim4r could you please tell us a bit more about your installation? Are the databases installed in the same machines as TheHive and/or Cortex?

Where does cassandra.service come from?

The issue looks like TheHive is started before Cassandra is ready.

[H2Cyber]

Yes all components (Cassandra, TheHive, Cortex, ES) are installed on the same box. cassandra.service is the service name for Cassandra.

I managed to prevent TheHive service from starting before Cassandra does, by editing thehive.service (sudo systemctl edit thehive.service), and adding the following two lines :

After=cassandra.service
Requires=cassandra.service

Same for Cortex, I had to edit cortex.service (sudo systemctl edit cortex.service) to ensure it only runs after ES is started.

I confirm the issue has been resolved for me this way.

[H2Cyber]

Closing as this seems to be resolved in 4.0.4 #1725