Can't start after upgrade thehive4 (4.1.16-1) over (4.0.0-1) [Bug]

[huesitosloco]

Request Type

Can't start The Hive after an upgrade [ (4.1.16-1) over (4.0.0-1)],

Work Environment

Question	Answer
OS version (server)	Ubuntu 18.04.4 LTS
Virtualized Env.	True
Dedicated RAM	8 GB
vCPU	4
TheHive version / git hash	(4.1.16-1) upgrading from (4.0.0-1)
Package Type	DEB
Database	Cassandra
Index type	Lucene
Attachments storage	Local

Problem Description

on the logs, I noticed a warning regarding a GLOBAL_OFFLINE variable that can not be overriden by the information found in /etc/thehive/application.conf

Full message:
2021-12-30 23:15:07,136 [WARN] from org.janusgraph.diskstorage.configuration.builder.ReadConfigurationBuilder in application-akka.actor.default-dispatcher-11 [|] Local setting index.search.index-name=scalligraph (Type: GLOBAL_OFFLINE) is overridden by globally managed value (janusgraph). Use the ManagementSystem interface instead of the local configuration to control this setting.

After this message, it continues to load, but encounters another problem:

2021-12-30 23:15:11,849 [TRACE] from org.thp.scalligraph.models.Database in application-akka.actor.default-dispatcher-11 [|mgmt-33c851d7] mgmt.makePropertyKey(colour).dataType(String.class).cardinality(SINGLE).make()
2021-12-30 23:15:11,857 [ERROR] from org.thp.scalligraph.models.Database in application-akka.actor.default-dispatcher-11 [|mgmt-33c851d7] Unable to add property colour
org.thp.scalligraph.InternalError: Property colour exists with incompatible type: SINGLE:class java.lang.String Vs SINGLE:class java.lang.Integer
	at org.thp.scalligraph.janus.JanusDatabase.addProperty(JanusDatabase.scala:409)
	at org.thp.scalligraph.janus.JanusDatabase.$anonfun$addProperties$4(JanusDatabase.scala:367)
	at org.thp.scalligraph.package$RichSeq.$anonfun$toTry$3(package.scala:17)
	at scala.collection.TraversableOnce$folder$1.apply(TraversableOnce.scala:196)
	at scala.collection.TraversableOnce$folder$1.apply(TraversableOnce.scala:194)
	at scala.collection.immutable.HashMap$HashMap1.foreach(HashMap.scala:399)
	at scala.collection.immutable.HashMap$HashTrieMap.foreach(HashMap.scala:725)
	at scala.collection.immutable.HashMap$HashTrieMap.foreach(HashMap.scala:725)
	at scala.collection.TraversableOnce.foldLeft(TraversableOnce.scala:199)
	at scala.collection.TraversableOnce.foldLeft$(TraversableOnce.scala:192)
	at scala.collection.AbstractTraversable.foldLeft(Traversable.scala:108)
	at org.thp.scalligraph.package$RichSeq.toTry(package.scala:16)
	at org.thp.scalligraph.janus.JanusDatabase.addProperties(JanusDatabase.scala:358)
	at org.thp.scalligraph.janus.JanusDatabase.$anonfun$createSchema$4(JanusDatabase.scala:310)
	at scala.util.Success.flatMap(Try.scala:251)
	at org.thp.scalligraph.janus.JanusDatabase.$anonfun$createSchema$3(JanusDatabase.scala:309)
	at scala.util.Success.flatMap(Try.scala:251)
	at org.thp.scalligraph.janus.JanusDatabase.$anonfun$createSchema$1(JanusDatabase.scala:308)
	at org.thp.scalligraph.janus.JanusDatabase.$anonfun$managementTransaction$6(JanusDatabase.scala:292)
	at scala.util.Try$.apply(Try.scala:213)
	at org.thp.scalligraph.janus.JanusDatabase.$anonfun$managementTransaction$4(JanusDatabase.scala:292)
	at scala.util.Try$.apply(Try.scala:213)
	at org.thp.scalligraph.utils.Retry.org$thp$scalligraph$utils$Retry$$runTry(Retry.scala:61)
	at org.thp.scalligraph.utils.Retry.withTry(Retry.scala:26)
	at org.thp.scalligraph.janus.JanusDatabase.managementTransaction(JanusDatabase.scala:288)
	at org.thp.scalligraph.janus.JanusDatabase.createSchema(JanusDatabase.scala:305)
	at org.thp.scalligraph.janus.JanusDatabaseProvider.$anonfun$get$3(JanusDatabaseProvider.scala:138)
	at scala.util.Success.$anonfun$map$1(Try.scala:255)
	at scala.util.Success.map(Try.scala:213)
	at scala.concurrent.Future.$anonfun$map$1(Future.scala:292)
	at scala.concurrent.impl.Promise.liftedTree1$1(Promise.scala:33)
	at scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:33)
	at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:64)
	at org.thp.scalligraph.ContextPropagatingDispatcher$$anon$1.$anonfun$execute$2(ContextPropagatingDisptacher.scala:57)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
	at org.thp.scalligraph.DiagnosticContext$$anon$2.withContext(ContextPropagatingDisptacher.scala:77)
	at org.thp.scalligraph.ContextPropagatingDispatcher$$anon$1.$anonfun$execute$1(ContextPropagatingDisptacher.scala:57)
	at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:48)
	at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:48)
	at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
	at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
	at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
	at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175)
2021-12-30 23:15:11,861 [ERROR] from org.thp.scalligraph.utils.Retry in application-akka.actor.default-dispatcher-11 [|] uncaught error, not retrying
org.thp.scalligraph.InternalError: Property colour exists with incompatible type: SINGLE:class java.lang.String Vs SINGLE:class java.lang.Integer
	at org.thp.scalligraph.janus.JanusDatabase.addProperty(JanusDatabase.scala:409)
	at org.thp.scalligraph.janus.JanusDatabase.$anonfun$addProperties$4(JanusDatabase.scala:367)
	at org.thp.scalligraph.package$RichSeq.$anonfun$toTry$3(package.scala:17)
	at scala.collection.TraversableOnce$folder$1.apply(TraversableOnce.scala:196)
	at scala.collection.TraversableOnce$folder$1.apply(TraversableOnce.scala:194)
	at scala.collection.immutable.HashMap$HashMap1.foreach(HashMap.scala:399)
	at scala.collection.immutable.HashMap$HashTrieMap.foreach(HashMap.scala:725)
	at scala.collection.immutable.HashMap$HashTrieMap.foreach(HashMap.scala:725)
	at scala.collection.TraversableOnce.foldLeft(TraversableOnce.scala:199)
	at scala.collection.TraversableOnce.foldLeft$(TraversableOnce.scala:192)
	at scala.collection.AbstractTraversable.foldLeft(Traversable.scala:108)
	at org.thp.scalligraph.package$RichSeq.toTry(package.scala:16)
	at org.thp.scalligraph.janus.JanusDatabase.addProperties(JanusDatabase.scala:358)
	at org.thp.scalligraph.janus.JanusDatabase.$anonfun$createSchema$4(JanusDatabase.scala:310)
	at scala.util.Success.flatMap(Try.scala:251)
	at org.thp.scalligraph.janus.JanusDatabase.$anonfun$createSchema$3(JanusDatabase.scala:309)
	at scala.util.Success.flatMap(Try.scala:251)
	at org.thp.scalligraph.janus.JanusDatabase.$anonfun$createSchema$1(JanusDatabase.scala:308)
	at org.thp.scalligraph.janus.JanusDatabase.$anonfun$managementTransaction$6(JanusDatabase.scala:292)
	at scala.util.Try$.apply(Try.scala:213)
	at org.thp.scalligraph.janus.JanusDatabase.$anonfun$managementTransaction$4(JanusDatabase.scala:292)
	at scala.util.Try$.apply(Try.scala:213)
	at org.thp.scalligraph.utils.Retry.org$thp$scalligraph$utils$Retry$$runTry(Retry.scala:61)
	at org.thp.scalligraph.utils.Retry.withTry(Retry.scala:26)
	at org.thp.scalligraph.janus.JanusDatabase.managementTransaction(JanusDatabase.scala:288)
	at org.thp.scalligraph.janus.JanusDatabase.createSchema(JanusDatabase.scala:305)
	at org.thp.scalligraph.janus.JanusDatabaseProvider.$anonfun$get$3(JanusDatabaseProvider.scala:138)
	at scala.util.Success.$anonfun$map$1(Try.scala:255)
	at scala.util.Success.map(Try.scala:213)
	at scala.concurrent.Future.$anonfun$map$1(Future.scala:292)
	at scala.concurrent.impl.Promise.liftedTree1$1(Promise.scala:33)
	at scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:33)
	at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:64)
	at org.thp.scalligraph.ContextPropagatingDispatcher$$anon$1.$anonfun$execute$2(ContextPropagatingDisptacher.scala:57)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
	at org.thp.scalligraph.DiagnosticContext$$anon$2.withContext(ContextPropagatingDisptacher.scala:77)
	at org.thp.scalligraph.ContextPropagatingDispatcher$$anon$1.$anonfun$execute$1(ContextPropagatingDisptacher.scala:57)
	at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:48)
	at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:48)
	at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
	at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
	at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
	at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175)

Steps to Reproduce

1. Applied the changes to /etc/thehive/application.conf as writen in https://docs.thehive-project.org/thehive/operations/update/

This is an extract of my config file:

db.janusgraph {
  storage {
    backend: cql
    hostname: ["127.0.0.1"]
    cql {
      cluster-name: thp
      keyspace: thehive
      read-consistency-level: ONE
      write-consistency-level: ONE
    }
  }


 index {
      search {
        backend: lucene
        directory: /opt/thp/thehive/index
      }
    }
  // storage.backend: berkeleyje
  // storage.directory: /path/to/berkeleydb
  // berkeleyje.freeDisk: 200 
}

storage {

  provider: localfs
  localfs.directory: /opt/thp_data/files/thehive
  // provider: hdfs
  // hdfs {
  //   root: "hdfs://localhost:10000" # namenode server hostname
  //   location: "/thehive"           # location inside HDFS
  //   username: thehive              # file owner
  // }
}

2. Restarted The Hive, everything works properly
3. Installed the stable verion following this guide https://docs.thehive-project.org/thehive/installation-and-configuration/installation/step-by-step-guide/ , I executed only the steps on the "The Hive" section, since everything else seems to be already in place:

echo 'deb https://deb.thehive-project.org release main' | sudo tee -a /etc/apt/sources.list.d/thehive-project.list
sudo apt-get update
sudo apt-get install thehive4

4.-The hive does not come up again, everytime I restart the service (which appear as "running" as per "service thehive status") I get the same error stated above.

Possible Solutions

I did some reading, and it appears to be related to a variable that is set on the database, and can't be overwriten by the configuration file, since the warning stated the "harcoded" value of scalligraph can't be changed, and the error does refer scalligraph usage. I found a reference to change the GLOBAL_OFFLINE values, but I Tought it was better to ask.
https://docs.janusgraph.org/configs/#global-configuration

Complementary information

application.log.txt
install_application.log.txt

I added the logs from a service restart at application.log.txt, and full logs from the install at install_application.og.txt

[marpoe]

Unfortunately I faced exactely the same issue. Coming from "4.0.2-1".

[nadouani]

@huesitosloco can you please add this property to your application.conf file

db.janusgraph.forceDropAndRebuildIndex: true

Restart TheHive, it will rebuild the index and index the data. Once done, you can remove the db.janusgraph.forceDropAndRebuildIndex config or set it to false

[mihai1978]

I have the same issue and the workarround with db.janusgraph.forceDropAndRebuildIndex is not working for me.

[nadouani]

Hello @huesitosloco @mihai1978 we've investigated the issue with @marpoe and there is in fact a bug that needs to be fixed. Adding this issue to 4.1.17

[mihai1978]

I see, thank you for your answer.