[Bug] Unable to start TheHive after migration

[mamoedo]

Request Type

Bug

Work Environment

Question	Answer
OS version (server)	Debian
TheHive version / git hash	4
Package Type	https://download.thehive-project.org/thehive4-4.0.0-1.zip

Problem Description

TheHive4 crashes when starting with a not empty database. It makes some checks that seem to be slower than the configured timeout when the database is not empty (or it has few data)

Data size:

• Case/Observable: 141936
• Case/Task: 700

Steps to Reproduce

1. Start TheHive
2. Populate the Cassandra database (in this case, using the migration tool)
3. Restart TheHive

Possible Solutions

Increase the timeout. Although it would be nice to increase the checks speed.

Complementary information

[info] o.j.g.d.m.GraphIndexStatusWatcher [|] Some key(s) on index ObservabletypeName_1 do not currently have status(es) [REGISTERED]: name=INSTALLED,_label=INSTALLED
Oops, cannot start the server.
java.util.concurrent.TimeoutException: Futures timed out after [5 minutes]
        at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:259)
        at scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:263)
        at scala.concurrent.Await$.$anonfun$result$1(package.scala:220)
        at scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:57)
        at scala.concurrent.Await$.result(package.scala:146)
        at org.thp.thehive.models.DatabaseProvider.get(SchemaUpdaterActor.scala:53)
        at org.thp.thehive.models.DatabaseProvider.get(SchemaUpdaterActor.scala:19)
        at com.google.inject.internal.ProviderInternalFactory.provision(ProviderInternalFactory.java:85)
        at com.google.inject.internal.BoundProviderFactory.provision(BoundProviderFactory.java:77)
        at com.google.inject.internal.ProviderInternalFactory.circularGet(ProviderInternalFactory.java:59)
        at com.google.inject.internal.BoundProviderFactory.get(BoundProviderFactory.java:61)
        at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:42)
        at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:65)
        at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:113)
        at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:91)
        at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:306)
        at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
        at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:168)
        at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:39)
        at com.google.inject.internal.FactoryProxy.get(FactoryProxy.java:62)
        at com.google.inject.internal.InternalInjectorCreator.loadEagerSingletons(InternalInjectorCreator.java:213)
        at com.google.inject.internal.InternalInjectorCreator.injectDynamically(InternalInjectorCreator.java:184)
        at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:111)
        at com.google.inject.Guice.createInjector(Guice.java:87)
        at com.google.inject.Guice.createInjector(Guice.java:78)
        at play.api.inject.guice.GuiceBuilder.injector(GuiceInjectorBuilder.scala:200)
        at play.api.inject.guice.GuiceApplicationBuilder.build(GuiceApplicationBuilder.scala:155)
        at play.api.inject.guice.GuiceApplicationLoader.load(GuiceApplicationLoader.scala:21)
        at play.core.server.ProdServerStart$.start(ProdServerStart.scala:54)
        at play.core.server.ProdServerStart$.main(ProdServerStart.scala:30)
        at play.core.server.ProdServerStart.main(ProdServerStart.scala)

[nekinie]

Were hitting the same timeout after updating from rc3 to 1.0. Unsure if related but we are also seeing the following before hitting the timeout.

[warn] o.t.s.u.Retry [|345a708b] An error occurs (Adding this property for key [_label] and value [Profile] violates a uniqueness constraint [ProfileName]), retrying (5)
[error] o.t.t.m.SchemaUpdaterActor [|] Database with TheHiveSchema schema update failure
org.thp.scalligraph.models.DatabaseException: Violation of database schema
        at org.thp.scalligraph.janus.JanusDatabase$$anonfun$1.applyOrElse(JanusDatabase.scala:207)
        at org.thp.scalligraph.janus.JanusDatabase$$anonfun$1.applyOrElse(JanusDatabase.scala:205)
        at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:38)
        at scala.util.Failure.recoverWith(Try.scala:236)
        at org.thp.scalligraph.janus.JanusDatabase.tryTransaction(JanusDatabase.scala:205)
        at org.thp.scalligraph.models.BaseDatabase.$anonfun$createSchemaFrom$1(Database.scala:205)
        at scala.util.Success.flatMap(Try.scala:251)
        at org.thp.scalligraph.models.BaseDatabase.createSchemaFrom(Database.scala:204)
        at org.thp.scalligraph.models.Operations.execute(Operation.scala:53)
        at org.thp.scalligraph.models.UpdatableSchema.update(Schema.scala:19)
Caused by: org.janusgraph.core.SchemaViolationException: Adding this property for key [_label] and value [Profile] violates a uniqueness constraint [ProfileName]
        at org.janusgraph.graphdb.transaction.StandardJanusGraphTx.addProperty(StandardJanusGraphTx.java:817)
        at org.janusgraph.graphdb.transaction.StandardJanusGraphTx.addProperty(StandardJanusGraphTx.java:745)
        at org.janusgraph.graphdb.vertices.AbstractVertex.property(AbstractVertex.java:152)
        at org.janusgraph.core.JanusGraphVertex.property(JanusGraphVertex.java:72)
        at org.janusgraph.core.JanusGraphVertex.property(JanusGraphVertex.java:33)
        at org.thp.scalligraph.models.BaseDatabase.$anonfun$setSingleProperty$1(Database.scala:330)
        at scala.Option.foreach(Option.scala:407)
        at org.thp.scalligraph.models.BaseDatabase.setSingleProperty(Database.scala:330)
        at org.thp.scalligraph.janus.JanusDatabase.createVertex(JanusDatabase.scala:543)
        at org.thp.scalligraph.models.InitialValue.create(Schema.scala:14)```

[To-om]

This errors occurs at each startup ?

[mamoedo]

This errors occurs at each startup ?

@To-om yes.

I tried several times and I got the same result

[To-om]

Can you enable debug by adding <logger name="org.thp" level="DEBUG" /> in the right place in /etc/thehive/logback.xml and provide me the logs ?

[mamoedo]

I changed it but I see almost the same logs. This is the few debug log at the beginning of starting thehive:

[debug] o.t.s.m.Database [|] Committing transaction
[debug] o.t.s.m.Database [|] End of transaction
[debug] o.t.t.m.SchemaUpdaterActor [|] Database open instances are: c0a870051268-3f069aa738e91
[info] a.c.s.ClusterSingletonProxy [|] Singleton identified at [akka://application/user/theHiveSchemaUpdaterSingletonManager/singleton]
[info] o.t.t.m.SchemaUpdaterActor [|] Database is ready to be updated
[debug] o.t.s.m.Database [|0649ab9f] Begin of readonly transaction
[debug] o.t.s.m.Database [|0649ab9f] End of readonly transaction

[debug] o.t.s.m.Database [|] Committing transaction
[debug] o.t.s.m.Database [|] End of transaction
[debug] o.t.s.m.Database [|] Committing transaction
[debug] o.t.s.m.Database [|] End of transaction

[To-om]

@mamoedo I can't reproduce this behavior. Try to start TheHive with a fresh database: change the cassandra keyspace in your configuration file (db.janusgraph.storage.keyspace: testKeyspace in application.conf).
If TheHive works, it means that the problem is related to your data. Otherwise there is no reason I can't reproduce (then I'll investigate on you configuration).

[mamoedo]

@To-om changing the cassandra keyspace doesn't solve the problem.

TheHive is stuck showing this errors on the logfile, and the web interface isn't up. Cassandra is up.

Caused by: java.lang.IllegalArgumentException (same stack trace as error #24)
35) Error injecting constructor, java.lang.IllegalArgumentException: Could not instantiate implementation: org.janusgraph.diskstorage.cql.CQLStoreManager
  at org.thp.scalligraph.janus.JanusDatabase.<init>(JanusDatabase.scala:71)
  at org.thp.scalligraph.janus.JanusDatabase.class(JanusDatabase.scala:56)
  while locating org.thp.scalligraph.janus.JanusDatabase
  while locating org.thp.scalligraph.models.Database
    for the 2nd parameter of org.thp.scalligraph.services.config.ApplicationConfig.<init>(ApplicationConfig.scala:20)
  at org.thp.scalligraph.services.config.ApplicationConfig.class(ApplicationConfig.scala:19)
  while locating org.thp.scalligraph.services.config.ApplicationConfig
    for the 2nd parameter of org.thp.thehive.services.TOTPAuthSrvProvider.<init>(TOTPAuthSrv.scala:101)
  at org.thp.thehive.services.TOTPAuthSrvProvider.class(TOTPAuthSrv.scala:100)
  while locating org.thp.thehive.services.TOTPAuthSrvProvider
Caused by: java.lang.IllegalArgumentException: Could not instantiate implementation: org.janusgraph.diskstorage.cql.CQLStoreManager
        at org.janusgraph.util.system.ConfigurationUtil.instantiate(ConfigurationUtil.java:64)
        at org.janusgraph.diskstorage.Backend.getImplementationClass(Backend.java:440)
        at org.janusgraph.diskstorage.Backend.getStorageManager(Backend.java:411)
        at org.janusgraph.graphdb.configuration.builder.GraphDatabaseConfigurationBuilder.build(GraphDatabaseConfigurationBuilder.java:50)
        at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:161)
        at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:132)
        at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:112)
        at org.thp.scalligraph.janus.JanusDatabase$.openDatabase(JanusDatabase.scala:51)
        at org.thp.scalligraph.janus.JanusDatabase.<init>(JanusDatabase.scala:72)
        at org.thp.scalligraph.janus.JanusDatabase$$FastClassByGuice$$113881e3.newInstance(<generated>)
        at com.google.inject.internal.DefaultConstructionProxyFactory$FastClassProxy.newInstance(DefaultConstructionProxyFactory.java:89)
        at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:114)
        at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:91)
        at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:306)
        at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
        at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:168)
        at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:39)
        at com.google.inject.internal.FactoryProxy.get(FactoryProxy.java:62)
        at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:42)
        at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:65)
        at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:113)
        at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:91)
        at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:306)
        at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
        at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:168)
        at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:39)
        at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:42)
        at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:65)
        at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:113)
        at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:91)
        at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:306)
        at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
        at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:168)
        at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:39)
        at com.google.inject.internal.InternalInjectorCreator.loadEagerSingletons(InternalInjectorCreator.java:213)
        at com.google.inject.internal.InternalInjectorCreator.injectDynamically(InternalInjectorCreator.java:184)
        at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:111)
        at com.google.inject.Guice.createInjector(Guice.java:87)
        at com.google.inject.Guice.createInjector(Guice.java:78)
        at play.api.inject.guice.GuiceBuilder.injector(GuiceInjectorBuilder.scala:200)
        at play.api.inject.guice.GuiceApplicationBuilder.build(GuiceApplicationBuilder.scala:155)
        at play.api.inject.guice.GuiceApplicationLoader.load(GuiceApplicationLoader.scala:21)
        at play.core.server.ProdServerStart$.start(ProdServerStart.scala:54)
        at play.core.server.ProdServerStart$.main(ProdServerStart.scala:30)
        at play.core.server.ProdServerStart.main(ProdServerStart.scala)
Caused by: java.lang.reflect.InvocationTargetException
        at sun.reflect.GeneratedConstructorAccessor7.newInstance(Unknown Source)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at org.janusgraph.util.system.ConfigurationUtil.instantiate(ConfigurationUtil.java:58)
        ... 44 more
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra-2/192.168.112.3:9042 (com.datastax.driver.core.exceptions.TransportException: [cassandra-2/192.168.112.3:9042] Cannot connect))
        at com.datastax.driver.core.ControlConnection.reconnectInternal(ControlConnection.java:268)
        at com.datastax.driver.core.ControlConnection.connect(ControlConnection.java:107)
        at com.datastax.driver.core.Cluster$Manager.negotiateProtocolVersionAndConnect(Cluster.java:1813)
        at com.datastax.driver.core.Cluster$Manager.init(Cluster.java:1726)
        at com.datastax.driver.core.Cluster.init(Cluster.java:214)
        at com.datastax.driver.core.Cluster.connectAsync(Cluster.java:387)
        at com.datastax.driver.core.Cluster.connectAsync(Cluster.java:366)
        at com.datastax.driver.core.Cluster.connect(Cluster.java:311)
        at org.janusgraph.diskstorage.cql.CQLStoreManager.initializeSession(CQLStoreManager.java:320)
        at org.janusgraph.diskstorage.cql.CQLStoreManager.<init>(CQLStoreManager.java:182)
        ... 48 more


Caused by: java.lang.IllegalArgumentException (same stack trace as error #33)
46) Error injecting constructor, java.lang.IllegalArgumentException: Could not instantiate implementation: org.janusgraph.diskstorage.cql.CQLStoreManager
  at org.thp.scalligraph.janus.JanusDatabase.<init>(JanusDatabase.scala:71)
  at org.thp.scalligraph.janus.JanusDatabase.class(JanusDatabase.scala:56)
  while locating org.thp.scalligraph.janus.JanusDatabase
  while locating org.thp.scalligraph.models.Database
    for the 1st parameter of org.thp.thehive.models.DatabaseProvider.<init>(SchemaUpdaterActor.scala:20)
  at org.thp.thehive.models.DatabaseProvider.class(SchemaUpdaterActor.scala:19)
  while locating org.thp.thehive.models.DatabaseProvider
  while locating org.thp.scalligraph.models.Database annotated with @com.google.inject.name.Named(value=with-thehive-schema)
    for the 1st parameter of org.thp.thehive.services.LocalUserSrv.<init>(LocalUserSrv.scala:18)
  at org.thp.thehive.services.LocalUserSrv.class(LocalUserSrv.scala:17)
  while locating org.thp.thehive.services.LocalUserSrv
Caused by: java.lang.IllegalArgumentException (same stack trace as error #27)
47) Error injecting constructor, java.lang.IllegalArgumentException: Could not instantiate implementation: org.janusgraph.diskstorage.cql.CQLStoreManager
  at org.thp.scalligraph.janus.JanusDatabase.<init>(JanusDatabase.scala:71)
  at org.thp.scalligraph.janus.JanusDatabase.class(JanusDatabase.scala:56)
  while locating org.thp.scalligraph.janus.JanusDatabase
  while locating org.thp.scalligraph.models.Database
    for the 1st parameter of org.thp.thehive.models.DatabaseProvider.<init>(SchemaUpdaterActor.scala:20)
  at org.thp.thehive.models.DatabaseProvider.class(SchemaUpdaterActor.scala:19)
  while locating org.thp.thehive.models.DatabaseProvider
  while locating org.thp.scalligraph.models.Database annotated with @com.google.inject.name.Named(value=with-thehive-schema)
    for the 2nd parameter of org.thp.thehive.connector.cortex.models.DatabaseProvider.<init>(SchemaUpdaterActor.scala:18)
  at org.thp.thehive.connector.cortex.models.DatabaseProvider.class(SchemaUpdaterActor.scala:17)
  while locating org.thp.thehive.connector.cortex.models.DatabaseProvider
Caused by: java.lang.IllegalArgumentException (same stack trace as error #27)
48) Error injecting constructor, java.lang.IllegalArgumentException: Could not instantiate implementation: org.janusgraph.diskstorage.cql.CQLStoreManager
  at org.thp.scalligraph.janus.JanusDatabase.<init>(JanusDatabase.scala:71)
  at org.thp.scalligraph.janus.JanusDatabase.class(JanusDatabase.scala:56)
  while locating org.thp.scalligraph.janus.JanusDatabase
  while locating org.thp.scalligraph.models.Database
    for the 2nd parameter of org.thp.scalligraph.services.config.ApplicationConfig.<init>(ApplicationConfig.scala:20)
  at org.thp.scalligraph.services.config.ApplicationConfig.class(ApplicationConfig.scala:19)
  while locating org.thp.scalligraph.services.config.ApplicationConfig
    for the 2nd parameter of org.thp.thehive.services.TOTPAuthSrvProvider.<init>(TOTPAuthSrv.scala:101)
  at org.thp.thehive.services.TOTPAuthSrvProvider.class(TOTPAuthSrv.scala:100)
  while locating org.thp.thehive.services.TOTPAuthSrvProvider
  while locating org.thp.scalligraph.auth.AuthSrv
    for the 1st parameter of org.thp.scalligraph.controllers.Entrypoint.<init>(Entrypoint.scala:27)
  at org.thp.scalligraph.controllers.Entrypoint.class(Entrypoint.scala:26)
  while locating org.thp.scalligraph.controllers.Entrypoint
    for the 1st parameter of org.thp.thehive.connector.cortex.controllers.v0.ResponderCtrl.<init>(ResponderCtrl.scala:17)
  at org.thp.thehive.connector.cortex.controllers.v0.ResponderCtrl.class(ResponderCtrl.scala:16)
  while locating org.thp.thehive.connector.cortex.controllers.v0.ResponderCtrl
Caused by: java.lang.IllegalArgumentException (same stack trace as error #24)
49) Error injecting constructor, java.lang.IllegalArgumentException: Could not instantiate implementation: org.janusgraph.diskstorage.cql.CQLStoreManager
  at org.thp.scalligraph.janus.JanusDatabase.<init>(JanusDatabase.scala:71)
  at org.thp.scalligraph.janus.JanusDatabase.class(JanusDatabase.scala:56)
  while locating org.thp.scalligraph.janus.JanusDatabase
  while locating org.thp.scalligraph.models.Database
    for the 2nd parameter of org.thp.scalligraph.services.config.ApplicationConfig.<init>(ApplicationConfig.scala:20)
  at org.thp.scalligraph.services.config.ApplicationConfig.class(ApplicationConfig.scala:19)
  while locating org.thp.scalligraph.services.config.ApplicationConfig
    for the 2nd parameter of org.thp.thehive.services.TOTPAuthSrvProvider.<init>(TOTPAuthSrv.scala:101)
  at org.thp.thehive.services.TOTPAuthSrvProvider.class(TOTPAuthSrv.scala:100)
  while locating org.thp.thehive.services.TOTPAuthSrvProvider
  while locating org.thp.scalligraph.auth.AuthSrv
    for the 1st parameter of org.thp.scalligraph.controllers.Entrypoint.<init>(Entrypoint.scala:27)
  at org.thp.scalligraph.controllers.Entrypoint.class(Entrypoint.scala:26)
  while locating org.thp.scalligraph.controllers.Entrypoint
    for the 1st parameter of org.thp.thehive.connector.cortex.controllers.v0.AnalyzerCtrl.<init>(AnalyzerCtrl.scala:15)
  at org.thp.thehive.connector.cortex.controllers.v0.AnalyzerCtrl.class(AnalyzerCtrl.scala:14)
  while locating org.thp.thehive.connector.cortex.controllers.v0.AnalyzerCtrl
Caused by: java.lang.IllegalArgumentException (same stack trace as error #24)
49 errors
        at com.google.inject.internal.Errors.throwCreationExceptionIfErrorsExist(Errors.java:554)
        at com.google.inject.internal.InternalInjectorCreator.injectDynamically(InternalInjectorCreator.java:188)
        at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:111)
        at com.google.inject.Guice.createInjector(Guice.java:87)
        at com.google.inject.Guice.createInjector(Guice.java:78)
        at play.api.inject.guice.GuiceBuilder.injector(GuiceInjectorBuilder.scala:200)
        at play.api.inject.guice.GuiceApplicationBuilder.build(GuiceApplicationBuilder.scala:155)
        at play.api.inject.guice.GuiceApplicationLoader.load(GuiceApplicationLoader.scala:21)
        at play.core.server.ProdServerStart$.start(ProdServerStart.scala:54)
        at play.core.server.ProdServerStart$.main(ProdServerStart.scala:30)
        at play.core.server.ProdServerStart.main(ProdServerStart.scala)
[info] a.c.s.ClusterSingletonProxy [|] Singleton identified at [akka://application/user/flowSingletonManager/singleton]

[To-om]

Can you confirm that the server on which TheHive is installed can connect to Cassandra ? (check Firewall rules, SElinux/AppArmor policy).
Have you configured authentication or SSL for Cassandra ?

[mamoedo]

@To-om sorry, there wasn't connectivity, TheHive started before Cassandra was fully up, it happens sometimes in my setup.

So I did the testing that you asked me and TheHive is up and running correctly with a new keyspace. I can't see what could be wrong with my data then :( I just migrated from TheHive3 to TheHive4 using the tool. I think my data is too large to check in time for the configured timeouts.

[To-om]

The first start of TheHive (after a migration) can be a little bit long. It indexes data. I was supposing that 5 minutes is enough but if your data is large, you can get a timeout. I can make this timeout configurable.
I think that the debug log level is still disable in your setup. I don't know how you start TheHive but can you check that it is started with the parameter -Dlogger.file=/path/to/logback.xml ? (cf. previous comment). Then you should have more logs with more explaination.

[mamoedo]

@To-om in my setup it's a bit difficult to edit the logback.xml, because it isn't located in /etc/thehive/logback.xml but in /opt/thehive/conf/logback.xml instead (the same applies to logback-migration.xml, which is required to start the migration but it's not in /etc/thehive/ by default.) so I have to manually copy the file and edit it every time I deploy my environment. I enabled it in order to test what you asked me, but it's true that I don't have it enabled by default in my setup.

I was supposing that 5 minutes is enough but if your data is large, you can get a timeout. I can make this timeout configurable.

Thanks! It would help a lot to be able to configure it.