Running HA ArcadeDB in Kubernetes

[MrHassanMurtaza]

ArcadeDB Version: 22.8.1

JDK Version:

OS: Openshift on RH Linux 4.8

Expected behavior

Being able to deploy highly available statefulset on Kubernetes. I don't find a lot of information in the documentation itself:

Actual behavior

I'm seeing this error in each pod:

 █████╗ ██████╗  ██████╗ █████╗ ██████╗ ███████╗██████╗ ██████╗
██╔══██╗██╔══██╗██╔════╝██╔══██╗██╔══██╗██╔════╝██╔══██╗██╔══██╗
███████║██████╔╝██║     ███████║██║  ██║█████╗  ██║  ██║██████╔╝
██╔══██║██╔══██╗██║     ██╔══██║██║  ██║██╔══╝  ██║  ██║██╔══██╗
██║  ██║██║  ██║╚██████╗██║  ██║██████╔╝███████╗██████╔╝██████╔╝
╚═╝  ╚═╝╚═╝  ╚═╝ ╚═════╝╚═╝  ╚═╝╚═════╝ ╚══════╝╚═════╝ ╚═════╝
PLAY WITH DATA                                    arcadedb.com
./server.sh: line 82: can't create /home/arcadedb/bin/arcadedb.pid: Permission denied
Can't load log handler "java.util.logging.FileHandler"
java.nio.file.NoSuchFileException: log/arcadedb.log.0.lck
java.nio.file.NoSuchFileException: log/arcadedb.log.0.lck
        at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)
        at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
        at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116)
        at java.base/sun.nio.fs.UnixFileSystemProvider.newFileChannel(UnixFileSystemProvider.java:182)
        at java.base/java.nio.channels.FileChannel.open(FileChannel.java:292)
        at java.base/java.nio.channels.FileChannel.open(FileChannel.java:345)
        at java.logging/java.util.logging.FileHandler.openFiles(FileHandler.java:512)
        at java.logging/java.util.logging.FileHandler.<init>(FileHandler.java:279)
        at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
        at java.base/java.lang.Class.newInstance(Class.java:584)
        at java.logging/java.util.logging.LogManager.createLoggerHandlers(LogManager.java:1000)
        at java.logging/java.util.logging.LogManager$4.run(LogManager.java:970)
        at java.logging/java.util.logging.LogManager$4.run(LogManager.java:966)
        at java.base/java.security.AccessController.doPrivileged(Native Method)
        at java.logging/java.util.logging.LogManager.loadLoggerHandlers(LogManager.java:966)
        at java.logging/java.util.logging.LogManager.initializeGlobalHandlers(LogManager.java:2417)
        at java.logging/java.util.logging.LogManager$RootLogger.accessCheckedHandlers(LogManager.java:2511)
        at java.logging/java.util.logging.Logger.getHandlers(Logger.java:2089)
        at com.arcadedb.log.DefaultLogger.installCustomFormatter(DefaultLogger.java:79)
        at com.arcadedb.log.DefaultLogger.<init>(DefaultLogger.java:41)
        at com.arcadedb.log.DefaultLogger.<clinit>(DefaultLogger.java:36)
        at com.arcadedb.log.LogManager.<init>(LogManager.java:31)
        at com.arcadedb.log.LogManager.<clinit>(LogManager.java:29)
        at com.arcadedb.server.ArcadeDBServer.start(ArcadeDBServer.java:98)
        at com.arcadedb.server.ArcadeDBServer.main(ArcadeDBServer.java:90)
Can't load log handler "java.util.logging.FileHandler"
java.nio.file.NoSuchFileException: log/arcadedb.log.0.lck
java.nio.file.NoSuchFileException: log/arcadedb.log.0.lck
        at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)
        at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
        at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116)
        at java.base/sun.nio.fs.UnixFileSystemProvider.newFileChannel(UnixFileSystemProvider.java:182)
        at java.base/java.nio.channels.FileChannel.open(FileChannel.java:292)
        at java.base/java.nio.channels.FileChannel.open(FileChannel.java:345)
        at java.logging/java.util.logging.FileHandler.openFiles(FileHandler.java:512)
        at java.logging/java.util.logging.FileHandler.<init>(FileHandler.java:279)
        at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
        at java.base/java.lang.Class.newInstance(Class.java:584)
        at java.logging/java.util.logging.LogManager.createLoggerHandlers(LogManager.java:1000)
        at java.logging/java.util.logging.LogManager$4.run(LogManager.java:970)
        at java.logging/java.util.logging.LogManager$4.run(LogManager.java:966)
        at java.base/java.security.AccessController.doPrivileged(Native Method)
        at java.logging/java.util.logging.LogManager.loadLoggerHandlers(LogManager.java:966)
        at java.logging/java.util.logging.LogManager.initializeGlobalHandlers(LogManager.java:2417)
        at java.logging/java.util.logging.LogManager$RootLogger.accessCheckedHandlers(LogManager.java:2511)
        at java.logging/java.util.logging.Logger.getHandlers(Logger.java:2089)
        at com.arcadedb.log.DefaultLogger.installCustomFormatter(DefaultLogger.java:79)
        at com.arcadedb.log.DefaultLogger.<init>(DefaultLogger.java:41)
        at com.arcadedb.log.LogManager.<init>(LogManager.java:31)
        at com.arcadedb.log.LogManager.<clinit>(LogManager.java:29)
        at com.arcadedb.server.ArcadeDBServer.start(ArcadeDBServer.java:98)
        at com.arcadedb.server.ArcadeDBServer.main(ArcadeDBServer.java:90)

2022-12-23 12:27:22.249 INFO  [ArcadeDBServer] <ArcadeDB_0> ArcadeDB Server v22.8.1 (build b63f0402a9206830c3ec100766302c18038c8f17/1659468151386/main) is starting up...
2022-12-23 12:27:22.250 INFO  [ArcadeDBServer] <ArcadeDB_0> Starting ArcadeDB Server with plugins [] ...
2022-12-23 12:27:22.268 INFO  [ArcadeDBServer] <ArcadeDB_0> - JMX Metrics Started...
2022-12-23 12:27:22.323 INFO  [ServerSecurity] <ArcadeDB_0> Creating root user with the provided password<ArcadeDB_0> Error on saving security configuration to file 'server-users.jsonl'
java.io.FileNotFoundException: ../config/server-users.jsonl (Permission denied)
        at java.base/java.io.FileOutputStream.open0(Native Method)
        at java.base/java.io.FileOutputStream.open(FileOutputStream.java:298)
        at java.base/java.io.FileOutputStream.<init>(FileOutputStream.java:237)
        at java.base/java.io.FileOutputStream.<init>(FileOutputStream.java:187)
        at java.base/java.io.FileWriter.<init>(FileWriter.java:176)
        at com.arcadedb.server.security.SecurityUserFileRepository.save(SecurityUserFileRepository.java:48)
        at com.arcadedb.server.security.ServerSecurity.saveUsers(ServerSecurity.java:310)
        at com.arcadedb.server.security.ServerSecurity.askForRootPassword(ServerSecurity.java:419)
        at com.arcadedb.server.security.ServerSecurity.loadUsers(ServerSecurity.java:126)
        at com.arcadedb.server.ArcadeDBServer.start(ArcadeDBServer.java:127)
        at com.arcadedb.server.ArcadeDBServer.main(ArcadeDBServer.java:90)

2022-12-23 12:27:22.653 INFO  [HttpServer] <ArcadeDB_0> - Starting HTTP Server (host=0.0.0.0 port=2480-2489)...
2022-12-23 12:27:22.711 INFO  [undertow] starting server: Undertow - 2.2.18.Final
2022-12-23 12:27:22.722 INFO  [xnio] XNIO version 3.8.7.Final
2022-12-23 12:27:22.735 INFO  [nio] XNIO NIO Implementation Version 3.8.7.Final
2022-12-23 12:27:22.767 INFO  [threads] JBoss Threads version 3.1.0.Final
2022-12-23 12:27:22.812 INFO  [HttpServer] <ArcadeDB_0> - HTTP Server started (host=0.0.0.0 port=2480)Exception in thread "main" java.lang.NullPointerException
        at java.base/java.util.Objects.requireNonNull(Objects.java:221)
        at java.base/java.util.Arrays$ArrayList.<init>(Arrays.java:4323)
        at java.base/java.util.Arrays.asList(Arrays.java:4310)
        at com.arcadedb.server.ha.ReplicationLogFile.openLastFileChunk(ReplicationLogFile.java:425)
        at com.arcadedb.server.ha.ReplicationLogFile.<init>(ReplicationLogFile.java:95)
        at com.arcadedb.server.ha.HAServer.startService(HAServer.java:152)
        at com.arcadedb.server.ArcadeDBServer.start(ArcadeDBServer.java:139)
        at com.arcadedb.server.ArcadeDBServer.main(ArcadeDBServer.java:90)

Steps to reproduce

I'm deploying the following kubernetes manifest:

---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: arcadedb
spec:
  # serviceName: "nginx"
  replicas: 3
  selector:
    matchLabels:
      app: arcadedb
  template:
    metadata:
      labels:
        app: arcadedb
    spec:
      containers:
      - name: arcadedb
        image: arcadedata/arcadedb:22.8.1
        ports:
        - containerPort: 2480
        env: 
          - name: JAVA_OPTS
            value: "-Darcadedb.server.rootPassword=playwithdata -Darcadedb.ha.enabled=true -Darcadedb.ha.k8s=true -Darcadedb.ha.serverList=arcadedb-0,arcadedb-1,arcadedb-2"

<An SQL script to reproduce the problem or a JUnit test case will increase A LOT the chance to have a quick fix>

[lvca]

Why aren't using the default kubernates config?

Look at:

arcadedb/package/src/main/config/arcadedb-statefulset.yaml

Line 82 in 91efc9d

podAffinityTerm:

[MrHassanMurtaza]

Why aren't using the default kubernates config?

Look at:

arcadedb/package/src/main/config/arcadedb-statefulset.yaml

Line 82 in 91efc9d

podAffinityTerm:

Thanks for pointing me to this! I couldn't find it on the documentation. If you like, I can create a PR to point it to config -- it's not very obvious from documentation where the config is rn.

[lvca]

It would be awesome if you could contrib to the docs! Thanks in advance

[MrHassanMurtaza]

great, I see another issue. It seems like we can't pass rootPassword as an env in kubernetes deployment:

➜  arcadedb git:(main) ✗ oc logs -f arcadedb-0

 █████╗ ██████╗  ██████╗ █████╗ ██████╗ ███████╗██████╗ ██████╗
██╔══██╗██╔══██╗██╔════╝██╔══██╗██╔══██╗██╔════╝██╔══██╗██╔══██╗
███████║██████╔╝██║     ███████║██║  ██║█████╗  ██║  ██║██████╔╝
██╔══██║██╔══██╗██║     ██╔══██║██║  ██║██╔══╝  ██║  ██║██╔══██╗
██║  ██║██║  ██║╚██████╗██║  ██║██████╔╝███████╗██████╔╝██████╔╝
╚═╝  ╚═╝╚═╝  ╚═╝ ╚═════╝╚═╝  ╚═╝╚═════╝ ╚══════╝╚═════╝ ╚═════╝
PLAY WITH DATA                                    arcadedb.com

2022-12-23 13:47:38.440 WARNI [ByteArrayOutputStream] ARCADEDB 22.12.1-SNAPSHOT configuration:
- ARCADEDB
  + arcadedb.dumpConfigAtStartup = true
  + arcadedb.dumpMetricsEvery = 0
  + arcadedb.profile = default
  + arcadedb.test = false
  + arcadedb.maxPageRAM = 4096
  + arcadedb.initialPageCacheSize = 65535
  + arcadedb.flushOnlyAtClose = false
  + arcadedb.txWAL = true
  + arcadedb.txWalFlush = 0
  + arcadedb.freePageRAM = 50
  + arcadedb.typeDefaultBuckets = 8
  + arcadedb.bucketDefaultPageSize = 65536
  + arcadedb.asyncWorkerThreads = 15
  + arcadedb.asyncOperationsQueueImpl = standard
  + arcadedb.asyncOperationsQueueSize = 1024
  + arcadedb.asyncTxBatchSize = 10240
  + arcadedb.pageFlushQueue = 512
  + arcadedb.commitLockTimeout = 5000
  + arcadedb.txRetries = 3
  + arcadedb.sqlStatementCache = 300
  + arcadedb.command.timeout = 0
  + arcadedb.queryMaxHeapElementsAllowedPerOp = 500000
  + arcadedb.indexCompactionRAM = 300
  + arcadedb.indexCompactionMinPagesSchedule = 10
  + arcadedb.network.socketBufferSize = 0
  + arcadedb.network.socketTimeout = 30000
  + arcadedb.ssl.enabled = false
  + arcadedb.ssl.keyStore = null
  + arcadedb.ssl.keyStorePass = null
  + arcadedb.ssl.trustStore = null
  + arcadedb.ssl.trustStorePass = null
  + arcadedb.server.name = ArcadeDB_0
  + arcadedb.server.rootPassword = null
  + arcadedb.server.mode = development
  + arcadedb.serverMetrics = true
  + arcadedb.server.rootPath = null
  + arcadedb.server.databaseDirectory = ${arcadedb.server.rootPath}/databases
  + arcadedb.server.databaseLoadAtStartup = true
  + arcadedb.server.defaultDatabases = 
  + arcadedb.server.defaultDatabaseMode = READ_WRITE
  + arcadedb.server.plugins = 
  + arcadedb.server.httpIncomingHost = 0.0.0.0
  + arcadedb.server.httpIncomingPort = 2480-2489
  + arcadedb.server.httpTxExpireTimeout = 30
  + arcadedb.server.eventBusQueueSize = 1000
  + arcadedb.server.securityAlgorithm = PBKDF2WithHmacSHA256
  + arcadedb.server.securitySaltCacheSize = 64
  + arcadedb.server.saltIterations = 65536
  + arcadedb.ha.enabled = false
  + arcadedb.ha.clusterName = arcadedb
  + arcadedb.ha.serverList = 
  + arcadedb.ha.quorum = MAJORITY
  + arcadedb.ha.quorumTimeout = 10000
  + arcadedb.ha.replicationQueueSize = 512
  + arcadedb.ha.replicationFileMaxSize = 1073741824
  + arcadedb.ha.replicationChunkMaxSize = 16777216
  + arcadedb.ha.replicationIncomingHost = 0.0.0.0
  + arcadedb.ha.replicationIncomingPorts = 2424-2433
  + arcadedb.ha.k8s = false
  + arcadedb.ha.k8sSuffix = 
  + arcadedb.cypher.statementCache = 1000
  + arcadedb.postgres.port = 5432
  + arcadedb.postgres.host = 0.0.0.0
  + arcadedb.postgres.debug = false
  + arcadedb.redis.port = 6379
  + arcadedb.redis.host = 0.0.0.0

2022-12-23 13:47:38.453 INFO  [ArcadeDBServer] <arcadedb-0> ArcadeDB Server v22.12.1-SNAPSHOT (build 997390890ab4cf0893cd80299a498a80948873fd/1671758873204/main) is starting up...
2022-12-23 13:47:38.454 INFO  [ArcadeDBServer] <arcadedb-0> Starting ArcadeDB Server with plugins [] ...
2022-12-23 13:47:38.475 INFO  [ArcadeDBServer] <arcadedb-0> - JMX Metrics Started...
2022-12-23 13:47:38.516 SEVER [ServerSecurity] <arcadedb-0> Unable to start a server under Kubernetes if the environment variable `arcadedb.server.rootPassword` is not setException in thread "main" com.arcadedb.server.security.ServerSecurityException: Unable to start a server under Kubernetes if the environment variable `arcadedb.server.rootPassword` is not set
        at com.arcadedb.server.security.ServerSecurity.askForRootPassword(ServerSecurity.java:337)
        at com.arcadedb.server.security.ServerSecurity.loadUsers(ServerSecurity.java:126)
        at com.arcadedb.server.ArcadeDBServer.start(ArcadeDBServer.java:128)
        at com.arcadedb.server.ArcadeDBServer.main(ArcadeDBServer.java:91)

seeing this error - even though we have

            - name: arcadedb.server.rootPassword
              valueFrom:
                secretKeyRef:
                  name: server-root-password
                  key: arcadedb.server.rootPassword

in place. Currently I've to pass it explicitly through cmd: `- "-Darcadedb.server.rootPassword=xxxxxx":

[gramian]

I think, the issue #597 is related, as passing the secret via file(path) should resolve this, right @MrHassanMurtaza ?

[MrHassanMurtaza]

@gramian IMHO usually passing the secret as the environment is considered practical as compared to files but we can have options for both.