feat: start mgo with arg to use CA cert

[wallyworld]

Mongo 4.4.30 requires client connections be properly authenticated.
Some enablement work is need to make juju compatibile with this mongo release.
The mongo service needs to be started with a CA cert to use and client connections need to be configured with certificates.

We'll use the controller's CA cert when starting mongo.
For client connections, issue a new certificate off the CA cert. The client cert is valid for 15 minutes. Once the connection is created, it stays open. If a new connection is needed and the cert has expired, a new one is generated.

Also update the mongo args to use the new "tls" terminology.

Drive by: remove obsolete client pre-merge tests.

QA steps

juju bootstrap lxd test
juju switch controller
juju enable-ha
juju deploy ubuntu

You can exec into the controller and snap info juju-db shows 4.4.30 is installed.

NB new juju-db oci image still needs to be published - the test still uses the 4.4.24 oci image
but it the new args are compatible.
k8s smoke tests will fail unti new oci image is published

make microk8s-operator-update
juju bootstrap microk8s test
juju switch controller
juju deploy snappass-test

Links

Jira card: JUJU-

[SimonRichardson]

Code looks good, though I can't replicated the PR steps because when I attempt to bootstrap I get:

root@juju-92f03c-0:~# systemctl status snap.juju-db.*
● snap.juju-db.logrotate.timer - Timer logrotate for snap application juju-db.logrotate
     Loaded: loaded (/etc/systemd/system/snap.juju-db.logrotate.timer; disabled; vendor preset: enabled)
     Active: active (waiting) since Mon 2026-01-12 09:17:41 UTC; 3min 58s ago
    Trigger: Tue 2026-01-13 00:00:00 UTC; 14h left
   Triggers: ● snap.juju-db.logrotate.service

Jan 12 09:17:41 juju-92f03c-0 systemd[1]: Started Timer logrotate for snap application juju-db.logrotate.

● snap.juju-db.daemon.service - Service for snap application juju-db.daemon
     Loaded: loaded (/etc/systemd/system/snap.juju-db.daemon.service; enabled; vendor preset: enabled)
    Drop-In: /etc/systemd/system/snap.juju-db.daemon.service.d
             └─overrides.conf
     Active: failed (Result: exit-code) since Mon 2026-01-12 09:17:43 UTC; 3min 56s ago
    Process: 5060 ExecStart=/usr/bin/snap run juju-db.daemon (code=exited, status=14)
   Main PID: 5060 (code=exited, status=14)
        CPU: 175ms

Jan 12 09:17:43 juju-92f03c-0 systemd[1]: snap.juju-db.daemon.service: Scheduled restart job, restart counter is at 4.
Jan 12 09:17:43 juju-92f03c-0 systemd[1]: Stopped Service for snap application juju-db.daemon.
Jan 12 09:17:43 juju-92f03c-0 systemd[1]: snap.juju-db.daemon.service: Start request repeated too quickly.
Jan 12 09:17:43 juju-92f03c-0 systemd[1]: snap.juju-db.daemon.service: Failed with result 'exit-code'.
Jan 12 09:17:43 juju-92f03c-0 systemd[1]: Failed to start Service for snap application juju-db.daemon.

Here is the output from the bootstrap:

Installing Juju machine agent
2026-01-12 09:17:35 INFO juju.cmd supercommand.go:56 running jujud [2.9.54.1 2882471b366bfaabdd1f26fb04c5f2ecf7b58321 gc go1.25.5]
2026-01-12 09:17:35 INFO juju.agent identity.go:22 writing system identity file
2026-01-12 09:17:35 ERROR juju.mongo mongo.go:654 could not set the value of "/sys/kernel/mm/transparent_hugepage/defrag" to "never" because of: open /sys/kernel/mm/transparent_hugepage/defrag: permission denied
2026-01-12 09:17:35 ERROR juju.mongo mongo.go:654 could not set the value of "/proc/sys/net/core/netdev_max_backlog" to "1000" because of: "/proc/sys/net/core/netdev_max_backlog" does not exist, will not set "1000"
2026-01-12 09:17:35 ERROR juju.mongo mongo.go:654 could not set the value of "/sys/kernel/mm/transparent_hugepage/enabled" to "never" because of: open /sys/kernel/mm/transparent_hugepage/enabled: permission denied
2026-01-12 09:17:35 WARNING juju.mongo mongo.go:484 overwriting args.dataDir (set to /var/lib/juju) to /var/snap/juju-db/common
2026-01-12 09:17:35 INFO juju.mongo mongo.go:489 Ensuring mongo server is running; data directory /var/snap/juju-db/common; port 37017
2026-01-12 09:17:35 WARNING juju.mongo service.go:338 configuring mongod  with --noauth flag enabled
2026-01-12 09:17:35 INFO juju.packaging manager.go:103 installing "juju-db" via "snap"
2026-01-12 09:17:35 INFO juju.packaging.manager run.go:88 Running: snap install  --channel 4.4/candidate juju-db
2026-01-12 09:22:47 INFO juju.worker.peergrouper initiate.go:68 finished InitiateMongoServer
ERROR failed to start mongo: cannot initiate replica set: cannot dial mongo to initiate replicaset: no reachable servers
ERROR failed to bootstrap model: subprocess encountered error code 1

[wallyworld]

Code looks good, though I can't replicated the PR steps because when I attempt to bootstrap I get:

root@juju-92f03c-0:~# systemctl status snap.juju-db.*
● snap.juju-db.logrotate.timer - Timer logrotate for snap application juju-db.logrotate
     Loaded: loaded (/etc/systemd/system/snap.juju-db.logrotate.timer; disabled; vendor preset: enabled)
     Active: active (waiting) since Mon 2026-01-12 09:17:41 UTC; 3min 58s ago
    Trigger: Tue 2026-01-13 00:00:00 UTC; 14h left
   Triggers: ● snap.juju-db.logrotate.service

Jan 12 09:17:41 juju-92f03c-0 systemd[1]: Started Timer logrotate for snap application juju-db.logrotate.

● snap.juju-db.daemon.service - Service for snap application juju-db.daemon
     Loaded: loaded (/etc/systemd/system/snap.juju-db.daemon.service; enabled; vendor preset: enabled)
    Drop-In: /etc/systemd/system/snap.juju-db.daemon.service.d
             └─overrides.conf
     Active: failed (Result: exit-code) since Mon 2026-01-12 09:17:43 UTC; 3min 56s ago
    Process: 5060 ExecStart=/usr/bin/snap run juju-db.daemon (code=exited, status=14)
   Main PID: 5060 (code=exited, status=14)
        CPU: 175ms

Jan 12 09:17:43 juju-92f03c-0 systemd[1]: snap.juju-db.daemon.service: Scheduled restart job, restart counter is at 4.
Jan 12 09:17:43 juju-92f03c-0 systemd[1]: Stopped Service for snap application juju-db.daemon.
Jan 12 09:17:43 juju-92f03c-0 systemd[1]: snap.juju-db.daemon.service: Start request repeated too quickly.
Jan 12 09:17:43 juju-92f03c-0 systemd[1]: snap.juju-db.daemon.service: Failed with result 'exit-code'.
Jan 12 09:17:43 juju-92f03c-0 systemd[1]: Failed to start Service for snap application juju-db.daemon.

Here is the output from the bootstrap:

Installing Juju machine agent
2026-01-12 09:17:35 INFO juju.cmd supercommand.go:56 running jujud [2.9.54.1 2882471b366bfaabdd1f26fb04c5f2ecf7b58321 gc go1.25.5]
2026-01-12 09:17:35 INFO juju.agent identity.go:22 writing system identity file
2026-01-12 09:17:35 ERROR juju.mongo mongo.go:654 could not set the value of "/sys/kernel/mm/transparent_hugepage/defrag" to "never" because of: open /sys/kernel/mm/transparent_hugepage/defrag: permission denied
2026-01-12 09:17:35 ERROR juju.mongo mongo.go:654 could not set the value of "/proc/sys/net/core/netdev_max_backlog" to "1000" because of: "/proc/sys/net/core/netdev_max_backlog" does not exist, will not set "1000"
2026-01-12 09:17:35 ERROR juju.mongo mongo.go:654 could not set the value of "/sys/kernel/mm/transparent_hugepage/enabled" to "never" because of: open /sys/kernel/mm/transparent_hugepage/enabled: permission denied
2026-01-12 09:17:35 WARNING juju.mongo mongo.go:484 overwriting args.dataDir (set to /var/lib/juju) to /var/snap/juju-db/common
2026-01-12 09:17:35 INFO juju.mongo mongo.go:489 Ensuring mongo server is running; data directory /var/snap/juju-db/common; port 37017
2026-01-12 09:17:35 WARNING juju.mongo service.go:338 configuring mongod  with --noauth flag enabled
2026-01-12 09:17:35 INFO juju.packaging manager.go:103 installing "juju-db" via "snap"
2026-01-12 09:17:35 INFO juju.packaging.manager run.go:88 Running: snap install  --channel 4.4/candidate juju-db
2026-01-12 09:22:47 INFO juju.worker.peergrouper initiate.go:68 finished InitiateMongoServer
ERROR failed to start mongo: cannot initiate replica set: cannot dial mongo to initiate replicaset: no reachable servers
ERROR failed to bootstrap model: subprocess encountered error code 1

I just tested again with no issue.
Can you look at the content of /var/snap/juju-db/common/logs/mongodb.log?
Also, the content of /var/snap/juju-db/common/juju-db.config should be

# WARNING
# autogenerated by juju on 2026-01-12 23:06:20.51554226 +0000 UTC m=+1.695387613
# manual changes to this file are likely to be overwritten
bind_ip_all = true
auth = true
tlsAllowInvalidHostnames = true
oplogSize = 1024
tlsMode = requireTLS
tlsCertificateKeyFile = /var/snap/juju-db/common/server.pem
tlsCertificateKeyFilePassword=ignored = true
port = 37017
tlsCAFile = /var/snap/juju-db/common/ca.crt
storageEngine = wiredTiger
replSet = juju
dbpath = /var/snap/juju-db/common/db
logpath = /var/snap/juju-db/common/logs/mongodb.log
keyFile = /var/snap/juju-db/common/shared-secret
journal = true
slowms = 1000
quiet = true
ipv6 = truer

[wallyworld]

/build