upgrade form 2023.10.4 to 2024.4. Containers server and worker restart

[Julien-Quidam]

Describe the bug
I upgrade images the server and the worker
ghcr.io/goauthentik/server:2023.10.4 to ghcr.io/goauthentik/server:latest

Containers server and worker restart again and again

To Reproduce
1 - stop all containers
authentik-worker-1
authentik-server-1
authentik-redis-1
authentik-postgresql-1

2 - change image on authentik-worker-1 and authentik-server-1

3 - restart all containers

Ok :
authentik-redis-1
authentik-postgresql-1

Nok:
authentik-worker-1
authentik-server-1

Containers server and the worker restart again and again. No error in log.

Expected behavior
Containers server and worker start

Logs

{"event": "Loaded config", "level": "debug", "logger": "authentik.lib.config", "timestamp": 1715607634.2827733, "file": "/authentik/lib/default.yml"}
{"event": "Loaded environment variables", "level": "debug", "logger": "authentik.lib.config", "timestamp": 1715607634.283426, "count": 9}
{"event": "Starting authentik bootstrap", "level": "info", "logger": "authentik.lib.config", "timestamp": 1715607634.7895787}
{"event": "PostgreSQL connection successful", "level": "info", "logger": "authentik.lib.config", "timestamp": 1715607634.813482}
{"event": "Redis Connection successful", "level": "info", "logger": "authentik.lib.config", "timestamp": 1715607634.8301268}
{"event": "Finished authentik bootstrap", "level": "info", "logger": "authentik.lib.config", "timestamp": 1715607634.8303654}
{"event": "Booting authentik", "level": "info", "logger": "authentik.lib.config", "timestamp": 1715607638.2763298, "version": "2024.4.1"}
{"event": "Enabled authentik enterprise", "level": "info", "logger": "authentik.lib.config", "timestamp": 1715607638.3674917}
{"event": "Loaded app settings", "level": "debug", "logger": "authentik.lib.config", "timestamp": 1715607638.3692343, "path": "authentik.enterprise.settings"}
{"event": "Loaded app settings", "level": "debug", "logger": "authentik.lib.config", "timestamp": 1715607638.3716676, "path": "authentik.admin.settings"}
{"event": "Loaded app settings", "level": "debug", "logger": "authentik.lib.config", "timestamp": 1715607638.3725333, "path": "authentik.crypto.settings"}
{"event": "Loaded app settings", "level": "debug", "logger": "authentik.lib.config", "timestamp": 1715607638.37624, "path": "authentik.providers.scim.settings"}
{"event": "Loaded app settings", "level": "debug", "logger": "authentik.lib.config", "timestamp": 1715607638.378682, "path": "authentik.stages.authenticator_totp.settings"}
{"event": "Loaded app settings", "level": "debug", "logger": "authentik.lib.config", "timestamp": 1715607638.3818927, "path": "authentik.sources.ldap.settings"}
{"event": "Loaded app settings", "level": "debug", "logger": "authentik.lib.config", "timestamp": 1715607638.382671, "path": "authentik.events.settings"}
{"event": "Loaded app settings", "level": "debug", "logger": "authentik.lib.config", "timestamp": 1715607638.3830607, "path": "authentik.enterprise.settings"}
{"event": "Loaded app settings", "level": "debug", "logger": "authentik.lib.config", "timestamp": 1715607638.3843555, "path": "authentik.blueprints.settings"}
{"event": "Loaded app settings", "level": "debug", "logger": "authentik.lib.config", "timestamp": 1715607638.3850732, "path": "authentik.sources.oauth.settings"}
{"event": "Loaded app settings", "level": "debug", "logger": "authentik.lib.config", "timestamp": 1715607638.3878953, "path": "authentik.outposts.settings"}
{"event": "Loaded app settings", "level": "debug", "logger": "authentik.lib.config", "timestamp": 1715607638.390013, "path": "authentik.sources.plex.settings"}
{"event": "Loaded app settings", "level": "debug", "logger": "authentik.lib.config", "timestamp": 1715607638.3927524, "path": "authentik.policies.reputation.settings"}
{"domain_url": null, "event": "Loaded MMDB database", "file": "/geoip/GeoLite2-ASN.mmdb", "last_write": 1714147348.0, "level": "info", "logger": "authentik.events.context_processors.mmdb", "pid": 7, "schema_name": "public", "timestamp": "2024-05-13T13:40:39.885142"}
{"domain_url": null, "event": "Loaded MMDB database", "file": "/geoip/GeoLite2-City.mmdb", "last_write": 1714147347.0, "level": "info", "logger": "authentik.events.context_processors.mmdb", "pid": 7, "schema_name": "public", "timestamp": "2024-05-13T13:40:39.888174"}
{"app_name": "authentik.tenants", "domain_url": null, "event": "Imported related module", "level": "info", "logger": "authentik.blueprints.apps", "module": "authentik.tenants.checks", "pid": 7, "schema_name": "public", "timestamp": "2024-05-13T13:40:42.346837"}
{"app_name": "authentik.tenants", "domain_url": null, "event": "Imported related module", "level": "info", "logger": "authentik.blueprints.apps", "module": "authentik.tenants.signals", "pid": 7, "schema_name": "public", "timestamp": "2024-05-13T13:40:42.347727"}
{"app_name": "authentik.admin", "domain_url": null, "event": "Imported related module", "level": "info", "logger": "authentik.blueprints.apps", "module": "authentik.admin.tasks", "pid": 7, "schema_name": "public", "timestamp": "2024-05-13T13:40:42.449257"}
{"app_name": "authentik.admin", "domain_url": null, "event": "Imported related module", "level": "info", "logger": "authentik.blueprints.apps", "module": "authentik.admin.signals", "pid": 7, "schema_name": "public", "timestamp": "2024-05-13T13:40:42.450194"}
{"app_name": "authentik.crypto", "domain_url": null, "event": "Imported related module", "level": "info", "logger": "authentik.blueprints.apps", "module": "authentik.crypto.tasks", "pid": 7, "schema_name": "public", "timestamp": "2024-05-13T13:40:42.455979"}
{"app_name": "authentik.flows", "domain_url": null, "event": "Imported related module", "level": "info", "logger": "authentik.blueprints.apps", "module": "authentik.flows.signals", "pid": 7, "schema_name": "public", "timestamp": "2024-05-13T13:40:42.525525"}
{"app_name": "authentik.outposts", "domain_url": null, "event": "Imported related module", "level": "info", "logger": "authentik.blueprints.apps", "module": "authentik.outposts.tasks", "pid": 7, "schema_name": "public", "timestamp": "2024-05-13T13:40:42.579616"}
{"app_name": "authentik.outposts", "domain_url": null, "event": "Imported related module", "level": "info", "logger": "authentik.blueprints.apps", "module": "authentik.outposts.signals", "pid": 7, "schema_name": "public", "timestamp": "2024-05-13T13:40:42.581382"}
{"app_name": "authentik.policies.reputation", "domain_url": null, "event": "Imported related module", "level": "info", "logger": "authentik.blueprints.apps", "module": "authentik.policies.reputation.tasks", "pid": 7, "schema_name": "public", "timestamp": "2024-05-13T13:40:42.582738"}
{"app_name": "authentik.policies.reputation", "domain_url": null, "event": "Imported related module", "level": "info", "logger": "authentik.blueprints.apps", "module": "authentik.policies.reputation.signals", "pid": 7, "schema_name": "public", "timestamp": "2024-05-13T13:40:42.584056"}
{"app_name": "authentik.policies", "domain_url": null, "event": "Imported related module", "level": "info", "logger": "authentik.blueprints.apps", "module": "authentik.policies.signals", "pid": 7, "schema_name": "public", "timestamp": "2024-05-13T13:40:42.602606"}
{"app_name": "authentik.providers.proxy", "domain_url": null, "event": "Imported related module", "level": "info", "logger": "authentik.blueprints.apps", "module": "authentik.providers.proxy.tasks", "pid": 7, "schema_name": "public", "timestamp": "2024-05-13T13:40:42.603854"}
{"app_name": "authentik.providers.proxy", "domain_url": null, "event": "Imported related module", "level": "info", "logger": "authentik.blueprints.apps", "module": "authentik.providers.proxy.signals", "pid": 7, "schema_name": "public", "timestamp": "2024-05-13T13:40:42.604647"}
{"app_name": "authentik.providers.scim", "domain_url": null, "event": "Imported related module", "level": "info", "logger": "authentik.blueprints.apps", "module": "authentik.providers.scim.tasks", "pid": 7, "schema_name": "public", "timestamp": "2024-05-13T13:40:43.044267"}
{"app_name": "authentik.providers.scim", "domain_url": null, "event": "Imported related module", "level": "info", "logger": "authentik.blueprints.apps", "module": "authentik.providers.scim.signals", "pid": 7, "schema_name": "public", "timestamp": "2024-05-13T13:40:43.045852"}
{"app_name": "authentik.rbac", "domain_url": null, "event": "Imported related module", "level": "info", "logger": "authentik.blueprints.apps", "module": "authentik.rbac.signals", "pid": 7, "schema_name": "public", "timestamp": "2024-05-13T13:40:43.047206"}
{"app_name": "authentik.sources.ldap", "domain_url": null, "event": "Imported related module", "level": "info", "logger": "authentik.blueprints.apps", "module": "authentik.sources.ldap.tasks", "pid": 7, "schema_name": "public", "timestamp": "2024-05-13T13:40:43.060057"}
{"app_name": "authentik.sources.ldap", "domain_url": null, "event": "Imported related module", "level": "info", "logger": "authentik.blueprints.apps", "module": "authentik.sources.ldap.signals", "pid": 7, "schema_name": "public", "timestamp": "2024-05-13T13:40:43.063793"}
{"app_name": "authentik.sources.oauth", "domain_url": null, "event": "Imported related module", "level": "info", "logger": "authentik.blueprints.apps", "module": "authentik.sources.oauth.tasks", "pid": 7, "schema_name": "public", "timestamp": "2024-05-13T13:40:43.078878"}
{"app_name": "authentik.sources.saml", "domain_url": null, "event": "Imported related module", "level": "info", "logger": "authentik.blueprints.apps", "module": "authentik.sources.saml.signals", "pid": 7, "schema_name": "public", "timestamp": "2024-05-13T13:40:43.079887"}
{"app_name": "authentik.sources.scim", "domain_url": null, "event": "Imported related module", "level": "info", "logger": "authentik.blueprints.apps", "module": "authentik.sources.scim.signals", "pid": 7, "schema_name": "public", "timestamp": "2024-05-13T13:40:43.080847"}
{"app_name": "authentik.stages.authenticator_duo", "domain_url": null, "event": "Imported related module", "level": "info", "logger": "authentik.blueprints.apps", "module": "authentik.stages.authenticator_duo.tasks", "pid": 7, "schema_name": "public", "timestamp": "2024-05-13T13:40:43.081868"}
{"app_name": "authentik.stages.authenticator_static", "domain_url": null, "event": "Imported related module", "level": "info", "logger": "authentik.blueprints.apps", "module": "authentik.stages.authenticator_static.signals", "pid": 7, "schema_name": "public", "timestamp": "2024-05-13T13:40:43.082638"}
{"app_name": "authentik.stages.authenticator_webauthn", "domain_url": null, "event": "Imported related module", "level": "info", "logger": "authentik.blueprints.apps", "module": "authentik.stages.authenticator_webauthn.tasks", "pid": 7, "schema_name": "public", "timestamp": "2024-05-13T13:40:43.150745"}
{"app_name": "authentik.stages.email", "domain_url": null, "event": "Imported related module", "level": "info", "logger": "authentik.blueprints.apps", "module": "authentik.stages.email.tasks", "pid": 7, "schema_name": "public", "timestamp": "2024-05-13T13:40:43.154305"}
{"app_name": "authentik.core", "domain_url": null, "event": "Imported related module", "level": "info", "logger": "authentik.blueprints.apps", "module": "authentik.core.tasks", "pid": 7, "schema_name": "public", "timestamp": "2024-05-13T13:40:43.157033"}
{"app_name": "authentik.core", "domain_url": null, "event": "Imported related module", "level": "info", "logger": "authentik.blueprints.apps", "module": "authentik.core.signals", "pid": 7, "schema_name": "public", "timestamp": "2024-05-13T13:40:43.157678"}
{"app_name": "authentik.enterprise", "domain_url": null, "event": "Imported related module", "level": "info", "logger": "authentik.blueprints.apps", "module": "authentik.enterprise.tasks", "pid": 7, "schema_name": "public", "timestamp": "2024-05-13T13:40:43.159569"}
{"app_name": "authentik.enterprise", "domain_url": null, "event": "Imported related module", "level": "info", "logger": "authentik.blueprints.apps", "module": "authentik.enterprise.signals", "pid": 7, "schema_name": "public", "timestamp": "2024-05-13T13:40:43.160965"}
{"app_name": "authentik.enterprise.providers.rac", "domain_url": null, "event": "Imported related module", "level": "info", "logger": "authentik.blueprints.apps", "module": "authentik.enterprise.providers.rac.signals", "pid": 7, "schema_name": "public", "timestamp": "2024-05-13T13:40:43.183413"}
{"app_name": "authentik.events", "domain_url": null, "event": "Imported related module", "level": "info", "logger": "authentik.blueprints.apps", "module": "authentik.events.tasks", "pid": 7, "schema_name": "public", "timestamp": "2024-05-13T13:40:43.184416"}
{"app_name": "authentik.events", "domain_url": null, "event": "Imported related module", "level": "info", "logger": "authentik.blueprints.apps", "module": "authentik.events.signals", "pid": 7, "schema_name": "public", "timestamp": "2024-05-13T13:40:43.184704"}

Type 'manage.py help <subcommand>' for help on a specific subcommand.

Available subcommands:

[auth]
    changepassword
    createsuperuser

[authenticator_webauthn]
    update_webauthn_mds

[blueprints]
    apply_blueprint
    export_blueprint
    make_blueprint_schema

[channels]
    runworker

[contenttypes]
    remove_stale_contenttypes

[core]
    bootstrap_tasks
    build_source_docs
    dev_server
    repair_permissions
    shell
    worker

[crypto]
    import_certificate

[daphne]
    runserver

[django]
    check
    compilemessages
    createcachetable
    dbshell
    diffsettings
    dumpdata
    flush
    inspectdb
    loaddata
    makemessages
    makemigrations
    optimizemigration
    sendtestemail
    showmigrations
    sqlflush
    sqlmigrate
    sqlsequencereset
    squashmigrations
    startapp
    startproject
    test
    testserver

[django_tenants]
    all_tenants_command
    clone_tenant
    collectstatic_schemas
    create_missing_schemas
    create_tenant
    create_tenant_superuser
    delete_tenant
    migrate
    migrate_schemas
    rename_schema
    tenant_command

[drf_spectacular]
    spectacular

[email]
    test_email

[flows]
    benchmark

[guardian]
    clean_orphan_obj_perms

[ldap]
    ldap_check_connection
    ldap_sync

[recovery]
    create_admin_group
    create_recovery_key

[rest_framework]
    generateschema

[scim]
    scim_sync

[sessions]
    clearsessions

[staticfiles]
    collectstatic
    findstatic
 *  Terminal will be reused by tasks, press any key to close it.

Version and Deployment (please complete the following information):

• authentik version: 2024.4
• Deployment: docker-compose

[depuits]

I'm having a similar issue when trying to upgrade from 2024.4.1 to 2024.4.2.

I do see following error in the worker log

DBG event=Loaded app settings logger=authentik.lib.config timestamp=1715538755.9956036 path=authentik.sources.plex.settings
DBG event=Loaded app settings logger=authentik.lib.config timestamp=1715538755.9970844 path=authentik.providers.scim.settings
DBG event=Loaded app settings logger=authentik.lib.config timestamp=1715538755.999732 path=authentik.crypto.settings
/ak-root/venv/lib/python3.12/site-packages/opencontainers/distribution/reggie/defaults.py:17: SyntaxWarning: invalid escape sequence '\('"http[s]?://(?:[a-zA-Z]|[0-9]|[$-_@.&+]|[!*\(\),]|(?:%[0-9a-fA-F][0-9a-fA-F]))+"
_runit-log:x:998:authentik

Reverting back to version 2024.4.1 does fix the problem.

[FaykoB]

I'm having a similar issue when trying to upgrade from 2024.4.1 to 2024.4.2.

I do see following error in the worker log

DBG event=Loaded app settings logger=authentik.lib.config timestamp=1715538755.9956036 path=authentik.sources.plex.settings
DBG event=Loaded app settings logger=authentik.lib.config timestamp=1715538755.9970844 path=authentik.providers.scim.settings
DBG event=Loaded app settings logger=authentik.lib.config timestamp=1715538755.999732 path=authentik.crypto.settings
/ak-root/venv/lib/python3.12/site-packages/opencontainers/distribution/reggie/defaults.py:17: SyntaxWarning: invalid escape sequence '\('"http[s]?://(?:[a-zA-Z]|[0-9]|[$-_@.&+]|[!*\(\),]|(?:%[0-9a-fA-F][0-9a-fA-F]))+"
_runit-log:x:998:authentik

Reverting back to version 2024.4.1 does fix the problem.

I removed redis,postgres, and both authentik containers then reinstalled the first 2 with the latest tags and the authentik worker with 2024.4.1 and I'm still getting an escape sequence failure. Should i remove everything and go even earlier?

[bnounours]

Hello,
I have the same with the migration from 2024.4.1 to 2024.4.2. In fact the escape error is not the root cause there is the same warning in 2024.4.1.
But I connected inside the container with a docker exec -it <name of server container> bash
I tried to use `manage.py script and got a core dump

root@656567b6e290:/# export AUTHENTIK_LOG_LEVEL=trace
root@656567b6e290:/# ./manage.py 
{"event": "Loaded config", "level": "debug", "logger": "authentik.lib.config", "timestamp": 1717231859.026418, "file": "/authentik/lib/default.yml"}
{"event": "Loaded environment variables", "level": "debug", "logger": "authentik.lib.config", "timestamp": 1717231859.027352, "count": 15}
{"event": "Starting authentik bootstrap", "level": "info", "logger": "authentik.lib.config", "timestamp": 1717231860.2170274}
{"event": "PostgreSQL connection successful", "level": "info", "logger": "authentik.lib.config", "timestamp": 1717231860.234503}
{"event": "Redis Connection successful", "level": "info", "logger": "authentik.lib.config", "timestamp": 1717231860.237499}
{"event": "Finished authentik bootstrap", "level": "info", "logger": "authentik.lib.config", "timestamp": 1717231860.2379699}
{"event": "Booting authentik", "level": "info", "logger": "authentik.lib.config", "timestamp": 1717231866.3635058, "version": "2024.4.2"}
{"event": "Enabled authentik enterprise", "level": "info", "logger": "authentik.lib.config", "timestamp": 1717231866.5071452}
{"event": "Loaded app settings", "level": "debug", "logger": "authentik.lib.config", "timestamp": 1717231866.5095181, "path": "authentik.enterprise.settings"}
{"event": "Loaded app settings", "level": "debug", "logger": "authentik.lib.config", "timestamp": 1717231866.511984, "path": "authentik.outposts.settings"}
{"event": "Loaded app settings", "level": "debug", "logger": "authentik.lib.config", "timestamp": 1717231866.5135534, "path": "authentik.sources.plex.settings"}
{"event": "Loaded app settings", "level": "debug", "logger": "authentik.lib.config", "timestamp": 1717231866.5165665, "path": "authentik.admin.settings"}
{"event": "Loaded app settings", "level": "debug", "logger": "authentik.lib.config", "timestamp": 1717231866.5177824, "path": "authentik.policies.reputation.settings"}
{"event": "Loaded app settings", "level": "debug", "logger": "authentik.lib.config", "timestamp": 1717231866.5207522, "path": "authentik.sources.ldap.settings"}
{"event": "Loaded app settings", "level": "debug", "logger": "authentik.lib.config", "timestamp": 1717231866.5235906, "path": "authentik.sources.oauth.settings"}
{"event": "Loaded app settings", "level": "debug", "logger": "authentik.lib.config", "timestamp": 1717231866.5267992, "path": "authentik.events.settings"}
{"event": "Loaded app settings", "level": "debug", "logger": "authentik.lib.config", "timestamp": 1717231866.5306711, "path": "authentik.crypto.settings"}
{"event": "Loaded app settings", "level": "debug", "logger": "authentik.lib.config", "timestamp": 1717231866.5318296, "path": "authentik.blueprints.settings"}
{"event": "Loaded app settings", "level": "debug", "logger": "authentik.lib.config", "timestamp": 1717231866.5359561, "path": "authentik.stages.authenticator_totp.settings"}
{"event": "Loaded app settings", "level": "debug", "logger": "authentik.lib.config", "timestamp": 1717231866.5364292, "path": "authentik.enterprise.settings"}
{"event": "Loaded app settings", "level": "debug", "logger": "authentik.lib.config", "timestamp": 1717231866.5375683, "path": "authentik.providers.scim.settings"}
/ak-root/venv/lib/python3.12/site-packages/opencontainers/distribution/reggie/defaults.py:17: SyntaxWarning: invalid escape sequence '\('
  "http[s]?://(?:[a-zA-Z]|[0-9]|[$-_@.&+]|[!*\(\),]|(?:%[0-9a-fA-F][0-9a-fA-F]))+"
Illegal instruction (core dumped)

I think this is the reason why the server restart over an over again

[bnounours]

Continuing the investigation, it is when loading lib avatars

Python 3.12.3 (main, Apr 24 2024, 11:28:46) [GCC 12.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import authentik.lib.avatars
Illegal instruction (core dumped)

It is when loading the lxml lib

> /authentik/lib/avatars.py(12)<module>()
-> from lxml import etree  # nosec
(Pdb) 
Illegal instruction (core dumped)

I tried directly on the container

root@af1eb6b4d850:/# source ak-root/venv/bin/activate
(venv) root@af1eb6b4d850:/# python
Python 3.12.3 (main, Apr 24 2024, 11:28:46) [GCC 12.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from lxml import etree
Illegal instruction (core dumped)

[bnounours]

Upgrade lxml to 5.2.2 in the container unblock the server start. In the pyproject.toml there is no version limit for lxml in release 2024.4.2 in release 2024.4.1 there is a version fixed. I tried the same version as the one in the release 2024.4.1 but there is a compatibility error raising after.
The version lxml==5.2.1 seems to be the one problematic

To upgrade in the container:

sudo docker exec -it <authentik-server-container> bash
source /ak-root/venv/bin/activate
pip install lxml==5.2.2

You can test with

(venv) root@4437d18bb20e:/# python
Python 3.12.3 (main, Apr 24 2024, 11:28:46) [GCC 12.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from lxml import etree
>>> 

If you don't have core dump it is resolved. Then restart (not recreate !!) container and server works. The same needs to be done on worker (same image)

If the image restart too quickly here is one line command to do it

docker exec -it --user root <worker or server container> bash -c 'source /ak-root/venv/bin/activate; pip install lxml==5.2.2'
``

[depuits]

Upgrading to version 2024.6.0 now works for me.

[authentik-automation]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.