attempt to concatenate a userdata value

[mkopeyka]

Is there an existing issue for this?

• I have searched the existing issues

Kong version ($ kong version)

3.10.0.1

Current Behavior

I'm trying to migrate to latest version of kong.
My current version is 3.7.1.1 and I migrated database to 3.8.1.1, then to 3.9.1.1 and finally to 3.10.0.1

Kong doesn't work, I can't open kong admin UI, I see a lot of errors:

NOTE: setting KONG_LOG_LEVEL to error
NOTE: setting KONG_PROXY_LISTEN to 0.0.0.0:8000
NOTE: setting KONG_ADMIN_LISTEN to 0.0.0.0:8001
NOTE: starting - Kong Enterprise 3.10.0.1
...
2025/04/18 18:50:56 [debug] 30#0: *2 [lua] plugins.lua:696: load_plugin(): Loading plugin: upstream-oauth
2025/04/18 18:51:04 [error] 59#0: *1774 [lua] api_helpers.lua:544: handle_error(): /usr/local/share/lua/5.1/kong/tools/redis/config_utils.lua:159: attempt to concatenate a userdata value

stack traceback:
        /usr/local/share/lua/5.1/kong/tools/redis/config_utils.lua:159: in function 'gen_poolname'
        ./openid-connect/sessions.lua:137: in function 'session_open'
        ./openid-connect/handler.lua:352: in function <./openid-connect/handler.lua:178>
        ./invoke_plugin.lua:29: in function 'apply_plugin'
        ./invoke_plugin.lua:170: in function 'invoke_plugin'
        ./api_helpers.lua:188: in function 'authenticate'
        ./api_helpers.lua:638: in function 'filter'
        /usr/local/share/lua/5.1/lapis/application.lua:22: in function 'run_before_filter'
        /usr/local/share/lua/5.1/lapis/application.lua:177: in function 'resolve'
        /usr/local/share/lua/5.1/lapis/application.lua:217: in function </usr/local/share/lua/5.1/lapis/application.lua:215>
        [C]: in function 'xpcall'
        /usr/local/share/lua/5.1/lapis/application.lua:215: in function 'dispatch'
        /usr/local/share/lua/5.1/lapis/nginx.lua:231: in function 'serve'
        /usr/local/share/lua/5.1/kong/init.lua:2203: in function 'admin_content'
        content_by_lua(nginx-kong.conf:482):2: in main chunk, client: 10.132.34.5, server: kong_admin, request: "GET /"
2025/04/18 18:50:10.132.34.5 - - [18/Apr/2025:18:51:04 +0000] "GET /" 500 42 "-" "-"
2025/04/18 18:51:05 [error] 60#0: *1775 [lua] api_helpers.lua:544: handle_error(): /usr/local/share/lua/5.1/kong/tools/redis/config_utils.lua:159: attempt to concatenate a userdata value

stack traceback:
        /usr/local/share/lua/5.1/kong/tools/redis/config_utils.lua:159: in function 'gen_poolname'
        ./openid-connect/sessions.lua:137: in function 'session_open'
        ./openid-connect/handler.lua:352: in function <./openid-connect/handler.lua:178>
        ./invoke_plugin.lua:29: in function 'apply_plugin'
        ./invoke_plugin.lua:170: in function 'invoke_plugin'
        ./api_helpers.lua:188: in function 'authenticate'
        ./api_helpers.lua:638: in function 'filter'
        /usr/local/share/lua/5.1/lapis/application.lua:22: in function 'run_before_filter'
        /usr/local/share/lua/5.1/lapis/application.lua:177: in function 'resolve'
        /usr/local/share/lua/5.1/lapis/application.lua:217: in function </usr/local/share/lua/5.1/lapis/application.lua:215>
        [C]: in function 'xpcall'
        /usr/local/share/lua/5.1/lapis/application.lua:215: in function 'dispatch'
        /usr/local/share/lua/5.1/lapis/nginx.lua:231: in function 'serve'
        /usr/local/share/lua/5.1/kong/init.lua:2203: in function 'admin_content'
        content_by_lua(nginx-kong.conf:482):2: in main chunk, client: 10.132.34.4, server: kong_admin, request: "GET /"
10.132.34.4 - - [18/Apr/2025:18:51:05 +0000] "GET /" 500 42 "-" "-"
2025/04/18 18:51:05 [error] 59#0: *668 [kong] exporter.lua:32 failed retrieving entity counts: timeout, context: ngx.timer
2025/04/18 18:51:05 [error] 59#0: *669 [kong] event_hooks.lua:444 [postgres] timeout, context: ngx.timer
2025/04/18 18:51:05 [crit] 59#0: *664 [lua] init.lua:268: init(): failed creating balancer for upstream tws-upstream-test: failed fetching targets for upstream tws-upstream-test, context: ngx.timer

Expected Behavior

It should work without errors

Steps To Reproduce

1. upgrade 3.7.1.1 to 3.10.0.1
2. start 3.10.0.1
3. check console, try to open admin page

Anything else?

No response

[mkopeyka]

I found the root cause. Seems to me it's related to KONG_ADMIN_GUI_AUTH_CONF variable.
If I use this config kong generates a lot of errors (see above) however it works fine if I use "session_storage": "cookie" instead of "session_storage": "redis"
Redis server is up and running, I tested it via CLI.

{
    "issuer": "https://auth-dev.local.test.com/auth/realms/test",
    "admin_claim": "email",
    "client_id": [
        "test"
    ],
    "client_secret": [
        "xxxxxxxxxxxxx"
    ],
    "authenticated_groups_claim": [
        "groups"
    ],
    "redirect_uri": [
        "https://kong-api-dev.qwerty.com/auth"
    ],
    "login_redirect_uri": [
        "https://kong-ui-dev.qwerty.com"
    ],
    "logout_redirect_uri": [
        "https://kong-ui-dev.qwerty.com"
    ],
    "scopes": [
        "openid",
        "email",
        "offline_access"
    ],
    "admin_auto_create_rbac_token_disabled": false,
    "admin_auto_create": true,
    "authorization_cookie_domain": ".qwerty.com",
    "session_cookie_domain": ".qwerty.com",
    "session_redis_host": "xx.xxx.xxx.xxx",
    "session_redis_password": "xxxxxxxxxxxxxxx",
    "session_redis_port": 6379,
    "session_redis_prefix": "sessions-kgd",
    "authorization_cookie_secure": true,
    "authorization_cookie_http_only": true,
    "session_cookie_same_site": "Lax",
    "authorization_cookie_same_site": "Default",
    "session_cookie_secure": true,
    "session_cookie_name": "sessions-kgd"
    "session_storage": "redis",
}

[Water-Melon]

https://docs.konghq.com/hub/kong-inc/openid-connect/configuration/#config-session_storage

Please confirm that this works as intended.

[mkopeyka]

It doesn't work with redis