[bug] Upgrade failed from v4.8.1 to v4.9.0 (Invalid environment variables)

[wattebausch]

Describe the bug

Hi All,

First of all, thank you for this great application. After a Redeploy from our sourcebot instance to image v.4.9.0 i got a error on the web "Internal Server Error"

The logs show me that

[34m[Info] DATABASE_URL is not set. Using embeded database.�[0m
[34m[Info] REDIS_URL is not set. Using embeded redis.�[0m
[34m[Info] Sourcebot version: v4.9.0�[0m
[33m[Warning] SOURCEBOT_ENCRYPTION_KEY is not set.�[0m
[34m[Info] Loading environment variables from /data/.sourcebot/.secret�[0m
[33m[Warning] AUTH_SECRET is not set.�[0m
[34m[Info] Loading environment variables from /data/.sourcebot/.authjs-secret�[0m
[34m[Info] Upgraded from version v4.8.1 to v4.9.0�[0m
localhost:5432 - no response
[34m[Info] Waiting for the database to be ready...�[0m
2025-11-05 07:13:47.704 UTC [15] LOG:  starting PostgreSQL 16.10 on x86_64-alpine-linux-musl, compiled by gcc (Alpine 13.2.1_git20231014) 13.2.1 20231014, 64-bit
2025-11-05 07:13:47.704 UTC [15] LOG:  listening on IPv6 address "::1", port 5432
2025-11-05 07:13:47.704 UTC [15] LOG:  listening on IPv4 address "127.0.0.1", port 5432
2025-11-05 07:13:47.706 UTC [15] LOG:  listening on Unix socket "/run/postgresql/.s.PGSQL.5432"
2025-11-05 07:13:47.708 UTC [20] LOG:  database system was interrupted; last known up at 2025-10-30 07:40:55 UTC
2025-11-05 07:13:47.822 UTC [20] LOG:  database system was not properly shut down; automatic recovery in progress
2025-11-05 07:13:47.824 UTC [20] LOG:  redo starts at 1/A4B26138
2025-11-05 07:13:47.824 UTC [20] LOG:  invalid record length at 1/A4B26170: expected at least 24, got 0
2025-11-05 07:13:47.824 UTC [20] LOG:  redo done at 1/A4B26138 system usage: CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s
2025-11-05 07:13:47.826 UTC [18] LOG:  checkpoint starting: end-of-recovery immediate wait
2025-11-05 07:13:47.830 UTC [18] LOG:  checkpoint complete: wrote 3 buffers (0.0%); 0 WAL file(s) added, 0 removed, 0 recycled; write=0.002 s, sync=0.001 s, total=0.005 s; sync files=2, longest=0.001 s, average=0.001 s; distance=0 kB, estimate=0 kB; lsn=1/A4B26170, redo lsn=1/A4B26170
2025-11-05 07:13:47.832 UTC [15] LOG:  database system is ready to accept connections
localhost:5432 - accepting connections
Database 'sourcebot' already exists; skipping creation.
�[34m[Info] Running database migration...�[0m
Prisma schema loaded from prisma/schema.prisma
Datasource "db": PostgreSQL database "sourcebot", schema "public" at "localhost:5432"

45 migrations found in prisma/migrations

Applying migration `20251031012851_change_connection_type_to_enum`
Applying migration `20251031014307_change_repo_code_host_type_to_enum`
Applying migration `20251031191203_remove_secret_table`
Applying migration `20251105021913_move_permission_syncing_to_account_level`

The following migration(s) have been applied:

migrations/
  └─ 20251031012851_change_connection_type_to_enum/
    └─ migration.sql
  └─ 20251031014307_change_repo_code_host_type_to_enum/
    └─ migration.sql
  └─ 20251031191203_remove_secret_table/
    └─ migration.sql
  └─ 20251105021913_move_permission_syncing_to_account_level/
    └─ migration.sql
      
All migrations have been successfully applied.
2025-11-05 07:13:50,409 CRIT Supervisor is running as root.  Privileges were not dropped because no user is specified in the config file.  If you intend to run as root, you can set user=root in the config file to avoid this message.
2025-11-05 07:13:50,411 INFO supervisord started with pid 7
2025-11-05 07:13:51,413 INFO spawned: 'redis' with pid 117
2025-11-05 07:13:51,415 INFO spawned: 'zoekt' with pid 118
2025-11-05 07:13:51,417 INFO spawned: 'backend' with pid 119
2025-11-05 07:13:51,419 INFO spawned: 'web' with pid 120
117:C 05 Nov 2025 07:13:51.426 * oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
117:C 05 Nov 2025 07:13:51.426 * Redis version=7.2.9, bits=64, commit=00000000, modified=0, pid=117, just started
117:C 05 Nov 2025 07:13:51.426 * Configuration loaded
117:M 05 Nov 2025 07:13:51.428 * monotonic clock: POSIX clock_gettime
117:M 05 Nov 2025 07:13:51.429 * Running mode=standalone, port=6379.
117:M 05 Nov 2025 07:13:51.429 * Server initialized
117:M 05 Nov 2025 07:13:51.430 * Loading RDB produced by version 7.2.9
117:M 05 Nov 2025 07:13:51.430 * RDB age 2 seconds
117:M 05 Nov 2025 07:13:51.430 * RDB memory usage when created 406.37 Mb
[zoekt] | 2025/11/05 07:13:51 [INFO] loading 378 shard(s): 1_101_v16.00000.zoekt, 1_103_v16.00000.zoekt, 1_104_v16.00000.zoekt, 1_105_v16.00000.zoekt, 1_106_v16.00000.zoekt... 373 more
[web] |    ▲ Next.js 15.5.0
[web] |    - Local:        http://localhost:3000
[web] |    - Network:      http://0.0.0.0:3000
[web] | 
[web] |  ✓ Starting...
[web] | ❌ Invalid environment variables: [
[web] |   {
[web] |     code: 'invalid_type',
[web] |     expected: 'string',
[web] |     received: 'undefined',
[web] |     path: [ 'CONFIG_PATH' ],
[web] |     message: 'Required'
[web] |   }
[web] | ]
[web] |  ✓ Ready in 361ms
[web] | Failed to prepare server Error: An error occurred while loading instrumentation hook: Invalid environment variables
[web] |     at i (.next/server/chunks/1072.js:32:241629)
[web] |     at e (.next/server/chunks/1072.js:32:241804)
[web] |     at <unknown> (.next/server/chunks/5078.js:2:112222)
[web] | Error: An error occurred while loading instrumentation hook: Invalid environment variables
[web] |     at i (.next/server/chunks/1072.js:32:241629)
[web] |     at e (.next/server/chunks/1072.js:32:241804)
[web] |     at <unknown> (.next/server/chunks/5078.js:2:112222)
[web] |  ⨯ unhandledRejection:  Error: An error occurred while loading instrumentation hook: Invalid environment variables
[web] |     at i (.next/server/chunks/1072.js:32:241629)
[web] |     at e (.next/server/chunks/1072.js:32:241804)
[web] |     at <unknown> (.next/server/chunks/5078.js:2:112222)
[backend] | ❌ Invalid environment variables: [
[backend] |   {
[backend] |     code: 'invalid_type',
[backend] |     expected: 'string',
[backend] |     received: 'undefined',
[backend] |     path: [ 'CONFIG_PATH' ],
[backend] |     message: 'Required'
[backend] |   }
[backend] | ]
[backend] | file:///app/node_modules/@t3-oss/env-core/dist/index.js:61
[backend] |         throw new Error("Invalid environment variables");
[backend] |               ^
[backend] | 
[backend] | Error: Invalid environment variables
[backend] |     at onValidationError (file:///app/node_modules/@t3-oss/env-core/dist/index.js:61:15)
[backend] |     at createEnv (file:///app/node_modules/@t3-oss/env-core/dist/index.js:67:16)
[backend] |     at file:///app/packages/shared/dist/env.server.js:51:20
[backend] | 
[backend] | Node.js v20.18.1
2025-11-05 07:13:52,386 WARN exited: backend (exit status 1; not expected)
117:M 05 Nov 2025 07:13:53.302 * Done loading RDB, keys loaded: 171271, keys expired: 0.
117:M 05 Nov 2025 07:13:53.302 * DB loaded from disk: 1.872 seconds
117:M 05 Nov 2025 07:13:53.302 * Ready to accept connections tcp
2025-11-05 07:13:53,302 INFO success: redis entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2025-11-05 07:13:53,302 INFO success: zoekt entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2025-11-05 07:13:53,302 INFO success: web entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2025-11-05 07:13:54,303 INFO spawned: 'backend' with pid 173
[backend] | ❌ Invalid environment variables: [
[backend] |   {
[backend] |     code: 'invalid_type',
[backend] |     expected: 'string',
[backend] |     received: 'undefined',
[backend] |     path: [ 'CONFIG_PATH' ],
[backend] |     message: 'Required'
[backend] |   }
[backend] | ]
[backend] | file:///app/node_modules/@t3-oss/env-core/dist/index.js:61
[backend] |         throw new Error("Invalid environment variables");
[backend] |               ^
[backend] | 
[backend] | Error: Invalid environment variables
[backend] |     at onValidationError (file:///app/node_modules/@t3-oss/env-core/dist/index.js:61:15)
[backend] |     at createEnv (file:///app/node_modules/@t3-oss/env-core/dist/index.js:67:16)
[backend] |     at file:///app/packages/shared/dist/env.server.js:51:20
[backend] | 
[backend] | Node.js v20.18.1
2025-11-05 07:13:55,131 WARN exited: backend (exit status 1; not expected)
2025-11-05 07:13:57,135 INFO spawned: 'backend' with pid 186
[backend] | ❌ Invalid environment variables: [
[backend] |   {
[backend] |     code: 'invalid_type',
[backend] |     expected: 'string',
[backend] |     received: 'undefined',
[backend] |     path: [ 'CONFIG_PATH' ],
[backend] |     message: 'Required'
[backend] |   }
[backend] | ]
[backend] | file:///app/node_modules/@t3-oss/env-core/dist/index.js:61
[backend] |         throw new Error("Invalid environment variables");
[backend] |               ^
[backend] | 
[backend] | Error: Invalid environment variables
[backend] |     at onValidationError (file:///app/node_modules/@t3-oss/env-core/dist/index.js:61:15)
[backend] |     at createEnv (file:///app/node_modules/@t3-oss/env-core/dist/index.js:67:16)
[backend] |     at file:///app/packages/shared/dist/env.server.js:51:20
[backend] | 
[backend] | Node.js v20.18.1
2025-11-05 07:13:57,972 WARN exited: backend (exit status 1; not expected)
2025-11-05 07:14:00,976 INFO spawned: 'backend' with pid 199
[backend] | ❌ Invalid environment variables: [
[backend] |   {
[backend] |     code: 'invalid_type',
[backend] |     expected: 'string',
[backend] |     received: 'undefined',
[backend] |     path: [ 'CONFIG_PATH' ],
[backend] |     message: 'Required'
[backend] |   }
[backend] | ]
[backend] | file:///app/node_modules/@t3-oss/env-core/dist/index.js:61
[backend] |         throw new Error("Invalid environment variables");
[backend] |               ^
[backend] | 
[backend] | Error: Invalid environment variables
[backend] |     at onValidationError (file:///app/node_modules/@t3-oss/env-core/dist/index.js:61:15)
[backend] |     at createEnv (file:///app/node_modules/@t3-oss/env-core/dist/index.js:67:16)
[backend] |     at file:///app/packages/shared/dist/env.server.js:51:20
[backend] | 
[backend] | Node.js v20.18.1
2025-11-05 07:14:01,767 WARN exited: backend (exit status 1; not expected)
2025-11-05 07:14:02,768 INFO gave up: backend entered FATAL state, too many start retries too quickly
[web] | Failed to prepare server Error: An error occurred while loading instrumentation hook: Invalid environment variables
[web] |     at i (.next/server/chunks/1072.js:32:241629)
[web] |     at e (.next/server/chunks/1072.js:32:241804)
[web] |     at <unknown> (.next/server/chunks/5078.js:2:112222)
[web] | Error: An error occurred while loading instrumentation hook: Invalid environment variables
[web] |     at i (.next/server/chunks/1072.js:32:241629)
[web] |     at e (.next/server/chunks/1072.js:32:241804)
[web] |     at <unknown> (.next/server/chunks/5078.js:2:112222)
[web] | Error: An error occurred while loading instrumentation hook: Invalid environment variables
[web] |     at i (.next/server/chunks/1072.js:32:241629)
[web] |     at e (.next/server/chunks/1072.js:32:241804)
[web] |     at <unknown> (.next/server/chunks/5078.js:2:112222)
[web] | Error: An error occurred while loading instrumentation hook: Invalid environment variables
[web] |     at i (.next/server/chunks/1072.js:32:241629)
[web] |     at e (.next/server/chunks/1072.js:32:241804)
[web] |     at <unknown> (.next/server/chunks/5078.js:2:112222)

To reproduce

We upgrade our deployment from Sourcebot since v3.0 with new image tag on our kubernetes deployment with flux2. deployment below.

Sourcebot deployment information

Sourcebot version (v4.8.1 --> v4.9.0):
Kubernetes K3s 1.34.1
Flux2 2.7.2

kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
  - ../../_base/sourcebot

patches:
  - patch: |-
      - op: replace
        path: /spec/rules/0/host
        value: sourcebot.example.com
    target:
      kind: Ingress
  - patch: |-
      - op: replace
        path: /spec/template/spec/containers/0/env/1/value
        value: https://sourcebot.example.com
    target:
      kind: Deployment

deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: deployment
spec:
  selector:
    matchLabels:
      app: sourcebot
  template:
    metadata:
      labels:
        app: sourcebot
    spec:
      containers:
        - image: ghcr.io/sourcebot-dev/sourcebot:v4.8.1
          name: sourcebot
          resources:
            requests:
              memory: "1Gi"
              cpu: "0.1"
            limits:
              memory: "4Gi"
          env:
            - name: SOURCEBOT_AUTH_ENABLED
              value: "true"
            - name: AUTH_URL
              value: "replace_me"
          volumeMounts:
            - name: sourcebot-data
              mountPath: /data
      volumes:
        - name: sourcebot-data
          persistentVolumeClaim:
            claimName: sourcebot-data-pvc

Additional information

No response

[brendan-kellam]

Heya - could you try explicitly setting CONFIG_PATH to where you store your config? (Guessing /data/config.json).

Looks like we inadvertently made a breaking change here. I'll take a closer look tomorrow morning. Thanks for raising.

[wattebausch]

We do not have an explicit configuration because we have a very small environment. I did the entire configuration via the web.

We have a test and production instance, so there's no rush. I just wanted to raise an issue so you'd know about it.
If you like, I can try with a configuration via Kubernetes.

[brendan-kellam]

Ah kk that makes sense - for context, we deprecated support for configuring via the web in #515 since it was a remnant of when we had a SaaS offering briefly. In v4.9.0 we made CONFIG_PATH required, hence why you are seeing this error.

If you could configure your instance via the config file that will solve the issue. Going to close this issue but feel free to re-open if you run into it again.