Langfuse deployment issue on k8

[hrone-dev-satyammishra]

Langfuse deployment on Kubernetes: The Langfuse pod continuously exits and restarts.

During each restart, it attempts to run a migration script but seems to get stuck in a loop, repeatedly executing the same migration steps without completing. Meanwhile, the Postgres pod is functioning normally, but the Langfuse pod faces persistent issues with this restart cycle.

Script executed successfully.
Prisma schema loaded from packages/shared/prisma/schema.prisma
Datasource "db": PostgreSQL database "langfuse_db", schema "public" at "postgresdbsvc:5432"
 
266 migrations found in prisma/migrations
 
 
No pending migrations to apply.
  ▲ Next.js 14.2.15

 
✓ Starting...
Running init scripts...
✓ Ready in 15.5s
=========================================================================
NAME                          READY   STATUS      RESTARTS      AGE
langfuse-5b647d9f7c-dg2z6     0/1     OOMKilled   3 (64s ago)   2m49s
postgresdb-65d6bbdf95-nrkgk   1/1     Running     0             9m5s
=========================================================================



# Events:
=========================================================================

  Type     Reason     Age                     From               Message

  ----     ------     ----                    ----               -------

  Normal   Scheduled  7m36s                   default-scheduler  Successfully assigned langfuse/langfuse-5b647d9f7c-dg2z6 to aks-agentpool-22640134-vmss00007i

  Normal   Pulled     7m34s                   kubelet            Successfully pulled image "langfuse/langfuse:latest" in 1.501s (1.501s including waiting)

  Normal   Pulled     7m5s                    kubelet            Successfully pulled image "langfuse/langfuse:latest" in 1.472s (1.472s including waiting)

  Normal   Pulled     6m19s                   kubelet            Successfully pulled image "langfuse/langfuse:latest" in 1.468s (1.468s including waiting)

  Normal   Created    5m21s (x4 over 7m34s)   kubelet            Created container langfuse

  Normal   Started    5m21s (x4 over 7m34s)   kubelet            Started container langfuse

  Normal   Pulled     5m21s                   kubelet            Successfully pulled image "langfuse/langfuse:latest" in 1.443s (1.443s including waiting)

  Normal   Pulling    4m3s (x5 over 7m36s)    kubelet            Pulling image "langfuse/langfuse:latest"

  Normal   Pulled     4m1s                    kubelet            Successfully pulled image "langfuse/langfuse:latest" in 1.479s (1.479s including waiting)

  Warning  BackOff    2m30s (x13 over 6m36s)  kubelet            Back-off restarting failed container langfuse in pod langfuse-5b647d9f7c-dg2z6_langfuse(597d5304-9537-43ca-b73e-9c65084ce492)

[dosubot[bot]]

The issue you're experiencing with the Langfuse pod on Kubernetes continuously exiting and restarting with an OOMKilled status is likely due to the pod exceeding its memory limits. Here are some steps you can take to resolve this:

1. Increase Memory Limits: Adjust the memory limits for the Langfuse pod in your Kubernetes configuration. You can modify the resources section in your pod or deployment YAML file to allocate more memory:

   resources:
     limits:
       memory: "512Mi"  # Increase this value as needed
     requests:
       memory: "256Mi"  # Increase this value as needed

2. Optimize Configuration: Review the Langfuse configuration to ensure it is optimized for your deployment size. For example, you can adjust the DB_EXPORT_PAGE_SIZE to a smaller value to reduce memory usage during exports:

   env:
     - name: DB_EXPORT_PAGE_SIZE
       value: "500"  # Adjust this value to optimize performance

3. Check for Memory Leaks: Ensure there are no memory leaks in your application or any custom code running within the Langfuse pod.

4. Monitor Resource Usage: Use Kubernetes monitoring tools to observe the memory usage of the Langfuse pod and adjust the resources accordingly.

By increasing the memory limits and optimizing the configuration, you should be able to prevent the Langfuse pod from being OOMKilled ^[1].

To continue talking to Dosu, mention @dosu.

[Steffen911]

@hrone-dev-satyammishra Thank you for your issue report.

I have a couple of questions based on your report that would help me understand what the root cause might be:

• What resources have you configured for the langfuse container and how many resources are available on the host? Your snippet shows OOMKilled which indicates that the langfuse pod consumed more than it's resource request. It may help to increase the memory request and set it to the same value as the memory limit.
• Do you have any healthchecks configured on the langfuse pod? We've observed cases where the health checks marked the pod as failed due to long-running migrations.
• Do you have any logs you can share?

My take is that it's most likely the first point and setting requests.memory = limits.memory may help here. We recommend at least 3 GB of RAM for the langfuse pod as documented in our self-hoster documentation.

[hrone-dev-satyammishra]

@Steffen911

[Steffen911]

@hrone-dev-satyammishra Are these your nodes? What is the configuration for the Langfuse pod itself?

[hrone-dev-satyammishra]

Hi @Steffen911 , I am using Azure Kubernetes Services and the node size is Standard_D2s_v3 ( 2 vCPUs & 8 GiB memory).
The Langfuse pod is using 512Mi memory & 250 Mi cpu

[Steffen911]

@hrone-dev-satyammishra Are the 512Mi memory set as request and limit? Could you provide it more resources, ideally 3GB, per our self-hoster documentation?