Diagnosing Dagster slow starting task in k8s #24386
-
|
Hello, I've been using Dagster for a while, deployed on k8s and things are working well. Lately though, the startup time for jobs has gotten to be excessively long. The pod will spin up and it will take about 10 mins with dagster running at 100% CPU before the task starts executing. Is there a way to enable logging or another way to determine what the dagster app is doing that is taking so long to start up? 
Thank you for your help! |
Beta Was this translation helpful? Give feedback.
Replies: 1 comment 1 reply
-
|
Hey Joel, in this other discussion we have a walkthrough for how you can profile your code to identify any bottlenecks and sources of latency: #14771 |
Beta Was this translation helpful? Give feedback.
-
|
Thank you! With py-spy I was able to see the code hanging in multiprocess executor files. That led me to find the concurrency limits were being hit. Would be great to have a log or some indication on the job that it is not running due to concurrency limits. |
Beta Was this translation helpful? Give feedback.
Hey Joel, in this other discussion we have a walkthrough for how you can profile your code to identify any bottlenecks and sources of latency: #14771