[Bug][Backend] Service freezing during pipeline runs

[robaca]

Search before asking

• I had searched in the issues and found no similar issues.

What happened

Hi,

I just installed DevLake on a VM with 4GB RAM and 2 CPU, with the same versions (~1.0.2rc1) and docker-compose/project setup like I have on my local development machine. Some of the jobs that run flawlessly on my local machine here stop after a while collecting GitHub or Jira data.

The last logs I get from the backend are roughtly like

devlake-1  | time="2025-06-26 14:00:26" level=info msg=" [pipeline service] [pipeline #18] [task #93] finished step: 8 / 32"
devlake-1  | time="2025-06-26 14:00:26" level=info msg=" [pipeline service] [pipeline #18] [task #93] executing subtask extractIssueType"
devlake-1  | time="2025-06-26 14:00:26" level=info msg=" [pipeline service] [pipeline #18] [task #93] [extractIssueType] extract IssueType, connection_id=1, board_id=3153"
devlake-1  | time="2025-06-26 14:00:26" level=info msg=" [pipeline service] [pipeline #18] [task #93] [extractIssueType] get data from _raw_jira_api_issue_types where params={\"ConnectionId\":1,\"BoardId\":3153} and got 152"
devlake-1  | time="2025-06-26 14:00:26" level=info msg=" [pipeline service] [pipeline #18] [task #93] [extractIssueType] finished records: 1(not exactly)"
devlake-1  | time="2025-06-26 14:00:26" level=info msg=" [pipeline service] [pipeline #18] [task #93] subtask extractIssueType finished in 126 ms"
devlake-1  | time="2025-06-26 14:00:26" level=info msg=" [pipeline service] [pipeline #18] [task #93] finished step: 9 / 32"
devlake-1  | time="2025-06-26 14:00:26" level=info msg=" [pipeline service] [pipeline #18] [task #93] executing subtask collectIssues"
devlake-1  | time="2025-06-26 14:00:26" level=info msg=" [pipeline service] [pipeline #18] [task #93] [collectIssues] got user's timezone: UTC"
devlake-1  | time="2025-06-26 14:00:26" level=info msg=" [pipeline service] [pipeline #18] [task #93] [collectIssues] start api collection"
devlake-1  | time="2025-06-26 14:00:42" level=info msg=" [pipeline service] [pipeline #18] [task #93] [collectIssues] finished records: 1(not exactly)"

Shortly after that, the backend doesn't react any more to any incoming requests from the config-ui (HTTP 504) or the API endpoints. They are hanging and on the database side nothing happens any more (show processlist => all sleeping). New HTTP requests do not get logged. Only a restart and then abort of the running pipeline is helping.

I don't see memory issues (btw. are there any CPU/memory recommendations for the services?). "Smaller" projects are finishing successfully.
Could it be a connection pool leak somewhere? Is it possible to increase verbosity or set some limits on the processing side (connection rate limits I already reduced)?

Any help is highly appreciated.

What do you expect to happen

Service should not freeze.

How to reproduce

Not sure, maybe by reducing memory and CPU on an existing setup.

Anything else

It happens on every execution of the same blueprints, at the same place

Version

1.0.2-rc1

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[robaca]

I tried to reproduce the issue by giving the same cpu/memory constraints to the docker containers on my local machine that configured above.

It did not freeze locally, but the job failed silently when running a gitextractor task. The frontend was not aware of the failure but still showed the job and task as in progress.

Increasing the devlake container memory to 6 GB solved the issue (locally). I then saw multiple git subprocesses running in the container process list. It seems that these processes fail due to the limited memory and the devlake process does not detect such failures.

I will try the same on the initial remote VM when having more memory at hand.

[klesh]

How big is your repo? And how long were you collecting?

[robaca]

I found the reason for the freezing: The process has been instrumented by Dynatrace. Disabling it solved the issue.

DevLake could benefit from an update of the Go version - the currently used version 1.20.5 is not maintained any more and not supported by Dynatrace.

[robaca]

@klesh the repository is ~630 GBMB (.git folder) in size and has 8 submodules.

[klesh]

Got it.

[github-actions]

This issue has been automatically marked as stale because it has been inactive for 60 days. It will be closed in next 7 days if no further activity occurs.

[github-actions]

This issue has been closed because it has been inactive for a long time. You can reopen it if you encounter the similar problem in the future.