Fix running processes count endpoint (issue #1258)#1384
Open
torkashvand wants to merge 2 commits intomainfrom
Open
Fix running processes count endpoint (issue #1258)#1384torkashvand wants to merge 2 commits intomainfrom
torkashvand wants to merge 2 commits intomainfrom
Conversation
torkashvand
force-pushed
the
fix/running-processes-count-1258
branch
3 times, most recently
from
099cf3f to
0d97328
Compare
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #1384 +/- ##
==========================================
+ Coverage 80.33% 80.43% +0.10%
==========================================
Files 275 276 +1
Lines 13854 13977 +123
Branches 1369 1377 +8
==========================================
+ Hits 11129 11242 +113
- Misses 2410 2418 +8
- Partials 315 317 +2 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
Replace flawed increment/decrement counter with actual database queries. The previous implementation used engine_settings.running_processes which was incremented when processes started and decremented when they finished. However, this counter could drift over time due to: - Failed transactions that don't commit the decrement - Crashes or exceptions that prevent the finally block from executing - Race conditions in concurrent environments This fix: - Adds get_actual_running_processes_count() function that queries the actual count from ProcessTable where last_status == 'running' - Updates generate_engine_global_status() to use the actual count - Updates generate_engine_status_response() to override the flawed counter with the actual count - Updates test to create actual running processes instead of manipulating the counter Fixes: #1258
torkashvand
force-pushed
the
fix/running-processes-count-1258
branch
from
0d97328 to
975ced7
Compare
tjeerddie
reviewed
Feb 9, 2026
tjeerddie
approved these changes
Feb 9, 2026
Mark90
reviewed
Feb 9, 2026
- Add WorkerStatusMonitor with periodic background updates - Track actual running jobs via ThreadPool active job counter - Replace database-based counting with worker inspection - Add thread-safe singleton with double-checked locking - Use threading.Event for clean shutdown handling - Make update interval configurable (WORKER_STATUS_INTERVAL setting) - Add refresh_once() for deterministic testing - Refactor active job tracking with context manager to reduce complexity.
torkashvand
force-pushed
the
fix/running-processes-count-1258
branch
from
994371d to
0689aff
Compare
Mark90
approved these changes
Feb 16, 2026
Contributor
Mark90
left a comment
Mark90
left a comment
There was a problem hiding this comment.
Very nice :)
Left a few questions, but LGTM!
| self.number_of_workers_online = getattr(thread_pool, "_max_workers", -1) | ||
| self.number_of_queued_jobs = thread_pool._work_queue.qsize() if hasattr(thread_pool, "_work_queue") else 0 | ||
| self.number_of_running_jobs = len(getattr(thread_pool, "_threads", [])) | ||
| self.number_of_running_jobs = get_active_threadpool_jobs_count() |
Contributor
There was a problem hiding this comment.
Ah this was indeed broken, didn't know that. Thanks for fixing this as well :)
|
|
||
| _workflow_executor = None | ||
| _active_threadpool_jobs = 0 | ||
| _active_jobs_lock = __import__("threading").Lock() |
Contributor
There was a problem hiding this comment.
Was it not possible to import it the usual way?
| """Get the global WorkerStatusMonitor instance. | ||
|
|
||
| Thread-safe singleton pattern with double-checked locking. | ||
| Restarts the monitor if it was previously stopped. |
Comment on lines
+42
to
+43
| with self._lock: | ||
| self._running_jobs_count = count |
Contributor
There was a problem hiding this comment.
Assigning a new value is atomic, so I don't think this requires locking
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Replace flawed increment/decrement counter with actual database queries.
The previous implementation used engine_settings.running_processes which was incremented when processes started and decremented when they finished. However, this counter could drift over time due to:
This fix:
Fixes: #1258