Skip to content

feat: speed up & revamp of workers (v3) #9693

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 98 commits into from
Oct 20, 2025
Merged

feat: speed up & revamp of workers (v3) #9693

merged 98 commits into from
Oct 20, 2025

Conversation

@abuaboud
Copy link
Member

What does this PR do?

Explain How the Feature Works

Relevant User Scenarios

Fixes # (issue)

abuaboud and others added 20 commits October 10, 2025 13:04
@abuaboud
refactor: redis conection is moved to server-shared
b7929f9
@abuaboud
Revert "revert: "perf(worker): improve worker performance by checking…
c26edb7 
… memory before locking""
@abuaboud
fix: add file existence check in readCacheFromFile to prevent errors …
de8b940 
…when cache file is missing
@amrdb
Merge pull request #9649 from activepieces/refactor/redis
a55a614 
refactor: redis conection is moved to server-shared
@abuaboud
refactor: remove DELAYED_FLOW job type
a92be07
@abuaboud
feat: redis connection moved to worker
a703db4
@abuaboud
feat: implement cache ID management with distributed locking
87d393f
@abuaboud
refactor: update Redis connection handling to use optional retrieval …
4f6517a 
…for configuration properties
@abuaboud
Merge pull request #9686 from activepieces/refactor/redis-connection
07e34b1 
feat: redis connection moved to worker
@abuaboud
Merge remote-tracking branch 'origin/feat/workerv3' into ENG-951
e9355e0
@abuaboud
refactor: remove proper-lockfile dependency and update cache manageme…
64103f2 
…nt to use logging
@abuaboud
fix: correct package name in cache management for pieces-common
686a6c9
@abuaboud
fix: increase timeout duration for cache save operation to improve re…
81db85a 
…liability
@abuaboud
Merge pull request #9689 from activepieces/ENG-951
60c5497 
feat: share volume across machines
@abuaboud
Merge branch 'feat/workerv3' into ENG-1015
4a6b2e9
@abuaboud
refactor: clean up code formatting and remove unused dependency from …
c8415ac 
…package.json
@abuaboud
Merge pull request #9673 from activepieces/ENG-1015
8723c14 
refactor: remove DELAYED_FLOW job type
@abuaboud
Merge remote-tracking branch 'origin/main' into feat/workerv3
4bf3114
@abuaboud
feat: move queue consumption to workers
f1aab15
@abuaboud
Merge branch 'main' into feat/workerv3
d00b7f7
@nx-cloud
Copy link

nx-cloud bot commented Oct 15, 2025 •
edited
Loading

🤖 Nx Cloud AI Fix Eligible

An automatically generated fix could have helped fix failing tasks for this run, but Self-healing CI is disabled for this workspace. Visit workspace settings to enable it and get automatic fixes in future runs.

To disable these notifications, a workspace admin can disable them in workspace settings.

View your CI Pipeline Execution ↗ for commit 0161d73

Command Status Duration Result
nx affected --target=lint --exclude=pieces-* --... ❌ Failed 2m 5s View ↗

☁️ Nx Cloud last updated this comment at 2025-10-20 03:01:58 UTC

amrdb and others added 27 commits October 19, 2025 20:26
@amrdb
Merge branch 'feat/workerv3' of https://github.com/activepieces/activ…
7498825 
…epieces into feat/workerv3
@abuaboud
refactor: adjust job delay logic to dynamically calculate based on at…
4996caf 
…tempt count
@abuaboud
Merge branch 'feat/workerv3' of https://github.com/activepieces/activ…
a7cade9 
…epieces into feat/workerv3
@abuaboud
refactor: clean up formatting in job queue worker and flow job execut…
630a60a 
…or files
@abuaboud
refactor: replace workerDistributedLock with memoryLock for cache sav…
e251c2c 
…e operation
@amrdb
fix: allow simulate webhooks when flow is disbaled
ec098a4
@amrdb
Merge branch 'feat/workerv3' of https://github.com/activepieces/activ…
747faa3 
…epieces into feat/workerv3
@abuaboud
refactor: update paused job migration to use new job queue and versio…
998adcc 
…n key
@abuaboud
refactor: import MoreThan from typeorm for improved query handling in…
812fbc5 
… paused job migration
@abuaboud
Merge branch 'feat/workerv3' of https://github.com/activepieces/activ…
7c85eb5 
…epieces into feat/workerv3
@abuaboud
fix: refill paused flow
e702ced
@abuaboud
fix: update paused job migration to handle job removal errors and inc…
b64c191 
…rement version key
@abuaboud
fix: remove redundant worker.run() call in jobQueueWorker start method
3f51ebe
@abuaboud
fix: resume worker if already initialized in jobQueueWorker start method
08a10c8
@abuaboud
feat: add logging for job queue worker pause and resume actions
9358fbe
@abuaboud
fix: remove pause call from flowWorker on disconnection
169536e
@abuaboud
fix: handle JSON parsing errors in flowRunLogs response
9f8afa0
@abuaboud
fix: remove isPausedTerminal parameter from flow run state check
ebc9348
@abuaboud
refactor: remove pause method from job queue worker
0dd8c65
@abuaboud
refactor: replace workerDistributedLock with memoryLock in worker-cache
cf149d7
@abuaboud
refactor: replace memoryLock with workerDistributedLock for cache ID …
0910668 
…and cache state management
@abuaboud
feat: enhance logging in job queue worker for job consumption and thr…
d560da8 
…ottling
@abuaboud
fix: update throttling log message in job queue worker for clarity
a7349fe
@abuaboud
refactor: unify cache management by replacing
67e630f
@abuaboud
feat: enhance logging in engine process manager and job queue worker …
891e380 
…for better monitoring
@abuaboud
feat: add timeout logging in engine process manager for improved task…
c1ac898 
… monitoring
@abuaboud
refactor: standardize timeout handling and improve cache management a…
0161d73 
…cross worker components
@abuaboud abuaboud changed the title feat: worker on steroid feat: speed up & revamp of workers (v3) Oct 20, 2025
@abuaboud abuaboud added the 🌟 feature new major feature label Oct 20, 2025
@abuaboud abuaboud merged commit 9e8b8e7 into main Oct 20, 2025
24 of 26 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

🌟 feature new major feature

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@abuaboud @amrdb