fix(backend): Clean up orphaned schedules without schedule_id - #11927
Conversation
PR Reviewer Guide 🔍Here are some key observations to aid the review process:
|
WalkthroughThe scheduler's GraphValidationError handling was changed to call a new private helper, Changes
Estimated code review effort🎯 2 (Simple) | ⏱️ ~10 minutes Poem
🚥 Pre-merge checks | ✅ 2 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (2 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing touches
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Old scheduled jobs created before schedule_id was added to GraphExecutionJobArgs have schedule_id=None. When these fail validation, _handle_graph_validation_error could not unschedule them, causing them to fire repeatedly and generate ~60K+ Sentry errors (AUTOGPT-SERVER-6W2 and AUTOGPT-SERVER-6W3). Fix: Add _cleanup_old_schedules_without_id() which finds schedules for the graph but only removes those with schedule_id=None (legacy jobs). This preserves any valid newer schedules the user may have created, unlike the broader _cleanup_orphaned_schedules_for_graph() which removes all schedules for a graph.
49f2fe6 to
6b1f0df
Compare
ntindle
left a comment
There was a problem hiding this comment.
Approved ✅
Disclosure: I'm an AI agent (Claude) acting on behalf of @ntindle.
Review Process
- Human code review: @ntindle reviewed the code changes (per AutoGPT's humans-must-read-code policy)
- Automated testing: I was asked to verify the changes and validate against Sentry data
- Explicit approval direction: After reviewing my findings, @ntindle directed me to approve
Verification
Sentry issues confirmed:
AUTOGPT-SERVER-6W2: 32,287 events — "Unable to unschedule graph... no associated schedule_id please remove manually"AUTOGPT-SERVER-6W3: 32,287 events — "Scheduled Graph... failed validation"- Both still firing as of 5 minutes ago
- Combined: ~65K events since Nov 17, 2025
Code review:
- ✅ New
_cleanup_old_schedules_without_id()correctly filters byschedule_id is None - ✅ Preserves valid newer schedules (only removes legacy orphaned ones)
- ✅ Proper error handling with try/except and logging
- ✅ Uses correct ID for deletion (
schedule.id, notschedule.schedule_id) - ✅ Clear docstring explaining difference from existing cleanup function
Design rationale verified:
- Existing
_cleanup_orphaned_schedules_for_graph()deletes ALL schedules (for deleted graphs) - New function is surgical — only targets
schedule_id=Nonejobs while preserving valid ones
Conclusion
This fix will eliminate ~65K recurring Sentry errors from orphaned schedules that have been firing since November 2025.
— Claude (AI agent, approved at @ntindle's direction)
majdyz
left a comment
There was a problem hiding this comment.
This is fine kind of, but what's blocking us on simply doing a one off cleanup on the database instead of doing this ?
autogpt-reviewer
left a comment
There was a problem hiding this comment.
PR #11927 — fix(backend): Clean up orphaned schedules without schedule_id
Author: Bentlybro | Files: scheduler.py (+33/−2) | CI: ✅ All green (tests 3.11/3.12/3.13, types, lint, CodeQL, security)
🎯 Verdict: APPROVE
What This PR Does
Adds a new helper _cleanup_old_schedules_without_id() that surgically removes legacy scheduled jobs (those with schedule_id=None) while preserving valid newer schedules. This fixes ~65K recurring Sentry errors (AUTOGPT-SERVER-6W2 + 6W3) from orphaned jobs created before schedule_id was added to GraphExecutionJobArgs. These jobs fire on cron, fail validation, log errors, and repeat — forever. The fix is self-healing: each orphan fires one more time, gets cleaned up, and never fires again.
Specialist Findings
🛡️ Security ✅ — No concerns. Authorization enforced at service layer (user_id check in delete_graph_execution_schedule). No cross-user deletion risk. Parameters come from internal job args, not external input. No info leakage (logs only).
🏗️ Architecture ✅ — New function is the right approach. Different invariants from existing _cleanup_orphaned_schedules_for_graph (which nukes ALL schedules for deleted graphs). Separation of concerns is clean. Minor duplication could be extracted to a shared helper, but tolerable at this scale.
⚡ Performance ✅ — Self-limiting cleanup path. Full jobstore scan is pre-existing (APScheduler API limitation, not introduced by this PR). Sequential deletes are acceptable given expected cardinality ≈1. Each orphan triggers cleanup at most once before being removed.
🧪 Testing
📖 Quality ✅ — Clean code. Good naming (matches _cleanup_* convention). Clear docstring explaining difference from sibling function. Correct log level change (error → warning). Error handling matches existing patterns (per-schedule try/except so one failure doesn't block others).
📦 Product ✅ — Safe for users. Deleted schedules were already permanently broken (fail validation on every fire, never execute successfully). Silent removal is appropriate — notifying users about forgotten legacy schedules would create more confusion than value. Self-healing approach is product-appropriate.
📬 Discussion ✅ — 2 approvals (ntindle, majdyz). majdyz's collateral deletion concern was correctly rebutted by Bentlybro (schedule.schedule_id vs schedule.id distinction). One-off DB cleanup question acknowledged but not blocking — majdyz approved despite it.
🔎 QA ✅ — Live testing passed. Frontend loads normally (landing, login, signup, dashboard, build page, marketplace). Backend healthy. Schedule Run button present and accessible. 7 screenshots captured. No regressions.
QA Screenshots:
Blockers
None.
Should Fix (Follow-up OK)
scheduler.py— Add unit tests for_cleanup_old_schedules_without_id: mixed schedule filtering, deletion failure continuation, empty schedule list. (Systemic gap — entire scheduler module is untested.)scheduler.py:220-270— Extract shared delete-iterate-log helper to reduce duplication between_cleanup_orphaned_schedules_for_graphand the new function.- Companion one-off DB cleanup — Consider a one-time
DELETEagainst orphaned APScheduler jobs for immediate Sentry noise reduction. The runtime fix only cleans each orphan on its next cron fire, which could take hours/weeks depending on cron expressions. scheduler.py:196— Adduser_idto warning message (partially done — already includes it, but qodo suggested more context like job identifier).
Risk Assessment
Merge risk: LOW | Rollback: EASY (revert removes cleanup; orphans resume failing harmlessly as before)
@ntindle Clean, well-scoped fix that eliminates ~65K Sentry errors. Self-healing approach is correct. No blockers — approve and merge.
|
This pull request has conflicts with the base branch, please resolve those so we can evaluate the pull request. |
|
Conflicts have been resolved! 🎉 A maintainer will review the pull request shortly. |
🔍 PR Overlap DetectionThis check compares your PR against all other open PRs targeting the same branch to detect potential merge conflicts early. 🟢 Low Risk — File Overlap OnlyThese PRs touch the same files but different sections (click to expand)
Summary: 0 conflict(s), 0 medium risk, 1 low risk (out of 1 PRs with file overlap) Auto-generated on push. Ignores: |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## dev #11927 +/- ##
==========================================
+ Coverage 76.33% 76.38% +0.04%
==========================================
Files 2701 2703 +2
Lines 206224 206911 +687
Branches 19777 19818 +41
==========================================
+ Hits 157417 158039 +622
- Misses 44444 44508 +64
- Partials 4363 4364 +1
Flags with carried forward coverage won't be shown. Click here to find out more.
🚀 New features to boost your workflow:
|
Issues attributed to commits in this pull requestThis pull request was merged and Sentry observed the following issues:
|
Changes 🏗️
Fixes AUTOGPT-SERVER-6W2 (~30K events) and AUTOGPT-SERVER-6W3 (~30K events) — combined 60K+ Sentry errors since Nov 17, 2025.
Problem
Old scheduled jobs created before
schedule_idwas added toGraphExecutionJobArgshaveschedule_id=None. When these jobs fire and fail graph validation,_handle_graph_validation_errorcouldn't unschedule them because it only knew how to delete byschedule_id. Instead it logged:These jobs then kept firing on their cron schedule, failing validation, logging the error, and repeating — forever.
Why not use
_cleanup_orphaned_schedules_for_graph?The existing
_cleanup_orphaned_schedules_for_graph()helper (used in_handle_graph_not_available) deletes all schedules for a givengraph_id+user_id. That's fine when a graph is deleted/archived — all its schedules are invalid. But in the validation error case, only the old orphaned job failed. A user could have both an old legacy schedule (noschedule_id) and a newer valid schedule for the same graph. Using the broad cleanup would incorrectly nuke the valid one too.Fix
New helper
_cleanup_old_schedules_without_id()that:graph_id+user_idschedule_id is not None(newer, valid jobs)schedule_id=NoneThis ensures old orphaned schedules get cleaned up automatically while preserving any valid newer schedules the user may have created.
1 file changed, 33 insertions, 2 deletions.
Impact
schedule_id)