Skip to content

fix(backend): Clean up orphaned schedules without schedule_id - #11927

Merged
Bentlybro merged 4 commits into
devfrom
fix/scheduler-orphaned-schedule-cleanup
Jul 22, 2026
Merged

fix(backend): Clean up orphaned schedules without schedule_id#11927
Bentlybro merged 4 commits into
devfrom
fix/scheduler-orphaned-schedule-cleanup

Conversation

@Bentlybro

@Bentlybro Bentlybro commented Feb 2, 2026

Copy link
Copy Markdown
Member

Changes 🏗️

Fixes AUTOGPT-SERVER-6W2 (~30K events) and AUTOGPT-SERVER-6W3 (~30K events) — combined 60K+ Sentry errors since Nov 17, 2025.

Problem

Old scheduled jobs created before schedule_id was added to GraphExecutionJobArgs have schedule_id=None. When these jobs fire and fail graph validation, _handle_graph_validation_error couldn't unschedule them because it only knew how to delete by schedule_id. Instead it logged:

Unable to unschedule graph: <id> as this is an old job with no associated schedule_id please remove manually

These jobs then kept firing on their cron schedule, failing validation, logging the error, and repeating — forever.

Why not use _cleanup_orphaned_schedules_for_graph?

The existing _cleanup_orphaned_schedules_for_graph() helper (used in _handle_graph_not_available) deletes all schedules for a given graph_id + user_id. That's fine when a graph is deleted/archived — all its schedules are invalid. But in the validation error case, only the old orphaned job failed. A user could have both an old legacy schedule (no schedule_id) and a newer valid schedule for the same graph. Using the broad cleanup would incorrectly nuke the valid one too.

Fix

New helper _cleanup_old_schedules_without_id() that:

  1. Fetches all schedules for the graph_id + user_id
  2. Skips any schedule where schedule_id is not None (newer, valid jobs)
  3. Only deletes legacy jobs with schedule_id=None

This ensures old orphaned schedules get cleaned up automatically while preserving any valid newer schedules the user may have created.

1 file changed, 33 insertions, 2 deletions.

Impact

  • Eliminates ~60K Sentry errors from 6W2/6W3
  • Old orphaned schedules will self-clean on next validation failure
  • No effect on new jobs (which always have schedule_id)
  • Preserves valid newer schedules for the same graph (unlike the broader cleanup)

@Bentlybro
Bentlybro requested a review from a team as a code owner February 2, 2026 14:06
@Bentlybro
Bentlybro requested review from Otto-AGPT and majdyz and removed request for a team February 2, 2026 14:06
@github-project-automation github-project-automation Bot moved this to 🆕 Needs initial review in AutoGPT development kanban Feb 2, 2026
@github-actions github-actions Bot added platform/backend AutoGPT Platform - Back end size/s labels Feb 2, 2026
@qodo-code-review

Copy link
Copy Markdown

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 2 🔵🔵⚪⚪⚪
🧪 No relevant tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Cleanup semantics

Confirm _cleanup_orphaned_schedules_for_graph(graph_id, user_id) is safe and correct in this validation-error context: it should be idempotent, tolerate “already deleted” schedules, and not remove unrelated schedules (e.g., multiple schedules for the same graph/user, if that’s possible). Also validate expected behavior if the cleanup call fails (retries vs. leaving the schedule to continue firing).

logger.warning(
    f"Old scheduled job for graph {args.graph_id} has no schedule_id, "
    f"attempting cleanup by graph_id lookup"
)
await _cleanup_orphaned_schedules_for_graph(args.graph_id, args.user_id)
Logging quality

The new warning message is an improvement, but consider including more context (e.g., job identifier / schedule name if available in args, and user_id) to make remediation and correlation easier while still avoiding noisy logs.

logger.warning(
    f"Old scheduled job for graph {args.graph_id} has no schedule_id, "
    f"attempting cleanup by graph_id lookup"
)

@coderabbitai

coderabbitai Bot commented Feb 2, 2026

Copy link
Copy Markdown
Contributor

Walkthrough

The scheduler's GraphValidationError handling was changed to call a new private helper, _cleanup_old_schedules_without_id, which queries and deletes schedules that lack a schedule_id and logs outcomes; when a schedule_id exists the existing delete path is retained and no public APIs were changed.

Changes

Cohort / File(s) Summary
Scheduler - helper added & error handling updated
autogpt_platform/backend/backend/executor/scheduler.py
Added private async helper _cleanup_old_schedules_without_id(graph_id: str, user_id: str) that finds and deletes schedules with schedule_id is None. Updated _handle_graph_validation_error to log a warning and invoke this helper for schedules missing schedule_id; existing deletion path remains when a schedule_id is present.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~10 minutes

Poem

🐰 I sniffed out schedules lost in the mist,
I nibbled the None-IDs off the list,
A gentle warning, then tidy and spry,
Old ghosts removed — hop, clean, and goodbye! 🥕✨

🚥 Pre-merge checks | ✅ 2 | ❌ 1
❌ Failed checks (1 warning)
Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 60.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (2 passed)
Check name Status Explanation
Title check ✅ Passed The title 'fix(backend): Clean up orphaned schedules without schedule_id' clearly and concisely describes the main change: fixing an issue by cleaning up orphaned schedules that lack a schedule_id.
Description check ✅ Passed The PR description clearly relates to the changeset, detailing the problem with orphaned schedules and explaining the fix implemented in the scheduler code.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing touches
  • 📝 Generate docstrings
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Post copyable unit tests in a comment
  • Commit unit tests in branch fix/scheduler-orphaned-schedule-cleanup

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

Comment thread autogpt_platform/backend/backend/executor/scheduler.py Outdated
Old scheduled jobs created before schedule_id was added to
GraphExecutionJobArgs have schedule_id=None. When these fail
validation, _handle_graph_validation_error could not unschedule
them, causing them to fire repeatedly and generate ~60K+ Sentry
errors (AUTOGPT-SERVER-6W2 and AUTOGPT-SERVER-6W3).

Fix: Add _cleanup_old_schedules_without_id() which finds schedules
for the graph but only removes those with schedule_id=None (legacy
jobs). This preserves any valid newer schedules the user may have
created, unlike the broader _cleanup_orphaned_schedules_for_graph()
which removes all schedules for a graph.
@Bentlybro
Bentlybro force-pushed the fix/scheduler-orphaned-schedule-cleanup branch from 49f2fe6 to 6b1f0df Compare February 2, 2026 14:15
@github-actions github-actions Bot added the size/m label Feb 2, 2026
@Bentlybro
Bentlybro requested a review from Pwuts February 2, 2026 14:55
@Otto-AGPT
Otto-AGPT requested review from ntindle and removed request for Otto-AGPT February 6, 2026 17:21
ntindle
ntindle previously approved these changes Feb 6, 2026

@ntindle ntindle left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved ✅

Disclosure: I'm an AI agent (Claude) acting on behalf of @ntindle.

Review Process

  1. Human code review: @ntindle reviewed the code changes (per AutoGPT's humans-must-read-code policy)
  2. Automated testing: I was asked to verify the changes and validate against Sentry data
  3. Explicit approval direction: After reviewing my findings, @ntindle directed me to approve

Verification

Sentry issues confirmed:

  • AUTOGPT-SERVER-6W2: 32,287 events — "Unable to unschedule graph... no associated schedule_id please remove manually"
  • AUTOGPT-SERVER-6W3: 32,287 events — "Scheduled Graph... failed validation"
  • Both still firing as of 5 minutes ago
  • Combined: ~65K events since Nov 17, 2025

Code review:

  • ✅ New _cleanup_old_schedules_without_id() correctly filters by schedule_id is None
  • ✅ Preserves valid newer schedules (only removes legacy orphaned ones)
  • ✅ Proper error handling with try/except and logging
  • ✅ Uses correct ID for deletion (schedule.id, not schedule.schedule_id)
  • ✅ Clear docstring explaining difference from existing cleanup function

Design rationale verified:

  • Existing _cleanup_orphaned_schedules_for_graph() deletes ALL schedules (for deleted graphs)
  • New function is surgical — only targets schedule_id=None jobs while preserving valid ones

Conclusion

This fix will eliminate ~65K recurring Sentry errors from orphaned schedules that have been firing since November 2025.

— Claude (AI agent, approved at @ntindle's direction)

@github-project-automation github-project-automation Bot moved this from 🆕 Needs initial review to 👍🏼 Mergeable in AutoGPT development kanban Feb 6, 2026
majdyz
majdyz previously approved these changes Feb 8, 2026

@majdyz majdyz left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is fine kind of, but what's blocking us on simply doing a one off cleanup on the database instead of doing this ?

Comment thread autogpt_platform/backend/backend/executor/scheduler.py

@autogpt-reviewer autogpt-reviewer left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PR #11927 — fix(backend): Clean up orphaned schedules without schedule_id

Author: Bentlybro | Files: scheduler.py (+33/−2) | CI: ✅ All green (tests 3.11/3.12/3.13, types, lint, CodeQL, security)


🎯 Verdict: APPROVE


What This PR Does

Adds a new helper _cleanup_old_schedules_without_id() that surgically removes legacy scheduled jobs (those with schedule_id=None) while preserving valid newer schedules. This fixes ~65K recurring Sentry errors (AUTOGPT-SERVER-6W2 + 6W3) from orphaned jobs created before schedule_id was added to GraphExecutionJobArgs. These jobs fire on cron, fail validation, log errors, and repeat — forever. The fix is self-healing: each orphan fires one more time, gets cleaned up, and never fires again.


Specialist Findings

🛡️ Security ✅ — No concerns. Authorization enforced at service layer (user_id check in delete_graph_execution_schedule). No cross-user deletion risk. Parameters come from internal job args, not external input. No info leakage (logs only).

🏗️ Architecture ✅ — New function is the right approach. Different invariants from existing _cleanup_orphaned_schedules_for_graph (which nukes ALL schedules for deleted graphs). Separation of concerns is clean. Minor duplication could be extracted to a shared helper, but tolerable at this scale.

Performance ✅ — Self-limiting cleanup path. Full jobstore scan is pre-existing (APScheduler API limitation, not introduced by this PR). Sequential deletes are acceptable given expected cardinality ≈1. Each orphan triggers cleanup at most once before being removed.

🧪 Testing ⚠️ — No tests included, but the entire scheduler module has zero test coverage (systemic issue). Function is simple enough for high confidence by inspection. Should add tests (mixed schedule filtering, deletion failure continuation) but not a merge blocker for this fix.

📖 Quality ✅ — Clean code. Good naming (matches _cleanup_* convention). Clear docstring explaining difference from sibling function. Correct log level change (errorwarning). Error handling matches existing patterns (per-schedule try/except so one failure doesn't block others).

📦 Product ✅ — Safe for users. Deleted schedules were already permanently broken (fail validation on every fire, never execute successfully). Silent removal is appropriate — notifying users about forgotten legacy schedules would create more confusion than value. Self-healing approach is product-appropriate.

📬 Discussion ✅ — 2 approvals (ntindle, majdyz). majdyz's collateral deletion concern was correctly rebutted by Bentlybro (schedule.schedule_id vs schedule.id distinction). One-off DB cleanup question acknowledged but not blocking — majdyz approved despite it.

🔎 QA ✅ — Live testing passed. Frontend loads normally (landing, login, signup, dashboard, build page, marketplace). Backend healthy. Schedule Run button present and accessible. 7 screenshots captured. No regressions.

QA Screenshots:


Blockers

None.

Should Fix (Follow-up OK)

  1. scheduler.py — Add unit tests for _cleanup_old_schedules_without_id: mixed schedule filtering, deletion failure continuation, empty schedule list. (Systemic gap — entire scheduler module is untested.)
  2. scheduler.py:220-270 — Extract shared delete-iterate-log helper to reduce duplication between _cleanup_orphaned_schedules_for_graph and the new function.
  3. Companion one-off DB cleanup — Consider a one-time DELETE against orphaned APScheduler jobs for immediate Sentry noise reduction. The runtime fix only cleans each orphan on its next cron fire, which could take hours/weeks depending on cron expressions.
  4. scheduler.py:196 — Add user_id to warning message (partially done — already includes it, but qodo suggested more context like job identifier).

Risk Assessment

Merge risk: LOW | Rollback: EASY (revert removes cleanup; orphans resume failing harmlessly as before)

@ntindle Clean, well-scoped fix that eliminates ~65K Sentry errors. Self-healing approach is correct. No blockers — approve and merge.

@github-actions

Copy link
Copy Markdown
Contributor

This pull request has conflicts with the base branch, please resolve those so we can evaluate the pull request.

@github-actions github-actions Bot added the conflicts Automatically applied to PRs with merge conflicts label Mar 16, 2026
@CLAassistant

CLAassistant commented May 11, 2026

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

@Pwuts
Pwuts dismissed stale reviews from majdyz and ntindle via 5eb9b55 July 21, 2026 12:16
@github-actions github-actions Bot removed the conflicts Automatically applied to PRs with merge conflicts label Jul 21, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Conflicts have been resolved! 🎉 A maintainer will review the pull request shortly.

@github-actions github-actions Bot removed the size/s label Jul 21, 2026
@github-actions

github-actions Bot commented Jul 21, 2026

Copy link
Copy Markdown
Contributor

🔍 PR Overlap Detection

This check compares your PR against all other open PRs targeting the same branch to detect potential merge conflicts early.

🟢 Low Risk — File Overlap Only

These PRs touch the same files but different sections (click to expand)

Summary: 0 conflict(s), 0 medium risk, 1 low risk (out of 1 PRs with file overlap)


Auto-generated on push. Ignores: openapi.json, lock files.

Comment thread autogpt_platform/backend/backend/executor/scheduler.py Outdated
Pwuts
Pwuts previously approved these changes Jul 21, 2026
@codecov

codecov Bot commented Jul 21, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 76.38%. Comparing base (44a9d57) to head (12ae0be).
⚠️ Report is 4 commits behind head on dev.

Additional details and impacted files
@@            Coverage Diff             @@
##              dev   #11927      +/-   ##
==========================================
+ Coverage   76.33%   76.38%   +0.04%     
==========================================
  Files        2701     2703       +2     
  Lines      206224   206911     +687     
  Branches    19777    19818      +41     
==========================================
+ Hits       157417   158039     +622     
- Misses      44444    44508      +64     
- Partials     4363     4364       +1     
Flag Coverage Δ
platform-backend 83.15% <100.00%> (+0.04%) ⬆️
platform-frontend-e2e 30.93% <ø> (-0.22%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components Coverage Δ
Platform Backend 83.15% <100.00%> (+0.04%) ⬆️
Platform Frontend 50.21% <ø> (-0.08%) ⬇️
AutoGPT Libs ∅ <ø> (∅)
Classic AutoGPT 28.43% <ø> (ø)
🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Comment thread autogpt_platform/backend/backend/executor/scheduler.py Outdated
@Bentlybro
Bentlybro added this pull request to the merge queue Jul 22, 2026
Merged via the queue into dev with commit 65e3f48 Jul 22, 2026
41 of 42 checks passed
@Bentlybro
Bentlybro deleted the fix/scheduler-orphaned-schedule-cleanup branch July 22, 2026 09:41
@github-project-automation github-project-automation Bot moved this from 👍🏼 Mergeable to ✅ Done in AutoGPT development kanban Jul 22, 2026
@sentry

sentry Bot commented Aug 1, 2026

Copy link
Copy Markdown

Issues attributed to commits in this pull request

This pull request was merged and Sentry observed the following issues:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

Status: ✅ Done

Development

Successfully merging this pull request may close these issues.

6 participants