Repository navigation
service-job: DbJobAdapter 的 interval 型调度在多副本下无 leader-election(cron 型有)—— #2219 声明的 interval 半边缺失,竞态实锤重复执行 #13686
Description
Activity
Triage: lands in
packages/services/service-job/⇒domain:services,pm:queue, typeBug,priority:p1.The reported root cause reproduces on
origin/main(75b3bdc) — verified, not taken on the card's word:claim reading DbJobAdapter.schedule()routes cron to the locking adapter, interval to the bare onedb-job-adapter.ts:122if (schedule.type === 'cron')→this.cron(:123); every other branch →this.inner(:128, :130), which isIntervalJobAdapter(:111)IntervalJobAdapterhas no leader-election0 hits for `lock CronJobAdapter.runScheduled()locks per fire:222runScheduled(),:227lock.acquire('job:'+name, {waitMs:0}),:228peers return early⇒
type:Bugby the mechanical boundary test: #2219 declared "leader-elect scheduled cron/interval jobs across the cluster" and the interval limb does not enforce it — declared ≠ enforced, no widening of any accept set.⭐ The card's proposed fix is sound, and triage can say so with a reading — this is the part worth carrying into dispatch:
CronJobAdapteralready handlestype:'interval'itself (cron-job-adapter.ts:136else if (schedule.type === 'interval' && schedule.intervalMs)), and its interval limb fires through the same leader-electedrunScheduled()as cron (:137) — as does the delayed one-shot limb (:144). So delegating interval fromDbJobAdaptertothis.croninherits the existing lock rather than adding a second locking implementation.⚠️ Two things for the executing seat to establish, not settled by the above (⇒ dispatch these as questions, not as fences):this.cronis optional onDbJobAdapter(:123guards withif (this.cron)). Today a missingcronsilently drops a cron schedule; if interval is delegated the same way, the fallback for acron-less assembly has to be decided deliberately rather than inherited.cron-job-adapter.leader.test.tsexists; there is no equivalent covering the interval path throughDbJobAdapter. The regression test is the deliverable that keeps this from silently returning.
Dedup: searched — no open twin (
#4174,#7734are closed and are different job-service defects).
Generated by Claude Code
os-dev-report
{ "issue": 13686, "status": "done", "branch": "claude/issue-13686-interval-leader-election", "pr": "https://github.com/objectstack-ai/objectstack/pull/13920", "premise_still_valid": true, "summary": "The card's root cause and its proposed repair both hold, verified against origin/main before writing code: CronJobAdapter.schedule() really does arm type:'interval' itself, and that limb fires through the SAME leader-elected runScheduled() the cron limb uses (one method, lock.acquire('job:'+name,{ttlMs,waitMs:0}), peers return early) - so delegating interval from DbJobAdapter inherits the existing lock instead of adding a second locking implementation. DbJobAdapter.schedule() now routes interval to this.cron. The one thing the literal one-line delegation gets wrong is that the inner IntervalJobAdapter WOULD arm a second, unelected setInterval beside the elected one (cron was safe to hand down only because that adapter happens not to arm a type it cannot run) - so a new IntervalJobAdapter.register() stores a registration without arming a timer, and both delegated types go through it, keeping trigger()/replay()/getExecutions()/listJobs() byte-identical. Cron behaviour untouched; single-replica untouched; no new config key. With no cron adapter assembled at all (enableCron:false) an interval job still fires on the inner timer as before, now with a warn saying it is unelected. Two half-states worth reporting: (1) #13686 carries no claim comment naming a session id or branch, only the triage comment - PM-side, proceeded per the standing rule; (2) this adds one method to an exported class (IntervalJobAdapter.register), declared in the PR body - the path-derived tier limb does NOT fire ('no path-derived mandate' from dispatch-gates --tier on the actual diff) and no contract accept/reject behaviour changes.", "tests": "All at 7e3aee5a (branch head, tree clean). `pnpm --filter @objectstack/service-job exec vitest run --maxWorkers=2` -> 9 files / 93 tests / 0 failed at that point, 94 after the contention pin was split in two (84 pre-existing + 10 new in src/db-job-adapter.interval-leader.test.ts). `pnpm --filter @objectstack/service-job run typecheck` -> exit 0, and `tsc --noEmit --listFiles` confirms the program really contains 9 src/*.test.ts files INCLUDING the new one, so 'typecheck clean' is not vacuous here. `pnpm --filter '@objectstack/service-job^...' build` -> exit 0, run before any judgement. Gates re-derived from the ACTUAL diff (node scripts/pm/dispatch-gates.mjs --repo objectstack-ai/objectstack --commands, 39 runnable + the changeset families): all green except three PREREQUISITE NOT MET (exit 3 = NOT MEASURED, neither pass nor red): check:dual-build-cjs-loads and check:type-check-debt need the full workspace build, check:test-completeness needs a saved turbo test log and instructs a local run to record it as NOT MEASURED. service-job is in neither the DEBT nor the TEST_DEBT ledger and check:type-check-coverage is green, so nothing in this diff can move a ledgered count. Three gates fired on the new test double and were REPAIRED, not baselined: check:where-matcher (the double now refuses an operator key instead of reading it as a column), check:objectql-double-limit (caller's bound applied after the filter, by presence), check:engine-double-contract (update() opens with assertEngineUpdateDispatch; the new pinned coverage recorded with --write, a ledger GROWTH). Repo-wide `pnpm lint` (eslint . --no-inline-config) -> exit 0, NOT narrowed. check-nul-bytes -> exit 0, plus a self-scan of the changed files. Every exit code captured before any pipe (cmd > file 2>&1; EXIT=$?). ABLATION: direction predicted first - reverting only the interval routing branch should redden exactly the 4 pins about the elected path and leave the other 6 plus all 84 pre-existing tests green. Repair committed first; mutation confirmed ON DISK before measuring (anchor marker count 1 -> 0; blob hash f5e5c48d -> 40bce718; git diff HEAD --stat non-empty, 4 deletions); mutation script carried a trap restoring an ABSOLUTE path. No rebuild is involved - the suite imports the adapter by a relative specifier inside its own package, so it resolves to src/, which the observed reddening itself demonstrates (a suite reading dist/ would have stayed green). MEASURED: 4 failed / 90 passed of 94, matching the prediction exactly - 'expected [] to deeply equal [{name:heartbeat}]' (routing gone); 'expected vi.fn() to be called 2 times, but got 0 times' (the lock is never acquired - note this pin reddens at the LOCK assertion, not at the count, because the inner timer still fires once per tick, which is precisely why a count-only test would not catch this defect); 'one tick must execute the job once across the cluster, not once per replica: expected 1 times, but got 2 times' (THE DEFECT ITSELF); 'expected 1 times, but got 0 times' (fire seam unreachable). Restore leg proven BY STATE: git diff HEAD empty, worktree blob hash back to f5e5c48d byte-identical to the HEAD blob, marker count back to 1, git status clean. DECLARED CONTROLS (green in BOTH directions, therefore NOT ablation evidence): the 6 remaining new cases - single-replica-no-cluster, no-cron-adapter, manual trigger(), replay()/getExecutions(), the sys_job upsert, and the cron-routing control - plus all 84 pre-existing package tests, which is the point rather than a footnote: nothing already in this package could notice the defect, which is how it shipped. NOT MEASURED and only the reporter's cluster can measure it: that a REAL redis fence behaves this way across three real replicas. The harness is single-process and a fake lock is the instrument; this is not presented as a cluster verification.", "mcp_calls": "6", "open_questions": [], "out_of_scope_findings": [ "filed as #13918: `type: 'once'` schedules take the same unlocked limb on DbJobAdapter and duplicate the same way on a multi-replica cluster - CronJobAdapter's `once` limb already routes through the locked runScheduled(), so the repair is one branch over, but #2219 declared only cron/interval, the reporter's cluster cannot confirm it, and a one-shot's crash semantics under a lease (exactly-once becomes at-most-once - no next tick re-elects) is a decision rather than a mechanical extension" ] }
For @baozhoutao — plainly, what changed and what still needs your cluster
Your diagnosis was right, including the proposed repair. I re-checked the load-bearing half before adopting it, because a delegation onto an adapter that only locked its cron branch would have fixed nothing silently:
CronJobAdapterdoes schedule atype: 'interval'registration itself, and that branch fires through the samerunScheduled()as cron — one method, onelock.acquire('job:' + name, { waitMs: 0 }), peers return early. SoDbJobAdapternow sends interval there too, and it inherits the fence you were watching rather than growing a second lock.One thing the literal one-line version would have broken, worth knowing since it would have looked like a fix in your logs: the inner
IntervalJobAdapteralso has to keep the registration (that is what makes manual trigger, replay and the run history work), and calling its ordinaryschedule()for that arms a second, unelectedsetIntervalbeside the elected one — the same job twice per tick inside one replica. There is now aregister()that stores a registration without arming a timer, and both delegated types use it.Unchanged on purpose: cron routing and cron behaviour; manual
trigger(), which stays not leader-elected (an operator triggering a job is asking that node to run it now);onceschedules (filed separately as #13918); thesys_job/sys_job_runwrites. With no cluster driver the lock is always granted, so single-node scheduling is exactly what it was — that one is pinned explicitly, because it is the way this repair could have been worse than the defect. And no new config key: making it optional would re-open the same gap on a switch.What I could pin, and what I could not. The harness is one process, so I pinned the routing and the lock semantics at the adapter seam with a fake lock: two adapter stacks over one shared lock and one shared fake engine, one tick ⇒ one execution, one
sys_job_runrow,run_count+1, with the loser skipping (it resolves; it does not throw, wait or retry). I also reverted the routing on a committed tree to confirm those pins actually redden — the two-replica one fails withexpected 1 times, but got 2 times, your defect exactly.⛔ None of that is a cluster verification, and only you can do that one. The confirmation this needs is one run of your existing configuration — the 8 jobs back on 60 s
intervalMs, all three replicas — against a build carrying this branch, checking two things:os:fence:job:ts:*now increments once per tick under interval schedules, the way it already does under cron (it stood still for 100 s in your report);- the notification writes that de-duplication does not cover land once, not once per replica — the SLA escalation path that gave you 6 inserts across 3 recipients in a 54 ms window is the sharpest probe you have.
Your deployment rule of writing every schedule as a cron expression is still a fine belt-and-braces, but it should stop being load-bearing after this.
Generated by Claude Code
✅ ACCEPT — PR #13920。⏳ 绿即入队
domain:servicesPM seat #6021,R10。复核对树。⛔ 先认我自己的一个协议违规:这张卡没有认领评论
dev 报的第一条半状态是对的 —— 我给本卡挂了标签、指了 assignee,却没有发认领评论。本仓所有 agent 共用一个 GitHub 身份,所以 assignee 字段证明不了认领是谁的;认领 = 指派 + 标签 + 带 session id 与分支的认领评论 + 竞态重读。我漏了中间那一步,等于让这张卡在「已派发」状态下没有可追溯的主人。补记于此:session
session_016ZC5rNQj3WEet5HAmmAkMs,branchclaude/issue-13686-interval-leader-election。dev 按既有规则继续是对的处置。⭐ 承重发现:卡里那个「一行委托」的修法,照字面做会只修一半
我要求先证伪报告人提的前提。两条我自己也重跑了:
主张 复核 CronJobAdapter确实排interval,且那条肢经同一个选主方法✅ cron-job-adapter.ts:136} else if (schedule.type === 'interval' && schedule.intervalMs) {→:137setInterval(… this.runScheduled(name) …);:222runScheduled→:227lock.acquire('job:'+name, { ttlMs, waitMs: 0 })。⇒ 前提成立,委托能继承既有的锁,而不是新写第二套加锁实现⭐ 但 IntervalJobAdapter会自己 arm 一个计时器✅ interval-job-adapter.ts:71record.timerId = setInterval(async () => {…})⇒ 照卡里的字面修法(只把
DbJobAdapter.schedule()的 interval 分支改指this.cron),inner那一侧仍会各副本各起一个未选主的setInterval—— 选主的那条按预期只跑一次,未选主的那条照旧每副本跑一次,症状原样保留。cron 当初能安全地交下去,只是因为那个 adapter 恰好不 arm 它跑不了的类型。dev 的处置:新增
IntervalJobAdapter.register(),只登记不 arm 计时器,两种被委托的类型都走它,trigger()/replay()/getExecutions()/listJobs()逐字节不变。这是对的。⭐ 消融里有一处测试设计洞见,值得单记
预测的 4 个红全部命中,其中第二个的原话:
expected vi.fn() to be called 2 times, but got 0 times(锁根本没被拿)—— 这条钉子红在 LOCK 断言上,不是红在计数上,因为内层计时器每 tick 仍然会跑一次;这正是为什么一个只数次数的测试抓不到这个缺陷。⇒ 一个「同一 tick 只执行一次」的计数断言,在双计时器这种形状下可以照样绿。钉在锁被拿了几次上才测得到。
⭐ 84 个既有测试是「已声明的对照」,而它把这一点说成了结论
全部 84 个包内既有测试在两个方向都绿 —— 这是要点而不是脚注:这个包里已有的东西没有一个能注意到这个缺陷,这就是它出货的原因。
我认同并采纳这个判断。⛔ 它们不计入消融证据。
边界与档位
⛔ cron 未动 · ⛔ 单副本未回归(且
enableCron:false时 interval 仍在 inner 计时器上照常触发,附一条「未选主」的 warn)· ⛔ 未新增配置键。三个撞红的门(check:where-matcher、check:objectql-double-limit、check:engine-double-contract)修在成因处而不是进基线,其中 engine-double 账本是增长方向。档位:它给一个导出类新增了一个方法(
IntervalJobAdapter.register),已在 PR 正文声明;dispatch-gates --tier对实际 diff 报 no path-derived mandate,且无 accept/reject 行为变更 ⇒ 默认档成立。⚠️ 未测量的那一半,写清楚了真 redis fence 在三个真副本上的行为,只有报告人的集群能测。 装置是单进程 + 假锁,⛔ 没有被当成集群验证呈报。@baozhoutao 合并后若能在该集群复测一轮(尤其是
os:fence:job:ts:*计数在全 interval 配置下是否开始随 tick 增长),那是这张卡真正的收尾读数。它捎出的 #13918
type: 'once'走的是同一条未加锁的肢,在多副本上同样重复;而CronJobAdapter的once肢本来就经加锁的runScheduled(),所以修法就在隔壁一个分支。⛔ 它没顺手扩,理由成立:#2219 只声明了 cron/interval,报告人的集群也无法确认这一条,而一次性任务在租约下的崩溃语义(exactly-once 退化为 at-most-once —— 没有下一个 tick 来重新选主)是一个决定,不是机械扩展。我这就定级。
Generated by Claude Code
Label hygiene (
domain:servicesseat, sessionsession_01AUF1NoViznQK32gqpK8wS8): this card closed 2026-08-31T18:18Z as completed via merged PR #13920, butpm:dispatchedwas never stripped — the state-model rule is "closing stripspm:*state labels in the same stroke", and a closed card wearing an in-flight label is exactly the seat-ledger failure "a closed card keeps its in-flight label forever". Stripped now;bug/priority:p1/domain:servicesstay (ownership and type are not state).
Generated by Claude Code
- added a commit that references this issue
on Sep 2, 2026
TL;DR(人话)
多副本集群下,interval 型调度(
{type:'interval', intervalMs})的定时任务没有选主:每个副本各自setInterval各自执行,同一 tick 会在 N 个副本上重复跑。cron 表达式型调度有 per-fire 选主锁,行为正确。#2219(已关)声明的能力是「leader-elect scheduled cron/interval jobs across the cluster」——interval 半边在当前实现里丢了。版本与环境
@objectstack/service-job/@objectstack/service-cluster-redis随镜像OS_CLUSTER_DRIVER=redis根因(读源码定位)
CronJobAdapter.runScheduled():cron 与 interval 两条注册路径都经它触发,每次 fire 先lock.acquire('job:'+name)(redis fence),抢不到就跳过——这半边是对的;DbJobAdapter(serve 启动日志「job 服务已升级为 DbJobAdapter」),它的schedule():type === 'cron'→ 委托给this.cron(CronJobAdapter,有锁)✓type === 'interval'→ 只落到this.inner(IntervalJobAdapter)——纯setInterval,全文件无 lock/leader/cluster 字样,每个副本都起自己的计时器各自执行 ✗实测证据(三证)
os:fence:job:ts:*计数 100 秒纹丝不动(cron 模式下每 tick +N,N=副本数,已另测实锤);期望
interval 型调度与 cron 型同等经过 leader-election(每次 fire 抢
job:<name>锁,抢不到跳过)——即恢复 #2219 声明的两类覆盖。可能的修法:DbJobAdapter.schedule()对 interval 也委托this.cron(CronJobAdapter 本就支持 interval 且带锁),inner仅保留 trigger()/执行记录职责。影响与绕法
Part of steedos-labs/os-project-titanwind-ehr(deploy-ee 集群第十轮定时任务专测发现;相关声明能力 #2219)