Skip to content

Chain don't get re-scheduled after being killed with notify_chain_stop() #789

Description

@0xgouda

Describe the bug
We have many chains with max_instances := 1, sometimes we want to disable one chain and kill the currently active execution of it - because it takes long to finish -, we use select timetable.notify_stop_chain(chain_id, worker_name);

We observe that the chain has stopped executing, but active_chain table still doesn't remove the old killed execution and hence after re-activating the chain subsequent runs keep logging Cannot proceed. Sleeping as they see the chain active.

To Reproduce
Steps to reproduce the behavior:

  1. Add the chain: SELECT timetable.add_job('test-finished-last_run', '* * * * *', 'select pg_sleep(300)', job_max_instances := 1);
  2. Inspect timetable.active_chain
  3. When the chain is active, kill it with select timetable.notify_chain_stop(chain_id, worker_name);
  4. observe that timetable.active_chain won't change at all
  5. see Cannot proceed, sleeping log messages in pg_timetable logs

Expected behavior
pg_timetable should correctly update timetable.active_chain when a chain gets killed by notify_chain_stop()

Screenshots

postgres=# SELECT timetable.add_job('test-finished-last_run', '* * * * *', 'select pg_sleep(300)', job_max_instances := 1);
 add_job
---------
      15
(1 row)

postgres=# select * from timetable.active_chain ;
 chain_id | client_name  |          started_at
----------+--------------+------------------------------
       15 | pg_timetable | 2026-07-10 17:22:31.32798+03
(1 row)

postgres=# select timetable.notify_chain_stop(15, 'pg_timetable');
 notify_chain_stop
-------------------

(1 row)

postgres=# update timetable.chain set live = false where chain_id = 15 ;
UPDATE 1
postgres=# update timetable.chain set live = true where chain_id = 15 ;
UPDATE 1
postgres=# select * from timetable.active_chain ;
 chain_id | client_name  |          started_at
----------+--------------+------------------------------
       15 | pg_timetable | 2026-07-10 17:22:31.32798+03
(1 row)

---------------------------------------------------------------

2026-07-10 17:22:31.329 [INFO] [chain:15] [task:14] [vxid:25769804593] Starting task
2026-07-10 17:22:31.331 [INFO] [chain:15] [task:14] [vxid:25769804593] Switching to autonomous task mode
2026-07-10 17:22:31.335 [INFO] [chain:1] [task:1] [vxid:30064771470] Starting task
2026-07-10 17:22:31.341 [INFO] [chain:1] [task:1] [vxid:30064771470] {"Description": "Log Execution"}
2026-07-10 17:22:31.348 [INFO] [chain:1] [task:1] [vxid:30064771470] Task executed successfully
2026-07-10 17:22:31.348 [INFO] [chain:1] [vxid:30064771470] Chain executed successfully
2026-07-10 17:23:31.319 [INFO] [notification:{119301 pg_timetable {"ConfigID": 15, "Command": "STOP", "Ts": 1783693360}}] [pid:155156] [signal:{15 STOP 1783693360 0}] Adding asynchronous chain to working queue
2026-07-10 17:23:31.319 [ERROR] [chain:15] [task:14] [sql:select pg_sleep(300)] [args:[]] [err:context canceled] [pid:155157] [time:59.988433701s] [vxid:25769804593] Query
2026-07-10 17:23:31.320 [INFO] [count:0] Retrieve interval chains to run
2026-07-10 17:23:31.323 [INFO] [count:1] Retrieve scheduled chains to run
2026-07-10 17:23:31.324 [INFO] [chain:1] Starting chain
2026-07-10 17:23:31.326 [INFO] [chain:1] [task:1] [vxid:30064771481] Starting task
2026-07-10 17:23:31.326 [INFO] [chain:15] [task:14] [vxid:25769804593] Closing standalone session
2026-07-10 17:23:31.326 [ERROR] [chain:15] [task:14] [error:context canceled] [vxid:25769804593] Task execution failed
2026-07-10 17:23:31.326 [INFO] [chain:15] [task:14] [vxid:25769804593] Ignoring task failure
2026-07-10 17:23:31.326 [ERROR] [chain:15] [sql:commit] [args:[]] [err:timeout: context already done: context canceled] [pid:155159] [time:2.333µs] [vxid:25769804593] Query
2026-07-10 17:23:31.326 [ERROR] [chain:15] [error:timeout: context already done: context canceled] [vxid:25769804593] Application cannot commit after job finished
2026-07-10 17:23:31.326 [INFO] [chain:15] [vxid:25769804593] Chain executed successfully
2026-07-10 17:23:31.326 [ERROR] [chain:15] [err:context canceled] [time:902ns] [vxid:25769804593] Acquire
2026-07-10 17:23:31.326 [ERROR] [error:context canceled] Cannot save information about the chain run status
2026-07-10 17:23:31.333 [INFO] [chain:1] [task:1] [vxid:30064771481] {"Description": "Log Execution"}
2026-07-10 17:23:31.333 [INFO] [chain:1] [task:1] [vxid:30064771481] Task executed successfully
2026-07-10 17:23:31.333 [INFO] [chain:1] [vxid:30064771481] Chain executed successfully
2026-07-10 17:24:31.321 [INFO] [count:0] Retrieve interval chains to run
2026-07-10 17:24:31.327 [INFO] [count:2] Retrieve scheduled chains to run
2026-07-10 17:24:31.329 [INFO] [chain:15] Cannot proceed. Sleeping

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions