workflow object not found maybe data is expired #1576
Description
当我从1.3.1版本升级到2.0.2后,我在运行工作流的时候出现错误:workflow object not found maybe data is expired。 但是这个错误也不是每次都会遇到,多重试几次会有一次成功运行。
后端具体报错信息如下:
[2025-08-26 10:28:36,580: WARNING/MainProcess] [2025-08-26 10:28:36.569073] [ERROR process-34032-138594811364928 bisheng.worker.workflow.tasks:98] - trace=ae9f0910b5b44de791e42da497d4f5d8 continue_workflow error
Traceback (most recent call last):
File "/root/anaconda3/envs/BiShengVENV/lib/python3.10/threading.py", line 973, in _bootstrap
self._bootstrap_inner()
| -> <function Thread._bootstrap_inner at 0x7e0f266e1c60>
-> <Thread(ThreadPoolExecutor-3_0, started 138594811364928)>
File "/root/anaconda3/envs/BiShengVENV/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
self.run()
| -> <function Thread.run at 0x7e0f266e1990>
-> <Thread(ThreadPoolExecutor-3_0, started 138594811364928)>
File "/root/anaconda3/envs/BiShengVENV/lib/python3.10/threading.py", line 953, in run
self._target(*self._args, **self._kwargs)
| | | | | -> {}
| | | | -> <Thread(ThreadPoolExecutor-3_0, started 138594811364928)>
| | | -> (<weakref at 0x7e0d8825cf90; to 'ThreadPoolExecutor' at 0x7e0d884cebf0>, <_queue.SimpleQueue object at 0x7e0d884f0770>, None,...
| | -> <Thread(ThreadPoolExecutor-3_0, started 138594811364928)>
| -> <function _worker at 0x7e0f25bb57e0>
-> <Thread(ThreadPoolExecutor-3_0, started 138594811364928)>
File "/root/anaconda3/envs/BiShengVENV/lib/python3.10/concurrent/futures/thread.py", line 83, in _worker
work_item.run()
| -> <function _WorkItem.run at 0x7e0f25bb5900>
-> <concurrent.futures.thread._WorkItem object at 0x7e0d78543c40>
File "/root/anaconda3/envs/BiShengVENV/lib/python3.10/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
| | | | | -> {}
| | | | -> <concurrent.futures.thread._WorkItem object at 0x7e0d78543c40>
| | | -> (<function fast_trace_task at 0x7e0f24c94040>, ('bisheng.worker.workflow.tasks.continue_workflow', '4e8800cf-4f33-407b-8d2b-a...
| | -> <concurrent.futures.thread._WorkItem object at 0x7e0d78543c40>
| -> <function apply_target at 0x7e0f25bb5e10>
-> <concurrent.futures.thread._WorkItem object at 0x7e0d78543c40>
File "/root/anaconda3/envs/BiShengVENV/lib/python3.10/site-packages/celery/concurrency/base.py", line 30, in apply_target
ret = target(*args, **kwargs)
| | -> {}
| -> ('bisheng.worker.workflow.tasks.continue_workflow', '4e8800cf-4f33-407b-8d2b-ad4666054416', {'lang': 'py', 'task': 'bisheng.w...
-> <function fast_trace_task at 0x7e0f24c94040>
File "/root/anaconda3/envs/BiShengVENV/lib/python3.10/site-packages/celery/app/trace.py", line 651, in fast_trace_task
R, I, T, Rstr = tasks.trace(
| -> 'bisheng.worker.workflow.tasks.continue_workflow'
-> {'celery.group': <@task: celery.group of bisheng at 0x7e0e6cf9dc90>, 'bisheng.worker.knowledge.file_worker.file_copy_celery':...
File "/root/anaconda3/envs/BiShengVENV/lib/python3.10/site-packages/celery/app/trace.py", line 453, in trace_task
R = retval = fun(*args, **kwargs)
| | -> {}
| -> ['ae9f0910b5b44de791e42da497d4f5d8', '1ea1c1282aac4d9bbf3280320bf7f792', '', '1']
-> <@task: bisheng.worker.workflow.tasks.continue_workflow of bisheng at 0x7e0e6cf9dc90>
File "/root/anaconda3/envs/BiShengVENV/lib/python3.10/site-packages/celery/app/trace.py", line 736, in protected_call
return self.run(*args, **kwargs)
| | | -> {}
| | -> ('ae9f0910b5b44de791e42da497d4f5d8', '1ea1c1282aac4d9bbf3280320bf7f792', '', '1')
| -> <staticmethod(<function continue_workflow at 0x7e0da1a83d00>)>
-> <@task: bisheng.worker.workflow.tasks.continue_workflow of bisheng at 0x7e0e6cf9dc90>
File "/workspaces/bisheng/src/backend/bisheng/worker/workflow/tasks.py", line 107, in continue_workflow
_continue_workflow(unique_id, workflow_id, chat_id, user_id)
| | | | -> '1'
| | | -> ''
| | -> '1ea1c1282aac4d9bbf3280320bf7f792'
| -> 'ae9f0910b5b44de791e42da497d4f5d8'
-> <function _continue_workflow at 0x7e0da1a83c70>
File "/workspaces/bisheng/src/backend/bisheng/worker/workflow/tasks.py", line 84, in _continue_workflow
raise Exception('workflow object not found maybe data is expired')
Exception: workflow object not found maybe data is expired
Activity
Vincent990413 commentedon Aug 27, 2025
是的,您好,我也曾遇到过这个问题很多次。目前不太清楚是否与 celery 的工作流队列有关,或者是模型的质量问题有关,期待有人能发现对应的问题和解决方案。
Yes, hello. I have also encountered this problem many times. Currently, it's not clear whether it's related to celery's workflow queue or the quality of the model. I hope someone can find the corresponding problem and solution.
zgqgit commentedon Aug 27, 2025
1、确保工作流的执行worker只有一个进程在运行
2、工作流运行期间worker重启会导致这个问题发生
Vincent990413 commentedon Aug 27, 2025
zgqgit commentedon Aug 27, 2025
应该是好的,是否在工作流执行期间重启过celery进程?
Vincent990413 commentedon Aug 27, 2025
Vincent990413 commentedon Aug 29, 2025
您好作者,我想知道,如何遇到这个问题,如果是docker运行的,应该如何解决呢?是需要重新docker-compose down然后up吗?之后如何去避免这个问题呢?我是使用Docker运行的。
yogboot commentedon Sep 3, 2025
我用的 celery -A bisheng.worker.main worker -l info --pool=solo -Q workflow_celery,knowledge_celery,celery,这个命令,没有重启也偶尔会出现这个问题,官方能优化下不
zgqgit commentedon Sep 3, 2025
目前工作流的运行时都是维护在内存中的,所以如果你的工作流运行到了输入节点时。此时无论什么原因导致的worker服务重启都会出现这个报错。后边版本会考虑将工作流的运行时持久化存储,这样就可以避免这个问题了
Vincent990413 commentedon Sep 3, 2025
zgqgit commentedon Sep 3, 2025
是的
yogboot commentedon Sep 5, 2025
好的,期待^-^