Skip to content

workflow object not found maybe data is expired #1576

Open

Description

@freecae

当我从1.3.1版本升级到2.0.2后,我在运行工作流的时候出现错误:workflow object not found maybe data is expired。 但是这个错误也不是每次都会遇到,多重试几次会有一次成功运行。
后端具体报错信息如下:
[2025-08-26 10:28:36,580: WARNING/MainProcess] [2025-08-26 10:28:36.569073] [ERROR process-34032-138594811364928 bisheng.worker.workflow.tasks:98] - trace=ae9f0910b5b44de791e42da497d4f5d8 continue_workflow error
Traceback (most recent call last):

File "/root/anaconda3/envs/BiShengVENV/lib/python3.10/threading.py", line 973, in _bootstrap
self._bootstrap_inner()
| -> <function Thread._bootstrap_inner at 0x7e0f266e1c60>
-> <Thread(ThreadPoolExecutor-3_0, started 138594811364928)>
File "/root/anaconda3/envs/BiShengVENV/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
self.run()
| -> <function Thread.run at 0x7e0f266e1990>
-> <Thread(ThreadPoolExecutor-3_0, started 138594811364928)>
File "/root/anaconda3/envs/BiShengVENV/lib/python3.10/threading.py", line 953, in run
self._target(*self._args, **self._kwargs)
| | | | | -> {}
| | | | -> <Thread(ThreadPoolExecutor-3_0, started 138594811364928)>
| | | -> (<weakref at 0x7e0d8825cf90; to 'ThreadPoolExecutor' at 0x7e0d884cebf0>, <_queue.SimpleQueue object at 0x7e0d884f0770>, None,...
| | -> <Thread(ThreadPoolExecutor-3_0, started 138594811364928)>
| -> <function _worker at 0x7e0f25bb57e0>
-> <Thread(ThreadPoolExecutor-3_0, started 138594811364928)>
File "/root/anaconda3/envs/BiShengVENV/lib/python3.10/concurrent/futures/thread.py", line 83, in _worker
work_item.run()
| -> <function _WorkItem.run at 0x7e0f25bb5900>
-> <concurrent.futures.thread._WorkItem object at 0x7e0d78543c40>
File "/root/anaconda3/envs/BiShengVENV/lib/python3.10/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
| | | | | -> {}
| | | | -> <concurrent.futures.thread._WorkItem object at 0x7e0d78543c40>
| | | -> (<function fast_trace_task at 0x7e0f24c94040>, ('bisheng.worker.workflow.tasks.continue_workflow', '4e8800cf-4f33-407b-8d2b-a...
| | -> <concurrent.futures.thread._WorkItem object at 0x7e0d78543c40>
| -> <function apply_target at 0x7e0f25bb5e10>
-> <concurrent.futures.thread._WorkItem object at 0x7e0d78543c40>
File "/root/anaconda3/envs/BiShengVENV/lib/python3.10/site-packages/celery/concurrency/base.py", line 30, in apply_target
ret = target(*args, **kwargs)
| | -> {}
| -> ('bisheng.worker.workflow.tasks.continue_workflow', '4e8800cf-4f33-407b-8d2b-ad4666054416', {'lang': 'py', 'task': 'bisheng.w...
-> <function fast_trace_task at 0x7e0f24c94040>
File "/root/anaconda3/envs/BiShengVENV/lib/python3.10/site-packages/celery/app/trace.py", line 651, in fast_trace_task
R, I, T, Rstr = tasks.trace(
| -> 'bisheng.worker.workflow.tasks.continue_workflow'
-> {'celery.group': <@task: celery.group of bisheng at 0x7e0e6cf9dc90>, 'bisheng.worker.knowledge.file_worker.file_copy_celery':...
File "/root/anaconda3/envs/BiShengVENV/lib/python3.10/site-packages/celery/app/trace.py", line 453, in trace_task
R = retval = fun(*args, **kwargs)
| | -> {}
| -> ['ae9f0910b5b44de791e42da497d4f5d8', '1ea1c1282aac4d9bbf3280320bf7f792', '', '1']
-> <@task: bisheng.worker.workflow.tasks.continue_workflow of bisheng at 0x7e0e6cf9dc90>
File "/root/anaconda3/envs/BiShengVENV/lib/python3.10/site-packages/celery/app/trace.py", line 736, in protected_call
return self.run(*args, **kwargs)
| | | -> {}
| | -> ('ae9f0910b5b44de791e42da497d4f5d8', '1ea1c1282aac4d9bbf3280320bf7f792', '', '1')
| -> <staticmethod(<function continue_workflow at 0x7e0da1a83d00>)>
-> <@task: bisheng.worker.workflow.tasks.continue_workflow of bisheng at 0x7e0e6cf9dc90>

File "/workspaces/bisheng/src/backend/bisheng/worker/workflow/tasks.py", line 107, in continue_workflow
_continue_workflow(unique_id, workflow_id, chat_id, user_id)
| | | | -> '1'
| | | -> ''
| | -> '1ea1c1282aac4d9bbf3280320bf7f792'
| -> 'ae9f0910b5b44de791e42da497d4f5d8'
-> <function _continue_workflow at 0x7e0da1a83c70>

File "/workspaces/bisheng/src/backend/bisheng/worker/workflow/tasks.py", line 84, in _continue_workflow
raise Exception('workflow object not found maybe data is expired')

Exception: workflow object not found maybe data is expired

Activity

  1. Vincent990413 commented on Aug 27, 2025

    @Vincent990413

    是的,您好,我也曾遇到过这个问题很多次。目前不太清楚是否与 celery 的工作流队列有关,或者是模型的质量问题有关,期待有人能发现对应的问题和解决方案。
    Yes, hello. I have also encountered this problem many times. Currently, it's not clear whether it's related to celery's workflow queue or the quality of the model. I hope someone can find the corresponding problem and solution.

  2. zgqgit commented on Aug 27, 2025

    @zgqgit
    Collaborator

    1、确保工作流的执行worker只有一个进程在运行
    2、工作流运行期间worker重启会导致这个问题发生

  3. Vincent990413 commented on Aug 27, 2025

    @Vincent990413
  4. zgqgit commented on Aug 27, 2025

    @zgqgit
    Collaborator

    应该是好的,是否在工作流执行期间重启过celery进程?

  5. Vincent990413 commented on Aug 27, 2025

    @Vincent990413
  6. Vincent990413 commented on Aug 29, 2025

    @Vincent990413

    应该是好的,是否在工作流执行期间重启过celery进程?

    您好作者,我想知道,如何遇到这个问题,如果是docker运行的,应该如何解决呢?是需要重新docker-compose down然后up吗?之后如何去避免这个问题呢?我是使用Docker运行的。

  7. yogboot commented on Sep 3, 2025

    @yogboot

    我用的 celery -A bisheng.worker.main worker -l info --pool=solo -Q workflow_celery,knowledge_celery,celery,这个命令,没有重启也偶尔会出现这个问题,官方能优化下不

  8. zgqgit commented on Sep 3, 2025

    @zgqgit
    Collaborator

    目前工作流的运行时都是维护在内存中的,所以如果你的工作流运行到了输入节点时。此时无论什么原因导致的worker服务重启都会出现这个报错。后边版本会考虑将工作流的运行时持久化存储,这样就可以避免这个问题了

  9. Vincent990413 commented on Sep 3, 2025

    @Vincent990413
  10. zgqgit commented on Sep 3, 2025

    @zgqgit
    Collaborator

    是的

  11. yogboot commented on Sep 5, 2025

    @yogboot

    好的,期待^-^

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions