feat: improve phoenix workflow tracing by Blackoutta · Pull Request #35605 · langgenius/dify

Blackoutta · 2026-04-27T12:43:21Z

Important

Make sure you have read our contribution guidelines
Ensure there is an associated issue and you have been assigned to it
Use the correct syntax to link this PR: Fixes #<issue number>.

Summary

Fixes #35045.

This PR improves Phoenix trace hierarchy for workflow, chatflow, nested workflow, and workflow-as-tool execution.

What changed:

establish stable Phoenix workflow root/session semantics for top-level and nested workflows
reconstruct workflow node hierarchy in the Phoenix provider from execution metadata, graph parents, structured parents, and safe root fallbacks
propagate private workflow-as-tool parent trace context into nested workflow trace task metadata without overwriting user workflow inputs
bridge nested workflow spans back to the outer workflow tool span with Phoenix-local parent span context coordination
move bounded trace retry signaling behind core.ops so generic trace dispatch does not import Phoenix provider symbols
persist enterprise trace dispatch state across pending-parent retries to avoid duplicate enterprise trace emission
consolidate Phoenix provider tests into one provider test entrypoint

Cross-repo dependency

This is one feature split across Dify and Graphon:

Graphon owns the tool node runtime protocol behavior needed to pass the current tool node execution id to runtime adapters.
Dify owns Phoenix trace provider behavior, workflow-as-tool parent context propagation, trace task metadata, and retry handling.

Related artifacts:

Dify issue: Show node hierarchy/nesting details in Phoenix trace view for complex workflows #35045
Graphon companion issue: Preserve tool node execution context for workflow-as-tool tracing graphon#104
Graphon companion PR: fix: preserve tool runtime compatibility graphon#105

Merge/release order:

Merge fix: preserve tool runtime compatibility graphon#105 first.
Publish a Graphon release containing that change: chore(api): upgrade graphon to v0.3.0 #35469
Update Dify to consume the released Graphon version if needed.
Merge this Dify PR after the Graphon release is available.

Reviewer notes

This branch intentionally keeps the development diary and supporting design/review notes under docs/phoenix-trace/ so reviewers can inspect the cross-repo reasoning. After review passes, maintainers can remove those notes if they do not want to keep the intermediate documentation in the final history.

Screenshots

Before	After
Phoenix displayed complex workflow spans as flat or orphaned traces.	Phoenix can show stable workflow roots, inherited sessions, node hierarchy, and workflow-as-tool nested workflow parentage.

Workflow Example:

Tracing before/after this PR:

Verification

uv run --project api pytest api/providers/trace/trace-arize-phoenix/tests/unit_tests/arize_phoenix_trace/test_arize_phoenix_trace.py (67 passed)

From Codex

Checklist

This change requires a documentation update, included: Dify Document
I understand that this PR may be closed in case there was no previous discussion or issues. (This doesn't apply to typos!)
I've added a test for each change that was introduced, and I tried as much as possible to make a single atomic change.
I've updated the documentation accordingly.
I ran make lint && make type-check (backend) and cd web && pnpm exec vp staged (frontend) to appease the lint gods

Blackoutta · 2026-04-27T12:43:30Z

Cross-repo merge/release dependency note:

The companion Graphon changes in fix: preserve tool runtime compatibility graphon#105 must merge first.
A Graphon release containing fix: preserve tool runtime compatibility graphon#105 must be published first.
Only after Dify consumes that released Graphon behavior: chore(api): upgrade graphon to v0.3.0 #35469, should this Dify PR be merged and released.

This ordering is required because the Dify workflow-as-tool Phoenix parent span restoration depends on Graphon passing the current tool node execution id into the runtime protocol.

github-actions · 2026-04-27T12:45:33Z

Pyrefly Diff

base → PR

--- /tmp/pyrefly_base.txt	2026-05-11 08:28:26.986595395 +0000
+++ /tmp/pyrefly_pr.txt	2026-05-11 08:28:14.518563028 +0000
@@ -47,7 +47,7 @@
 ERROR No matching overload found for function `core.model_manager.ModelInstance.invoke_llm` called with arguments: (prompt_messages=list[SystemPromptMessage | UserPromptMessage], model_parameters=dict[str, float], stream=Literal[False]) [no-matching-overload]
    --> core/llm_generator/llm_generator.py:653:60
 ERROR Class member `OpsTraceProviderConfigMap.__getitem__` overrides parent class `UserDict` in an inconsistent manner [bad-override-param-name]
-   --> core/ops/ops_trace_manager.py:206:9
+   --> core/ops/ops_trace_manager.py:218:9
 ERROR Argument `Unknown | None` is not assignable to parameter `result_object` with type `dict[str, Any]` in function `WaterCrawlProvider._structure_data` [bad-argument-type]
    --> core/rag/extractor/watercrawl/provider.py:108:37
 ERROR Object of class `BaseOxmlElement` has no attribute `body` [missing-attribute]
@@ -3104,32 +3104,34 @@
    --> tests/unit_tests/core/app/apps/test_streaming_utils.py:127:12
 ERROR Cannot index into `str` [bad-index]
    --> tests/unit_tests/core/app/apps/test_streaming_utils.py:128:12
+ERROR No matching overload found for function `core.app.apps.workflow.app_generator.WorkflowAppGenerator.generate` called with arguments: (app_model=SimpleNamespace, workflow=SimpleNamespace, user=SimpleNamespace, args=dict[str, list[@_] | str | dict[str, str]], invoke_from=Literal['service-api'], streaming=Literal[False], call_depth=Literal[0]) [no-matching-overload]
+  --> tests/unit_tests/core/app/apps/test_workflow_app_generator.py:71:32
 ERROR Argument `SimpleNamespace` is not assignable to parameter `application_generate_entity` with type `WorkflowAppGenerateEntity` in function `core.app.apps.workflow.app_generator.WorkflowAppGenerator.resume` [bad-argument-type]
-  --> tests/unit_tests/core/app/apps/test_workflow_app_generator.py:39:37
+   --> tests/unit_tests/core/app/apps/test_workflow_app_generator.py:110:37
 ERROR Argument `tuple[Literal['layer']]` is not assignable to parameter `graph_engine_layers` with type `Sequence[GraphEngineLayer]` in function `core.app.apps.workflow.app_generator.WorkflowAppGenerator.resume` [bad-argument-type]
-  --> tests/unit_tests/core/app/apps/test_workflow_app_generator.py:43:29
+   --> tests/unit_tests/core/app/apps/test_workflow_app_generator.py:114:29
 ERROR Argument `SimpleNamespace` is not assignable to parameter `app_model` with type `App` in function `core.app.apps.workflow.app_generator.WorkflowAppGenerator._generate` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/test_workflow_app_generator.py:111:19
+   --> tests/unit_tests/core/app/apps/test_workflow_app_generator.py:182:19
 ERROR Argument `SimpleNamespace` is not assignable to parameter `application_generate_entity` with type `WorkflowAppGenerateEntity` in function `core.app.apps.workflow.app_generator.WorkflowAppGenerator._generate` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/test_workflow_app_generator.py:114:37
+   --> tests/unit_tests/core/app/apps/test_workflow_app_generator.py:185:37
 ERROR Argument `Literal['service-api']` is not assignable to parameter `invoke_from` with type `InvokeFrom` in function `core.app.apps.workflow.app_generator.WorkflowAppGenerator._generate` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/test_workflow_app_generator.py:115:21
+   --> tests/unit_tests/core/app/apps/test_workflow_app_generator.py:186:21
 ERROR Argument `tuple[Literal['base-layer']]` is not assignable to parameter `graph_engine_layers` with type `Sequence[GraphEngineLayer]` in function `core.app.apps.workflow.app_generator.WorkflowAppGenerator._generate` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/test_workflow_app_generator.py:119:29
+   --> tests/unit_tests/core/app/apps/test_workflow_app_generator.py:190:29
 ERROR Argument `SimpleNamespace` is not assignable to parameter `pause_state_config` with type `PauseStateLayerConfig | None` in function `core.app.apps.workflow.app_generator.WorkflowAppGenerator._generate` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/test_workflow_app_generator.py:121:28
+   --> tests/unit_tests/core/app/apps/test_workflow_app_generator.py:192:28
 ERROR Cannot index into `object` [bad-index]
-   --> tests/unit_tests/core/app/apps/test_workflow_app_generator.py:125:12
+   --> tests/unit_tests/core/app/apps/test_workflow_app_generator.py:196:12
 ERROR Cannot index into `object` [bad-index]
-   --> tests/unit_tests/core/app/apps/test_workflow_app_generator.py:126:12
+   --> tests/unit_tests/core/app/apps/test_workflow_app_generator.py:197:12
 ERROR Argument `SimpleNamespace` is not assignable to parameter `app_model` with type `App` in function `core.app.apps.workflow.app_generator.WorkflowAppGenerator.resume` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/test_workflow_app_generator.py:202:19
+   --> tests/unit_tests/core/app/apps/test_workflow_app_generator.py:273:19
 ERROR Argument `SimpleNamespace` is not assignable to parameter `workflow` with type `Workflow` in function `core.app.apps.workflow.app_generator.WorkflowAppGenerator.resume` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/test_workflow_app_generator.py:203:18
+   --> tests/unit_tests/core/app/apps/test_workflow_app_generator.py:274:18
 ERROR Argument `SimpleNamespace` is not assignable to parameter `application_generate_entity` with type `WorkflowAppGenerateEntity` in function `core.app.apps.workflow.app_generator.WorkflowAppGenerator.resume` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/test_workflow_app_generator.py:205:37
+   --> tests/unit_tests/core/app/apps/test_workflow_app_generator.py:276:37
 ERROR Argument `SimpleNamespace` is not assignable to parameter `pause_state_config` with type `PauseStateLayerConfig | None` in function `core.app.apps.workflow.app_generator.WorkflowAppGenerator.resume` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/test_workflow_app_generator.py:209:28
+   --> tests/unit_tests/core/app/apps/test_workflow_app_generator.py:280:28
 ERROR Argument `SimpleNamespace` is not assignable to parameter `queue_manager` with type `AppQueueManager` in function `core.app.apps.workflow_app_runner.WorkflowBasedAppRunner.__init__` [bad-argument-type]
   --> tests/unit_tests/core/app/apps/test_workflow_app_runner_core.py:47:55
 ERROR Argument `SimpleNamespace` is not assignable to parameter `queue_manager` with type `AppQueueManager` in function `core.app.apps.workflow_app_runner.WorkflowBasedAppRunner.__init__` [bad-argument-type]
@@ -3839,45 +3841,45 @@
 ERROR Argument `SimpleNamespace` is not assignable to parameter `node` with type `Node[Unknown]` in function `core.app.workflow.layers.observability.ObservabilityLayer.on_node_run_end` [bad-argument-type]
    --> tests/unit_tests/core/app/workflow/test_observability_layer_extra.py:210:31
 ERROR Argument `_RepoRecorder` is not assignable to parameter `workflow_execution_repository` with type `WorkflowExecutionRepository` in function `core.app.workflow.layers.persistence.WorkflowPersistenceLayer.__init__` [bad-argument-type]
-  --> tests/unit_tests/core/app/workflow/test_persistence_layer.py:96:39
+  --> tests/unit_tests/core/app/workflow/test_persistence_layer.py:97:39
 ERROR Argument `_RepoRecorder` is not assignable to parameter `workflow_node_execution_repository` with type `WorkflowNodeExecutionRepository` in function `core.app.workflow.layers.persistence.WorkflowPersistenceLayer.__init__` [bad-argument-type]
-  --> tests/unit_tests/core/app/workflow/test_persistence_layer.py:97:44
+  --> tests/unit_tests/core/app/workflow/test_persistence_layer.py:98:44
 ERROR Argument `object | None` is not assignable to parameter `trace_manager` with type `TraceQueueManager | None` in function `core.app.workflow.layers.persistence.WorkflowPersistenceLayer.__init__` [bad-argument-type]
-  --> tests/unit_tests/core/app/workflow/test_persistence_layer.py:98:23
+  --> tests/unit_tests/core/app/workflow/test_persistence_layer.py:99:23
 ERROR Argument `None` is not assignable to parameter `command_channel` with type `CommandChannel` in function `graphon.graph_engine.layers.base.GraphEngineLayer.initialize` [bad-argument-type]
-   --> tests/unit_tests/core/app/workflow/test_persistence_layer.py:100:55
+   --> tests/unit_tests/core/app/workflow/test_persistence_layer.py:101:55
 ERROR `object` is not assignable to attribute `_workflow_execution` with type `WorkflowExecution | None` [bad-assignment]
-   --> tests/unit_tests/core/app/workflow/test_persistence_layer.py:108:37
+   --> tests/unit_tests/core/app/workflow/test_persistence_layer.py:109:37
 ERROR Cannot set item in `dict[str, WorkflowNodeExecution]` [unsupported-operation]
-   --> tests/unit_tests/core/app/workflow/test_persistence_layer.py:109:49
+   --> tests/unit_tests/core/app/workflow/test_persistence_layer.py:110:49
 ERROR Cannot set item in `dict[str, _NodeRuntimeSnapshot]` [unsupported-operation]
-   --> tests/unit_tests/core/app/workflow/test_persistence_layer.py:110:43
+   --> tests/unit_tests/core/app/workflow/test_persistence_layer.py:111:43
 ERROR Object of class `object` has no attribute `status` [missing-attribute]
-   --> tests/unit_tests/core/app/workflow/test_persistence_layer.py:176:16
-ERROR Object of class `object` has no attribute `total_tokens` [missing-attribute]
    --> tests/unit_tests/core/app/workflow/test_persistence_layer.py:177:16
-ERROR Object of class `object` has no attribute `total_steps` [missing-attribute]
+ERROR Object of class `object` has no attribute `total_tokens` [missing-attribute]
    --> tests/unit_tests/core/app/workflow/test_persistence_layer.py:178:16
+ERROR Object of class `object` has no attribute `total_steps` [missing-attribute]
+   --> tests/unit_tests/core/app/workflow/test_persistence_layer.py:179:16
 ERROR Object of class `GraphRuntimeState` has no attribute `_graph_execution` [missing-attribute]
-   --> tests/unit_tests/core/app/workflow/test_persistence_layer.py:185:9
+   --> tests/unit_tests/core/app/workflow/test_persistence_layer.py:186:9
 ERROR Object of class `object` has no attribute `status` [missing-attribute]
-   --> tests/unit_tests/core/app/workflow/test_persistence_layer.py:192:16
-ERROR Object of class `object` has no attribute `exceptions_count` [missing-attribute]
    --> tests/unit_tests/core/app/workflow/test_persistence_layer.py:193:16
-ERROR Object of class `object` has no attribute `total_tokens` [missing-attribute]
+ERROR Object of class `object` has no attribute `exceptions_count` [missing-attribute]
    --> tests/unit_tests/core/app/workflow/test_persistence_layer.py:194:16
+ERROR Object of class `object` has no attribute `total_tokens` [missing-attribute]
+   --> tests/unit_tests/core/app/workflow/test_persistence_layer.py:195:16
 ERROR Object of class `object` has no attribute `status` [missing-attribute]
-   --> tests/unit_tests/core/app/workflow/test_persistence_layer.py:217:16
+   --> tests/unit_tests/core/app/workflow/test_persistence_layer.py:218:16
 ERROR Object of class `object` has no attribute `status` [missing-attribute]
-   --> tests/unit_tests/core/app/workflow/test_persistence_layer.py:227:16
+   --> tests/unit_tests/core/app/workflow/test_persistence_layer.py:281:16
 ERROR Object of class `object` has no attribute `error_message` [missing-attribute]
-   --> tests/unit_tests/core/app/workflow/test_persistence_layer.py:228:16
+   --> tests/unit_tests/core/app/workflow/test_persistence_layer.py:282:16
 ERROR Object of class `object` has no attribute `status` [missing-attribute]
-   --> tests/unit_tests/core/app/workflow/test_persistence_layer.py:239:16
+   --> tests/unit_tests/core/app/workflow/test_persistence_layer.py:293:16
 ERROR Object of class `object` has no attribute `outputs` [missing-attribute]
-   --> tests/unit_tests/core/app/workflow/test_persistence_layer.py:240:16
+   --> tests/unit_tests/core/app/workflow/test_persistence_layer.py:294:16
 ERROR Object of class `object` has no attribute `finished_at` [missing-attribute]
-   --> tests/unit_tests/core/app/workflow/test_persistence_layer.py:241:16
+   --> tests/unit_tests/core/app/workflow/test_persistence_layer.py:295:16
 ERROR Object of class `NoneType` has no attribute `status` [missing-attribute]
    --> tests/unit_tests/core/base/test_app_generator_tts_publisher.py:167:16
 ERROR Object of class `NoneType` has no attribute `status` [missing-attribute]
@@ -5458,10 +5460,12 @@
   --> tests/unit_tests/core/workflow/nodes/tool/test_tool_node_runtime.py:33:9
 ERROR Object of class `ModuleType` has no attribute `TraceTask` [missing-attribute]
   --> tests/unit_tests/core/workflow/nodes/tool/test_tool_node_runtime.py:34:9
+ERROR Object of class `object` has no attribute `parent_trace_context` [missing-attribute]
+   --> tests/unit_tests/core/workflow/nodes/tool/test_tool_node_runtime.py:183:12
 ERROR Argument `object` is not assignable to parameter `message` with type `ToolInvokeMessage.BlobChunkMessage | ToolInvokeMessage.BlobMessage | ToolInvokeMessage.FileMessage | ToolInvokeMessage.JsonMessage | ToolInvokeMessage.LogMessage | ToolInvokeMessage.RetrieverResourceMessage | ToolInvokeMessage.TextMessage | ToolInvokeMessage.VariableMessage | None` in function `core.workflow.node_runtime.DifyToolNodeRuntime._convert_message_payload` [bad-argument-type]
-   --> tests/unit_tests/core/workflow/nodes/tool/test_tool_node_runtime.py:213:48
+   --> tests/unit_tests/core/workflow/nodes/tool/test_tool_node_runtime.py:276:48
 ERROR Argument `object` is not assignable to parameter `message` with type `ToolInvokeMessage.BlobChunkMessage | ToolInvokeMessage.BlobMessage | ToolInvokeMessage.FileMessage | ToolInvokeMessage.JsonMessage | ToolInvokeMessage.LogMessage | ToolInvokeMessage.RetrieverResourceMessage | ToolInvokeMessage.TextMessage | ToolInvokeMessage.VariableMessage | None` in function `core.workflow.node_runtime.DifyToolNodeRuntime._convert_message_payload` [bad-argument-type]
-   --> tests/unit_tests/core/workflow/nodes/tool/test_tool_node_runtime.py:220:42
+   --> tests/unit_tests/core/workflow/nodes/tool/test_tool_node_runtime.py:283:42
 ERROR Missing argument `name` in function `core.workflow.nodes.trigger_webhook.entities.WebhookParameter.__init__` [missing-argument]
   --> tests/unit_tests/core/workflow/nodes/webhook/test_entities.py:54:25
 ERROR Missing argument `name` in function `core.workflow.nodes.trigger_webhook.entities.WebhookBodyParameter.__init__` [missing-argument]
@@ -5546,6 +5550,12 @@
    --> tests/unit_tests/core/workflow/test_node_runtime.py:257:32
 ERROR Object of class `NoneType` has no attribute `storage_key` [missing-attribute]
    --> tests/unit_tests/core/workflow/test_node_runtime.py:268:12
+ERROR Object of class `object` has no attribute `tool` [missing-attribute]
+   --> tests/unit_tests/core/workflow/test_node_runtime.py:353:12
+ERROR Object of class `object` has no attribute `parent_trace_context` [missing-attribute]
+   --> tests/unit_tests/core/workflow/test_node_runtime.py:354:12
+ERROR Object of class `object` has no attribute `tool` [missing-attribute]
+   --> tests/unit_tests/core/workflow/test_node_runtime.py:389:12
 ERROR Argument `TestWorkflowEntry.test_single_step_run_injects_code_limits.StubWorkflow` is not assignable to parameter `workflow` with type `Workflow` in function `core.workflow.workflow_entry.WorkflowEntry.single_step_run` [bad-argument-type]
    --> tests/unit_tests/core/workflow/test_workflow_entry.py:145:22
 ERROR Object of class `_FakeModelNodeMixin` has no attribute `node_data` [missing-attribute]
@@ -6275,6 +6285,20 @@
     --> tests/unit_tests/tasks/test_mail_send_task.py:1299:67
 ERROR Argument `dict[str, bool]` is not assignable to parameter `substitutions` with type `Mapping[str, str]` in function `tasks.mail_inner_task._render_template_with_strategy` [bad-argument-type]
     --> tests/unit_tests/tasks/test_mail_send_task.py:1300:64
+ERROR Object of class `ModuleType` has no attribute `is_enabled` [missing-attribute]
+  --> tests/unit_tests/tasks/test_ops_trace_task.py:39:5
+ERROR Object of class `ModuleType` has no attribute `OpsTraceManager` [missing-attribute]
+  --> tests/unit_tests/tasks/test_ops_trace_task.py:41:5
+ERROR Object of class `ModuleType` has no attribute `EnterpriseOtelTrace` [missing-attribute]
+  --> tests/unit_tests/tasks/test_ops_trace_task.py:50:9
+ERROR Missing argument `file_info` in function `celery.app.task.Task.push_request` [missing-argument]
+  --> tests/unit_tests/tasks/test_ops_trace_task.py:76:37
+ERROR Unexpected keyword argument `retries` in function `celery.app.task.Task.push_request` [unexpected-keyword]
+  --> tests/unit_tests/tasks/test_ops_trace_task.py:76:38
+ERROR `*` is not supported between `None` and `int` [unsupported-operation]
+   --> tests/unit_tests/tasks/test_ops_trace_task.py:219:12
+ERROR Argument `int | None` is not assignable to parameter `retries` with type `int` in function `_run_task` [bad-argument-type]
+   --> tests/unit_tests/tasks/test_ops_trace_task.py:297:38
 ERROR Argument `Iterator[Any]` is not assignable to parameter `response_stream` with type `Generator[BaseModel | Mapping[str, Any] | str]` in function `tasks.app_generate.workflow_execute_task._publish_streaming_response` [bad-argument-type]
   --> tests/unit_tests/tasks/test_workflow_execute_task.py:77:33
 ERROR Argument `UUID` is not assignable to parameter `workflow_run_id` with type `str` in function `tasks.app_generate.workflow_execute_task._publish_streaming_response` [bad-argument-type]

github-actions · 2026-04-27T12:47:16Z

Pyrefly Diff

base → PR

--- /tmp/pyrefly_base.txt	2026-04-27 12:46:43.108731864 +0000
+++ /tmp/pyrefly_pr.txt	2026-04-27 12:46:28.121467463 +0000
@@ -3080,32 +3080,34 @@
    --> tests/unit_tests/core/app/apps/test_streaming_utils.py:127:12
 ERROR Cannot index into `str` [bad-index]
    --> tests/unit_tests/core/app/apps/test_streaming_utils.py:128:12
+ERROR No matching overload found for function `core.app.apps.workflow.app_generator.WorkflowAppGenerator.generate` called with arguments: (app_model=SimpleNamespace, workflow=SimpleNamespace, user=SimpleNamespace, args=dict[str, dict[str, str] | list[@_] | str], invoke_from=Literal['service-api'], streaming=Literal[False], call_depth=Literal[0]) [no-matching-overload]
+  --> tests/unit_tests/core/app/apps/test_workflow_app_generator.py:69:32
 ERROR Argument `SimpleNamespace` is not assignable to parameter `application_generate_entity` with type `WorkflowAppGenerateEntity` in function `core.app.apps.workflow.app_generator.WorkflowAppGenerator.resume` [bad-argument-type]
-  --> tests/unit_tests/core/app/apps/test_workflow_app_generator.py:37:37
+   --> tests/unit_tests/core/app/apps/test_workflow_app_generator.py:109:37
 ERROR Argument `tuple[Literal['layer']]` is not assignable to parameter `graph_engine_layers` with type `Sequence[GraphEngineLayer]` in function `core.app.apps.workflow.app_generator.WorkflowAppGenerator.resume` [bad-argument-type]
-  --> tests/unit_tests/core/app/apps/test_workflow_app_generator.py:41:29
+   --> tests/unit_tests/core/app/apps/test_workflow_app_generator.py:113:29
 ERROR Argument `SimpleNamespace` is not assignable to parameter `app_model` with type `App` in function `core.app.apps.workflow.app_generator.WorkflowAppGenerator._generate` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/test_workflow_app_generator.py:109:19
+   --> tests/unit_tests/core/app/apps/test_workflow_app_generator.py:181:19
 ERROR Argument `SimpleNamespace` is not assignable to parameter `application_generate_entity` with type `WorkflowAppGenerateEntity` in function `core.app.apps.workflow.app_generator.WorkflowAppGenerator._generate` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/test_workflow_app_generator.py:112:37
+   --> tests/unit_tests/core/app/apps/test_workflow_app_generator.py:184:37
 ERROR Argument `Literal['service-api']` is not assignable to parameter `invoke_from` with type `InvokeFrom` in function `core.app.apps.workflow.app_generator.WorkflowAppGenerator._generate` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/test_workflow_app_generator.py:113:21
+   --> tests/unit_tests/core/app/apps/test_workflow_app_generator.py:185:21
 ERROR Argument `tuple[Literal['base-layer']]` is not assignable to parameter `graph_engine_layers` with type `Sequence[GraphEngineLayer]` in function `core.app.apps.workflow.app_generator.WorkflowAppGenerator._generate` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/test_workflow_app_generator.py:117:29
+   --> tests/unit_tests/core/app/apps/test_workflow_app_generator.py:189:29
 ERROR Argument `SimpleNamespace` is not assignable to parameter `pause_state_config` with type `PauseStateLayerConfig | None` in function `core.app.apps.workflow.app_generator.WorkflowAppGenerator._generate` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/test_workflow_app_generator.py:119:28
+   --> tests/unit_tests/core/app/apps/test_workflow_app_generator.py:191:28
 ERROR Cannot index into `object` [bad-index]
-   --> tests/unit_tests/core/app/apps/test_workflow_app_generator.py:123:12
+   --> tests/unit_tests/core/app/apps/test_workflow_app_generator.py:195:12
 ERROR Cannot index into `object` [bad-index]
-   --> tests/unit_tests/core/app/apps/test_workflow_app_generator.py:124:12
+   --> tests/unit_tests/core/app/apps/test_workflow_app_generator.py:196:12
 ERROR Argument `SimpleNamespace` is not assignable to parameter `app_model` with type `App` in function `core.app.apps.workflow.app_generator.WorkflowAppGenerator.resume` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/test_workflow_app_generator.py:200:19
+   --> tests/unit_tests/core/app/apps/test_workflow_app_generator.py:272:19
 ERROR Argument `SimpleNamespace` is not assignable to parameter `workflow` with type `Workflow` in function `core.app.apps.workflow.app_generator.WorkflowAppGenerator.resume` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/test_workflow_app_generator.py:201:18
+   --> tests/unit_tests/core/app/apps/test_workflow_app_generator.py:273:18
 ERROR Argument `SimpleNamespace` is not assignable to parameter `application_generate_entity` with type `WorkflowAppGenerateEntity` in function `core.app.apps.workflow.app_generator.WorkflowAppGenerator.resume` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/test_workflow_app_generator.py:203:37
+   --> tests/unit_tests/core/app/apps/test_workflow_app_generator.py:275:37
 ERROR Argument `SimpleNamespace` is not assignable to parameter `pause_state_config` with type `PauseStateLayerConfig | None` in function `core.app.apps.workflow.app_generator.WorkflowAppGenerator.resume` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/test_workflow_app_generator.py:207:28
+   --> tests/unit_tests/core/app/apps/test_workflow_app_generator.py:279:28
 ERROR Argument `SimpleNamespace` is not assignable to parameter `queue_manager` with type `AppQueueManager` in function `core.app.apps.workflow_app_runner.WorkflowBasedAppRunner.__init__` [bad-argument-type]
   --> tests/unit_tests/core/app/apps/test_workflow_app_runner_core.py:47:55
 ERROR Argument `SimpleNamespace` is not assignable to parameter `queue_manager` with type `AppQueueManager` in function `core.app.apps.workflow_app_runner.WorkflowBasedAppRunner.__init__` [bad-argument-type]
@@ -3789,45 +3791,45 @@
 ERROR Argument `SimpleNamespace` is not assignable to parameter `node` with type `Node[Unknown]` in function `core.app.workflow.layers.observability.ObservabilityLayer.on_node_run_end` [bad-argument-type]
    --> tests/unit_tests/core/app/workflow/test_observability_layer_extra.py:208:31
 ERROR Argument `_RepoRecorder` is not assignable to parameter `workflow_execution_repository` with type `WorkflowExecutionRepository` in function `core.app.workflow.layers.persistence.WorkflowPersistenceLayer.__init__` [bad-argument-type]
-  --> tests/unit_tests/core/app/workflow/test_persistence_layer.py:93:39
+  --> tests/unit_tests/core/app/workflow/test_persistence_layer.py:94:39
 ERROR Argument `_RepoRecorder` is not assignable to parameter `workflow_node_execution_repository` with type `WorkflowNodeExecutionRepository` in function `core.app.workflow.layers.persistence.WorkflowPersistenceLayer.__init__` [bad-argument-type]
-  --> tests/unit_tests/core/app/workflow/test_persistence_layer.py:94:44
+  --> tests/unit_tests/core/app/workflow/test_persistence_layer.py:95:44
 ERROR Argument `object | None` is not assignable to parameter `trace_manager` with type `TraceQueueManager | None` in function `core.app.workflow.layers.persistence.WorkflowPersistenceLayer.__init__` [bad-argument-type]
-  --> tests/unit_tests/core/app/workflow/test_persistence_layer.py:95:23
+  --> tests/unit_tests/core/app/workflow/test_persistence_layer.py:96:23
 ERROR Argument `None` is not assignable to parameter `command_channel` with type `CommandChannel` in function `graphon.graph_engine.layers.base.GraphEngineLayer.initialize` [bad-argument-type]
-  --> tests/unit_tests/core/app/workflow/test_persistence_layer.py:97:55
+  --> tests/unit_tests/core/app/workflow/test_persistence_layer.py:98:55
 ERROR `object` is not assignable to attribute `_workflow_execution` with type `WorkflowExecution | None` [bad-assignment]
-   --> tests/unit_tests/core/app/workflow/test_persistence_layer.py:105:37
+   --> tests/unit_tests/core/app/workflow/test_persistence_layer.py:106:37
 ERROR Cannot set item in `dict[str, WorkflowNodeExecution]` [unsupported-operation]
-   --> tests/unit_tests/core/app/workflow/test_persistence_layer.py:106:49
+   --> tests/unit_tests/core/app/workflow/test_persistence_layer.py:107:49
 ERROR Cannot set item in `dict[str, _NodeRuntimeSnapshot]` [unsupported-operation]
-   --> tests/unit_tests/core/app/workflow/test_persistence_layer.py:107:43
+   --> tests/unit_tests/core/app/workflow/test_persistence_layer.py:108:43
 ERROR Object of class `object` has no attribute `status` [missing-attribute]
-   --> tests/unit_tests/core/app/workflow/test_persistence_layer.py:173:16
-ERROR Object of class `object` has no attribute `total_tokens` [missing-attribute]
    --> tests/unit_tests/core/app/workflow/test_persistence_layer.py:174:16
-ERROR Object of class `object` has no attribute `total_steps` [missing-attribute]
+ERROR Object of class `object` has no attribute `total_tokens` [missing-attribute]
    --> tests/unit_tests/core/app/workflow/test_persistence_layer.py:175:16
+ERROR Object of class `object` has no attribute `total_steps` [missing-attribute]
+   --> tests/unit_tests/core/app/workflow/test_persistence_layer.py:176:16
 ERROR Object of class `GraphRuntimeState` has no attribute `_graph_execution` [missing-attribute]
-   --> tests/unit_tests/core/app/workflow/test_persistence_layer.py:182:9
+   --> tests/unit_tests/core/app/workflow/test_persistence_layer.py:183:9
 ERROR Object of class `object` has no attribute `status` [missing-attribute]
-   --> tests/unit_tests/core/app/workflow/test_persistence_layer.py:189:16
-ERROR Object of class `object` has no attribute `exceptions_count` [missing-attribute]
    --> tests/unit_tests/core/app/workflow/test_persistence_layer.py:190:16
-ERROR Object of class `object` has no attribute `total_tokens` [missing-attribute]
+ERROR Object of class `object` has no attribute `exceptions_count` [missing-attribute]
    --> tests/unit_tests/core/app/workflow/test_persistence_layer.py:191:16
+ERROR Object of class `object` has no attribute `total_tokens` [missing-attribute]
+   --> tests/unit_tests/core/app/workflow/test_persistence_layer.py:192:16
 ERROR Object of class `object` has no attribute `status` [missing-attribute]
-   --> tests/unit_tests/core/app/workflow/test_persistence_layer.py:214:16
+   --> tests/unit_tests/core/app/workflow/test_persistence_layer.py:215:16
 ERROR Object of class `object` has no attribute `status` [missing-attribute]
-   --> tests/unit_tests/core/app/workflow/test_persistence_layer.py:224:16
+   --> tests/unit_tests/core/app/workflow/test_persistence_layer.py:278:16
 ERROR Object of class `object` has no attribute `error_message` [missing-attribute]
-   --> tests/unit_tests/core/app/workflow/test_persistence_layer.py:225:16
+   --> tests/unit_tests/core/app/workflow/test_persistence_layer.py:279:16
 ERROR Object of class `object` has no attribute `status` [missing-attribute]
-   --> tests/unit_tests/core/app/workflow/test_persistence_layer.py:236:16
+   --> tests/unit_tests/core/app/workflow/test_persistence_layer.py:290:16
 ERROR Object of class `object` has no attribute `outputs` [missing-attribute]
-   --> tests/unit_tests/core/app/workflow/test_persistence_layer.py:237:16
+   --> tests/unit_tests/core/app/workflow/test_persistence_layer.py:291:16
 ERROR Object of class `object` has no attribute `finished_at` [missing-attribute]
-   --> tests/unit_tests/core/app/workflow/test_persistence_layer.py:238:16
+   --> tests/unit_tests/core/app/workflow/test_persistence_layer.py:292:16
 ERROR Object of class `NoneType` has no attribute `status` [missing-attribute]
    --> tests/unit_tests/core/base/test_app_generator_tts_publisher.py:166:16
 ERROR Object of class `NoneType` has no attribute `status` [missing-attribute]
@@ -5391,23 +5393,25 @@
 ERROR Returned type `VariableBase` is not assignable to declared return type `ArrayAnyVariable | ArrayBooleanVariable | ArrayFileVariable | ArrayNumberVariable | ArrayObjectVariable | ArrayStringVariable | BooleanVariable | FileVariable | FloatVariable | IntegerVariable | NoneVariable | ObjectVariable | SecretVariable | StringVariable` [bad-return]
    --> tests/unit_tests/core/workflow/nodes/test_start_node_json_object.py:288:12
 ERROR Object of class `ModuleType` has no attribute `TraceQueueManager` [missing-attribute]
-  --> tests/unit_tests/core/workflow/nodes/tool/test_tool_node.py:63:9
+  --> tests/unit_tests/core/workflow/nodes/tool/test_tool_node.py:70:9
 ERROR Object of class `ModuleType` has no attribute `TraceTask` [missing-attribute]
-  --> tests/unit_tests/core/workflow/nodes/tool/test_tool_node.py:64:9
+  --> tests/unit_tests/core/workflow/nodes/tool/test_tool_node.py:71:9
 ERROR Argument `_StubToolRuntime` is not assignable to parameter `runtime` with type `ToolNodeRuntimeProtocol` in function `graphon.nodes.tool.tool_node.ToolNode.__init__` [bad-argument-type]
-   --> tests/unit_tests/core/workflow/nodes/tool/test_tool_node.py:117:17
+   --> tests/unit_tests/core/workflow/nodes/tool/test_tool_node.py:124:17
 ERROR Argument `Iterator[ToolRuntimeMessage] | Iterator[Any]` is not assignable to parameter `messages` with type `Generator[ToolRuntimeMessage]` in function `graphon.nodes.tool.tool_node.ToolNode._transform_message` [bad-argument-type]
-   --> tests/unit_tests/core/workflow/nodes/tool/test_tool_node.py:133:18
+   --> tests/unit_tests/core/workflow/nodes/tool/test_tool_node.py:140:18
 ERROR Object of class `FunctionType` has no attribute `assert_called_once_with` [missing-attribute]
-   --> tests/unit_tests/core/workflow/nodes/tool/test_tool_node.py:224:5
+   --> tests/unit_tests/core/workflow/nodes/tool/test_tool_node.py:231:5
+ERROR Object of class `ToolNode` has no attribute `_get_tool_runtime` [missing-attribute]
+   --> tests/unit_tests/core/workflow/nodes/tool/test_tool_node.py:244:14
 ERROR Object of class `ModuleType` has no attribute `TraceQueueManager` [missing-attribute]
   --> tests/unit_tests/core/workflow/nodes/tool/test_tool_node_runtime.py:33:9
 ERROR Object of class `ModuleType` has no attribute `TraceTask` [missing-attribute]
   --> tests/unit_tests/core/workflow/nodes/tool/test_tool_node_runtime.py:34:9
 ERROR Argument `object` is not assignable to parameter `message` with type `ToolInvokeMessage.BlobChunkMessage | ToolInvokeMessage.BlobMessage | ToolInvokeMessage.FileMessage | ToolInvokeMessage.JsonMessage | ToolInvokeMessage.LogMessage | ToolInvokeMessage.RetrieverResourceMessage | ToolInvokeMessage.TextMessage | ToolInvokeMessage.VariableMessage | None` in function `core.workflow.node_runtime.DifyToolNodeRuntime._convert_message_payload` [bad-argument-type]
-   --> tests/unit_tests/core/workflow/nodes/tool/test_tool_node_runtime.py:213:48
+   --> tests/unit_tests/core/workflow/nodes/tool/test_tool_node_runtime.py:275:48
 ERROR Argument `object` is not assignable to parameter `message` with type `ToolInvokeMessage.BlobChunkMessage | ToolInvokeMessage.BlobMessage | ToolInvokeMessage.FileMessage | ToolInvokeMessage.JsonMessage | ToolInvokeMessage.LogMessage | ToolInvokeMessage.RetrieverResourceMessage | ToolInvokeMessage.TextMessage | ToolInvokeMessage.VariableMessage | None` in function `core.workflow.node_runtime.DifyToolNodeRuntime._convert_message_payload` [bad-argument-type]
-   --> tests/unit_tests/core/workflow/nodes/tool/test_tool_node_runtime.py:220:42
+   --> tests/unit_tests/core/workflow/nodes/tool/test_tool_node_runtime.py:282:42
 ERROR Missing argument `name` in function `core.workflow.nodes.trigger_webhook.entities.WebhookParameter.__init__` [missing-argument]
   --> tests/unit_tests/core/workflow/nodes/webhook/test_entities.py:54:25
 ERROR Missing argument `name` in function `core.workflow.nodes.trigger_webhook.entities.WebhookBodyParameter.__init__` [missing-argument]
@@ -5475,23 +5479,29 @@
 ERROR `SimpleNamespace` is not assignable to attribute `graph_runtime_state` with type `GraphRuntimeState` [bad-assignment]
    --> tests/unit_tests/core/workflow/test_node_factory.py:687:39
 ERROR Object of class `ExternalRecipient` has no attribute `reference_id` [missing-attribute]
-  --> tests/unit_tests/core/workflow/test_node_runtime.py:88:12
+  --> tests/unit_tests/core/workflow/test_node_runtime.py:90:12
 ERROR Argument `object` is not assignable to parameter `method` with type `EmailDeliveryMethod | InteractiveSurfaceDeliveryMethod` in function `core.workflow.node_runtime.apply_dify_debug_email_recipient` [bad-argument-type]
-   --> tests/unit_tests/core/workflow/test_node_runtime.py:103:45
+   --> tests/unit_tests/core/workflow/test_node_runtime.py:105:45
 ERROR Argument `SimpleNamespace` is not assignable to parameter `model_instance` with type `ModelInstance` in function `core.workflow.node_runtime.DifyPreparedLLM.__init__` [bad-argument-type]
-   --> tests/unit_tests/core/workflow/test_node_runtime.py:135:32
+   --> tests/unit_tests/core/workflow/test_node_runtime.py:137:32
 ERROR Argument `SimpleNamespace` is not assignable to parameter `model_instance` with type `ModelInstance` in function `core.workflow.node_runtime.DifyPreparedLLM.__init__` [bad-argument-type]
-   --> tests/unit_tests/core/workflow/test_node_runtime.py:170:32
+   --> tests/unit_tests/core/workflow/test_node_runtime.py:172:32
 ERROR Argument `SimpleNamespace` is not assignable to parameter `model_instance` with type `ModelInstance` in function `core.workflow.node_runtime.DifyPreparedLLM.__init__` [bad-argument-type]
-   --> tests/unit_tests/core/workflow/test_node_runtime.py:183:32
+   --> tests/unit_tests/core/workflow/test_node_runtime.py:185:32
 ERROR Argument `SimpleNamespace` is not assignable to parameter `model_instance` with type `ModelInstance` in function `core.workflow.node_runtime.DifyPreparedLLM.__init__` [bad-argument-type]
-   --> tests/unit_tests/core/workflow/test_node_runtime.py:209:32
+   --> tests/unit_tests/core/workflow/test_node_runtime.py:211:32
 ERROR Argument `Literal['chat']` is not assignable to parameter `model_mode` with type `LLMMode` in function `core.workflow.node_runtime.DifyPromptMessageSerializer.serialize` [bad-argument-type]
-   --> tests/unit_tests/core/workflow/test_node_runtime.py:220:20
+   --> tests/unit_tests/core/workflow/test_node_runtime.py:222:20
 ERROR Argument `SimpleNamespace` is not assignable to parameter `file_reference_factory` with type `FileReferenceFactoryProtocol` in function `core.workflow.node_runtime.DifyRetrieverAttachmentLoader.__init__` [bad-argument-type]
-   --> tests/unit_tests/core/workflow/test_node_runtime.py:255:32
+   --> tests/unit_tests/core/workflow/test_node_runtime.py:257:32
 ERROR Object of class `NoneType` has no attribute `storage_key` [missing-attribute]
-   --> tests/unit_tests/core/workflow/test_node_runtime.py:266:12
+   --> tests/unit_tests/core/workflow/test_node_runtime.py:268:12
+ERROR Object of class `object` has no attribute `tool` [missing-attribute]
+   --> tests/unit_tests/core/workflow/test_node_runtime.py:353:12
+ERROR Object of class `object` has no attribute `parent_trace_context` [missing-attribute]
+   --> tests/unit_tests/core/workflow/test_node_runtime.py:354:12
+ERROR Object of class `object` has no attribute `tool` [missing-attribute]
+   --> tests/unit_tests/core/workflow/test_node_runtime.py:389:12
 ERROR Argument `list[VariableBase]` is not assignable to parameter `variables` with type `Sequence[Variable]` in function `core.workflow.variable_pool_initializer.add_variables_to_pool` [bad-argument-type]
    --> tests/unit_tests/core/workflow/test_variable_pool.py:94:13
 ERROR Argument `TestWorkflowEntry.test_single_step_run_injects_code_limits.StubWorkflow` is not assignable to parameter `workflow` with type `Workflow` in function `core.workflow.workflow_entry.WorkflowEntry.single_step_run` [bad-argument-type]
@@ -6215,6 +6225,18 @@
     --> tests/unit_tests/tasks/test_mail_send_task.py:1299:67
 ERROR Argument `dict[str, bool]` is not assignable to parameter `substitutions` with type `Mapping[str, str]` in function `tasks.mail_inner_task._render_template_with_strategy` [bad-argument-type]
     --> tests/unit_tests/tasks/test_mail_send_task.py:1300:64
+ERROR Object of class `ModuleType` has no attribute `is_enabled` [missing-attribute]
+  --> tests/unit_tests/tasks/test_ops_trace_task.py:39:5
+ERROR Object of class `ModuleType` has no attribute `OpsTraceManager` [missing-attribute]
+  --> tests/unit_tests/tasks/test_ops_trace_task.py:41:5
+ERROR Object of class `ModuleType` has no attribute `EnterpriseOtelTrace` [missing-attribute]
+  --> tests/unit_tests/tasks/test_ops_trace_task.py:50:9
+ERROR Missing argument `file_info` in function `celery.app.task.Task.push_request` [missing-argument]
+  --> tests/unit_tests/tasks/test_ops_trace_task.py:76:37
+ERROR Unexpected keyword argument `retries` in function `celery.app.task.Task.push_request` [unexpected-keyword]
+  --> tests/unit_tests/tasks/test_ops_trace_task.py:76:38
+ERROR Argument `int | None` is not assignable to parameter `retries` with type `int` in function `_run_task` [bad-argument-type]
+   --> tests/unit_tests/tasks/test_ops_trace_task.py:254:38
 ERROR Argument `Iterator[Any]` is not assignable to parameter `response_stream` with type `Generator[BaseModel | Mapping[str, Any] | str]` in function `tasks.app_generate.workflow_execute_task._publish_streaming_response` [bad-argument-type]
   --> tests/unit_tests/tasks/test_workflow_execute_task.py:77:33
 ERROR Argument `UUID` is not assignable to parameter `workflow_run_id` with type `str` in function `tasks.app_generate.workflow_execute_task._publish_streaming_response` [bad-argument-type]

github-actions · 2026-04-28T01:46:11Z

Pyrefly Diff

base → PR

--- /tmp/pyrefly_base.txt	2026-04-28 01:45:38.801273839 +0000
+++ /tmp/pyrefly_pr.txt	2026-04-28 01:45:26.337128999 +0000
@@ -24,6 +24,8 @@
    --> core/app/task_pipeline/easy_ui_based_generate_task_pipeline.py:321:29
 ERROR Argument `list[Unknown]` is not assignable to parameter `value` with type `SQLCoreOperations[dict[str, Any]] | dict[str, Any]` in function `sqlalchemy.orm.base.Mapped.__set__` [bad-argument-type]
    --> core/app/task_pipeline/easy_ui_based_generate_task_pipeline.py:388:27
+ERROR Object of class `QuestionClassifierNode` has no attribute `model_instance` [missing-attribute]
+   --> core/app/workflow/layers/llm_quota.py:119:38
 ERROR No matching overload found for function `core.model_manager.ModelInstance.invoke_llm` called with arguments: (prompt_messages=list[UserPromptMessage], model_parameters=dict[str, int], stream=Literal[False]) [no-matching-overload]
    --> core/llm_generator/llm_generator.py:113:60
 ERROR No matching overload found for function `core.model_manager.ModelInstance.invoke_llm` called with arguments: (prompt_messages=list[UserPromptMessage], model_parameters=dict[str, float | int] | dict[str, object] | dict[Unknown, Unknown], stop=list[str] | list[Unknown], stream=Literal[False]) [no-matching-overload]
@@ -44,6 +46,10 @@
    --> core/llm_generator/llm_generator.py:653:60
 ERROR Class member `OpsTraceProviderConfigMap.__getitem__` overrides parent class `UserDict` in an inconsistent manner [bad-override-param-name]
    --> core/ops/ops_trace_manager.py:206:9
+ERROR Class member `PluginModelRuntime.invoke_llm` overrides parent class `ModelRuntime` in an inconsistent manner [bad-override]
+   --> core/plugin/impl/model_runtime.py:198:9
+ERROR Cannot instantiate `PluginModelRuntime` because the following members are abstract: `invoke_llm_with_structured_output` [bad-instantiation]
+   --> core/plugin/impl/model_runtime_factory.py:116:30
 ERROR Argument `dict[str, bytes | str]` is not assignable to parameter `headers` with type `Headers | Mapping[bytes, bytes] | Mapping[str, str] | Sequence[tuple[bytes, bytes]] | Sequence[tuple[str, str]] | None` in function `httpx._api.post` [bad-argument-type]
    --> core/rag/extractor/notion_extractor.py:106:25
 ERROR Argument `dict[str, bytes | str]` is not assignable to parameter `headers` with type `Headers | Mapping[bytes, bytes] | Mapping[str, str] | Sequence[tuple[bytes, bytes]] | Sequence[tuple[str, str]] | None` in function `httpx._api.request` [bad-argument-type]
@@ -628,18 +634,82 @@
    --> tests/integration_tests/services/test_workflow_draft_variable_service.py:651:21
 ERROR Object of class `TestWorkflowDraftVariableServiceResetVariable` has no attribute `_test_user_id` [missing-attribute]
    --> tests/integration_tests/services/test_workflow_draft_variable_service.py:817:21
+ERROR Unexpected keyword argument `system_variables` in function `graphon.runtime.variable_pool.VariablePool.__init__` [unexpected-keyword]
+  --> tests/integration_tests/workflow/nodes/test_code.py:49:9
+ERROR Unexpected keyword argument `user_inputs` in function `graphon.runtime.variable_pool.VariablePool.__init__` [unexpected-keyword]
+  --> tests/integration_tests/workflow/nodes/test_code.py:50:9
+ERROR Unexpected keyword argument `environment_variables` in function `graphon.runtime.variable_pool.VariablePool.__init__` [unexpected-keyword]
+  --> tests/integration_tests/workflow/nodes/test_code.py:51:9
+ERROR Unexpected keyword argument `conversation_variables` in function `graphon.runtime.variable_pool.VariablePool.__init__` [unexpected-keyword]
+  --> tests/integration_tests/workflow/nodes/test_code.py:52:9
+ERROR Unexpected keyword argument `system_variables` in function `graphon.runtime.variable_pool.VariablePool.__init__` [unexpected-keyword]
+  --> tests/integration_tests/workflow/nodes/test_http.py:59:9
+ERROR Unexpected keyword argument `user_inputs` in function `graphon.runtime.variable_pool.VariablePool.__init__` [unexpected-keyword]
+  --> tests/integration_tests/workflow/nodes/test_http.py:60:9
+ERROR Unexpected keyword argument `environment_variables` in function `graphon.runtime.variable_pool.VariablePool.__init__` [unexpected-keyword]
+  --> tests/integration_tests/workflow/nodes/test_http.py:61:9
+ERROR Unexpected keyword argument `conversation_variables` in function `graphon.runtime.variable_pool.VariablePool.__init__` [unexpected-keyword]
+  --> tests/integration_tests/workflow/nodes/test_http.py:62:9
 ERROR Argument `SSRFProxy` is not assignable to parameter `http_client` with type `HttpClientProtocol | None` in function `graphon.nodes.http_request.node.HttpRequestNode.__init__` [bad-argument-type]
   --> tests/integration_tests/workflow/nodes/test_http.py:83:21
-ERROR Argument `type[ToolFileManager]` is not assignable to parameter `tool_file_manager_factory` with type `() -> ToolFileManagerProtocol` in function `graphon.nodes.http_request.node.HttpRequestNode.__init__` [bad-argument-type]
+ERROR Argument `type[ToolFileManager]` is not assignable to parameter `tool_file_manager_factory` with type `(() -> ToolFileManagerProtocol) | None` in function `graphon.nodes.http_request.node.HttpRequestNode.__init__` [bad-argument-type]
   --> tests/integration_tests/workflow/nodes/test_http.py:84:35
+ERROR Unexpected keyword argument `system_variables` in function `graphon.runtime.variable_pool.VariablePool.__init__` [unexpected-keyword]
+   --> tests/integration_tests/workflow/nodes/test_http.py:208:9
+ERROR Unexpected keyword argument `user_inputs` in function `graphon.runtime.variable_pool.VariablePool.__init__` [unexpected-keyword]
+   --> tests/integration_tests/workflow/nodes/test_http.py:209:9
+ERROR Unexpected keyword argument `environment_variables` in function `graphon.runtime.variable_pool.VariablePool.__init__` [unexpected-keyword]
+   --> tests/integration_tests/workflow/nodes/test_http.py:210:9
+ERROR Unexpected keyword argument `conversation_variables` in function `graphon.runtime.variable_pool.VariablePool.__init__` [unexpected-keyword]
+   --> tests/integration_tests/workflow/nodes/test_http.py:211:9
 ERROR Argument `SSRFProxy` is not assignable to parameter `http_client` with type `HttpClientProtocol` in function `graphon.nodes.http_request.executor.Executor.__init__` [bad-argument-type]
    --> tests/integration_tests/workflow/nodes/test_http.py:242:25
+ERROR Unexpected keyword argument `system_variables` in function `graphon.runtime.variable_pool.VariablePool.__init__` [unexpected-keyword]
+   --> tests/integration_tests/workflow/nodes/test_http.py:706:9
+ERROR Unexpected keyword argument `user_inputs` in function `graphon.runtime.variable_pool.VariablePool.__init__` [unexpected-keyword]
+   --> tests/integration_tests/workflow/nodes/test_http.py:707:9
+ERROR Unexpected keyword argument `environment_variables` in function `graphon.runtime.variable_pool.VariablePool.__init__` [unexpected-keyword]
+   --> tests/integration_tests/workflow/nodes/test_http.py:708:9
+ERROR Unexpected keyword argument `conversation_variables` in function `graphon.runtime.variable_pool.VariablePool.__init__` [unexpected-keyword]
+   --> tests/integration_tests/workflow/nodes/test_http.py:709:9
 ERROR Argument `SSRFProxy` is not assignable to parameter `http_client` with type `HttpClientProtocol | None` in function `graphon.nodes.http_request.node.HttpRequestNode.__init__` [bad-argument-type]
    --> tests/integration_tests/workflow/nodes/test_http.py:731:21
-ERROR Argument `type[ToolFileManager]` is not assignable to parameter `tool_file_manager_factory` with type `() -> ToolFileManagerProtocol` in function `graphon.nodes.http_request.node.HttpRequestNode.__init__` [bad-argument-type]
+ERROR Argument `type[ToolFileManager]` is not assignable to parameter `tool_file_manager_factory` with type `(() -> ToolFileManagerProtocol) | None` in function `graphon.nodes.http_request.node.HttpRequestNode.__init__` [bad-argument-type]
    --> tests/integration_tests/workflow/nodes/test_http.py:732:35
+ERROR Unexpected keyword argument `system_variables` in function `graphon.runtime.variable_pool.VariablePool.__init__` [unexpected-keyword]
+  --> tests/integration_tests/workflow/nodes/test_llm.py:57:9
+ERROR Unexpected keyword argument `user_inputs` in function `graphon.runtime.variable_pool.VariablePool.__init__` [unexpected-keyword]
+  --> tests/integration_tests/workflow/nodes/test_llm.py:65:9
+ERROR Unexpected keyword argument `environment_variables` in function `graphon.runtime.variable_pool.VariablePool.__init__` [unexpected-keyword]
+  --> tests/integration_tests/workflow/nodes/test_llm.py:66:9
+ERROR Unexpected keyword argument `conversation_variables` in function `graphon.runtime.variable_pool.VariablePool.__init__` [unexpected-keyword]
+  --> tests/integration_tests/workflow/nodes/test_llm.py:67:9
+ERROR Unexpected keyword argument `system_variables` in function `graphon.runtime.variable_pool.VariablePool.__init__` [unexpected-keyword]
+  --> tests/integration_tests/workflow/nodes/test_parameter_extractor.py:60:9
+ERROR Unexpected keyword argument `user_inputs` in function `graphon.runtime.variable_pool.VariablePool.__init__` [unexpected-keyword]
+  --> tests/integration_tests/workflow/nodes/test_parameter_extractor.py:63:9
+ERROR Unexpected keyword argument `environment_variables` in function `graphon.runtime.variable_pool.VariablePool.__init__` [unexpected-keyword]
+  --> tests/integration_tests/workflow/nodes/test_parameter_extractor.py:64:9
+ERROR Unexpected keyword argument `conversation_variables` in function `graphon.runtime.variable_pool.VariablePool.__init__` [unexpected-keyword]
+  --> tests/integration_tests/workflow/nodes/test_parameter_extractor.py:65:9
+ERROR Unexpected keyword argument `system_variables` in function `graphon.runtime.variable_pool.VariablePool.__init__` [unexpected-keyword]
+  --> tests/integration_tests/workflow/nodes/test_template_transform.py:70:9
+ERROR Unexpected keyword argument `user_inputs` in function `graphon.runtime.variable_pool.VariablePool.__init__` [unexpected-keyword]
+  --> tests/integration_tests/workflow/nodes/test_template_transform.py:71:9
+ERROR Unexpected keyword argument `environment_variables` in function `graphon.runtime.variable_pool.VariablePool.__init__` [unexpected-keyword]
+  --> tests/integration_tests/workflow/nodes/test_template_transform.py:72:9
+ERROR Unexpected keyword argument `conversation_variables` in function `graphon.runtime.variable_pool.VariablePool.__init__` [unexpected-keyword]
+  --> tests/integration_tests/workflow/nodes/test_template_transform.py:73:9
 ERROR Argument `_SimpleJinja2Renderer` is not assignable to parameter `jinja2_template_renderer` with type `Jinja2TemplateRenderer` in function `graphon.nodes.template_transform.template_transform_node.TemplateTransformNode.__init__` [bad-argument-type]
   --> tests/integration_tests/workflow/nodes/test_template_transform.py:94:34
+ERROR Unexpected keyword argument `system_variables` in function `graphon.runtime.variable_pool.VariablePool.__init__` [unexpected-keyword]
+  --> tests/integration_tests/workflow/nodes/test_tool.py:45:9
+ERROR Unexpected keyword argument `user_inputs` in function `graphon.runtime.variable_pool.VariablePool.__init__` [unexpected-keyword]
+  --> tests/integration_tests/workflow/nodes/test_tool.py:46:9
+ERROR Unexpected keyword argument `environment_variables` in function `graphon.runtime.variable_pool.VariablePool.__init__` [unexpected-keyword]
+  --> tests/integration_tests/workflow/nodes/test_tool.py:47:9
+ERROR Unexpected keyword argument `conversation_variables` in function `graphon.runtime.variable_pool.VariablePool.__init__` [unexpected-keyword]
+  --> tests/integration_tests/workflow/nodes/test_tool.py:48:9
 ERROR `dict[str, str | None]` is not assignable to attribute `env` with type `dict[str, str]` [bad-assignment]
    --> tests/test_containers_integration_tests/conftest.py:204:39
 ERROR Argument value `Literal[0]` violates Pydantic `ge` constraint `Literal[1]` for field `page` [bad-argument-type]
@@ -681,23 +751,23 @@
 ERROR Attribute `workflow_run_service` of class `TestPauseStatePersistenceLayerTestContainers` is a read-only descriptor with no `__set__` and cannot be set [read-only]
    --> tests/test_containers_integration_tests/core/app/layers/test_pause_state_persist_layer.py:157:9
 ERROR Argument `dict[str, str]` is not assignable to parameter `outputs` with type `dict[str, object] | None` in function `TestPauseStatePersistenceLayerTestContainers._create_graph_runtime_state` [bad-argument-type]
-   --> tests/test_containers_integration_tests/core/app/layers/test_pause_state_persist_layer.py:310:21
+   --> tests/test_containers_integration_tests/core/app/layers/test_pause_state_persist_layer.py:312:21
 ERROR Argument `dict[tuple[str, str], dict[str, str] | str]` is not assignable to parameter `variables` with type `dict[tuple[str, str], object] | None` in function `TestPauseStatePersistenceLayerTestContainers._create_graph_runtime_state` [bad-argument-type]
-   --> tests/test_containers_integration_tests/core/app/layers/test_pause_state_persist_layer.py:313:23
+   --> tests/test_containers_integration_tests/core/app/layers/test_pause_state_persist_layer.py:315:23
 ERROR Argument `dict[tuple[str, str], dict[str, str] | list[int] | str]` is not assignable to parameter `variables` with type `dict[tuple[str, str], object] | None` in function `TestPauseStatePersistenceLayerTestContainers._create_graph_runtime_state` [bad-argument-type]
-   --> tests/test_containers_integration_tests/core/app/layers/test_pause_state_persist_layer.py:377:23
+   --> tests/test_containers_integration_tests/core/app/layers/test_pause_state_persist_layer.py:379:23
 ERROR Object of class `FixtureFunctionDefinition` has no attribute `_workflow_run_repo` [missing-attribute]
-   --> tests/test_containers_integration_tests/core/app/layers/test_pause_state_persist_layer.py:389:24
+   --> tests/test_containers_integration_tests/core/app/layers/test_pause_state_persist_layer.py:391:24
 ERROR Object of class `AdvancedChatAppGenerateEntity` has no attribute `workflow_execution_id` [missing-attribute]
-   --> tests/test_containers_integration_tests/core/app/layers/test_pause_state_persist_layer.py:403:16
+   --> tests/test_containers_integration_tests/core/app/layers/test_pause_state_persist_layer.py:405:16
 ERROR Argument `dict[str, LiteralString]` is not assignable to parameter `outputs` with type `dict[str, object] | None` in function `TestPauseStatePersistenceLayerTestContainers._create_graph_runtime_state` [bad-argument-type]
-   --> tests/test_containers_integration_tests/core/app/layers/test_pause_state_persist_layer.py:444:21
+   --> tests/test_containers_integration_tests/core/app/layers/test_pause_state_persist_layer.py:446:21
 ERROR Object of class `AdvancedChatAppGenerateEntity` has no attribute `workflow_execution_id` [missing-attribute]
-   --> tests/test_containers_integration_tests/core/app/layers/test_pause_state_persist_layer.py:468:16
+   --> tests/test_containers_integration_tests/core/app/layers/test_pause_state_persist_layer.py:470:16
 ERROR Object of class `FixtureFunctionDefinition` has no attribute `_workflow_run_repo` [missing-attribute]
-   --> tests/test_containers_integration_tests/core/app/layers/test_pause_state_persist_layer.py:530:24
+   --> tests/test_containers_integration_tests/core/app/layers/test_pause_state_persist_layer.py:532:24
 ERROR Object of class `AdvancedChatAppGenerateEntity` has no attribute `workflow_execution_id` [missing-attribute]
-   --> tests/test_containers_integration_tests/core/app/layers/test_pause_state_persist_layer.py:533:16
+   --> tests/test_containers_integration_tests/core/app/layers/test_pause_state_persist_layer.py:535:16
 ERROR Object of class `NoneType` has no attribute `id` [missing-attribute]
   --> tests/test_containers_integration_tests/core/rag/retrieval/test_dataset_retrieval_integration.py:36:23
 ERROR Object of class `NoneType` has no attribute `id` [missing-attribute]
@@ -758,6 +828,12 @@
    --> tests/test_containers_integration_tests/core/rag/retrieval/test_dataset_retrieval_integration.py:574:23
 ERROR Object of class `NoneType` has no attribute `id` [missing-attribute]
    --> tests/test_containers_integration_tests/core/rag/retrieval/test_dataset_retrieval_integration.py:584:23
+ERROR Unexpected keyword argument `system_variables` in function `graphon.runtime.variable_pool.VariablePool.__init__` [unexpected-keyword]
+  --> tests/test_containers_integration_tests/core/workflow/test_human_input_resume_node_execution.py:70:9
+ERROR Unexpected keyword argument `user_inputs` in function `graphon.runtime.variable_pool.VariablePool.__init__` [unexpected-keyword]
+  --> tests/test_containers_integration_tests/core/workflow/test_human_input_resume_node_execution.py:76:9
+ERROR Unexpected keyword argument `conversation_variables` in function `graphon.runtime.variable_pool.VariablePool.__init__` [unexpected-keyword]
+  --> tests/test_containers_integration_tests/core/workflow/test_human_input_resume_node_execution.py:77:9
 ERROR Argument `Connection | Engine` is not assignable to parameter `session_factory` with type `Engine | sessionmaker[Unknown]` in function `core.repositories.sqlalchemy_workflow_execution_repository.SQLAlchemyWorkflowExecutionRepository.__init__` [bad-argument-type]
    --> tests/test_containers_integration_tests/core/workflow/test_human_input_resume_node_execution.py:264:29
 ERROR Argument `Connection | Engine` is not assignable to parameter `session_factory` with type `Engine | sessionmaker[Unknown]` in function `core.repositories.sqlalchemy_workflow_node_execution_repository.SQLAlchemyWorkflowNodeExecutionRepository.__init__` [bad-argument-type]
@@ -2726,6 +2802,8 @@
   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline_core.py:99:23
 ERROR Argument `(**kwargs: Unknown) -> None` is not assignable to parameter `draft_var_saver_factory` with type `DraftVariableSaverFactory` in function `core.app.apps.advanced_chat.generate_task_pipeline.AdvancedChatAppGenerateTaskPipeline.__init__` [bad-argument-type]
    --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline_core.py:105:33
+ERROR Unexpected keyword argument `system_variables` in function `graphon.runtime.variable_pool.VariablePool.__init__` [unexpected-keyword]
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline_core.py:135:40
 ERROR `SimpleNamespace` is not assignable to attribute `_message_cycle_manager` with type `MessageCycleManager` [bad-assignment]
    --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline_core.py:183:43
 ERROR Argument `SimpleNamespace` is not assignable to parameter `event` with type `QueueTextChunkEvent` in function `core.app.apps.advanced_chat.generate_task_pipeline.AdvancedChatAppGenerateTaskPipeline._handle_text_chunk_event` [bad-argument-type]
@@ -2761,43 +2839,43 @@
 ERROR `(**kwargs: Unknown) -> Literal['loop_done']` is not assignable to attribute `workflow_loop_completed_to_stream_response` with type `(self: WorkflowResponseConverter, *, task_id: str, workflow_execution_id: str, event: QueueLoopCompletedEvent) -> LoopNodeCompletedStreamResponse` [bad-assignment]
    --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline_core.py:313:92
 ERROR `(**kwargs: Unknown) -> Literal['finish']` is not assignable to attribute `workflow_finish_to_stream_response` with type `(self: WorkflowResponseConverter, *, task_id: str, workflow_id: str, status: WorkflowExecutionStatus, graph_runtime_state: GraphRuntimeState, error: str | None = None, exceptions_count: int = 0) -> WorkflowFinishStreamResponse` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline_core.py:378:84
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline_core.py:380:84
 ERROR `(**kwargs: Unknown) -> list[str]` is not assignable to attribute `workflow_pause_to_stream_response` with type `(self: WorkflowResponseConverter, *, event: QueueWorkflowPausedEvent, task_id: str, graph_runtime_state: GraphRuntimeState) -> list[StreamResponse]` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline_core.py:379:83
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline_core.py:381:83
 ERROR `(err: Unknown) -> Unknown` is not assignable to attribute `error_to_stream_response` with type `(self: BasedGenerateTaskPipeline, e: Exception) -> ErrorStreamResponse` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline_core.py:384:65
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline_core.py:386:65
 ERROR `(**kwargs: Unknown) -> SimpleNamespace` is not assignable to attribute `_get_message` with type `(self: AdvancedChatAppGenerateTaskPipeline, *, session: Session) -> Message` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline_core.py:385:33
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline_core.py:387:33
 ERROR `(**kwargs: Unknown) -> Literal['node_finish']` is not assignable to attribute `workflow_node_finish_to_stream_response` with type `(self: WorkflowResponseConverter, *, event: QueueNodeExceptionEvent | QueueNodeFailedEvent | QueueNodeSucceededEvent, task_id: str) -> NodeFinishStreamResponse | None` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline_core.py:416:89
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline_core.py:418:89
 ERROR `SimpleNamespace` is not assignable to attribute `_message_cycle_manager` with type `MessageCycleManager` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline_core.py:451:43
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline_core.py:453:43
 ERROR Argument `SimpleNamespace` is not assignable to parameter `event` with type `QueueTextChunkEvent` in function `core.app.apps.advanced_chat.generate_task_pipeline.AdvancedChatAppGenerateTaskPipeline._handle_text_chunk_event` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline_core.py:457:47
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline_core.py:459:47
 ERROR Argument `TestAdvancedChatGenerateTaskPipeline.test_handle_text_chunk_event_tracks_streaming_metrics._Publisher` is not assignable to parameter `tts_publisher` with type `AppGeneratorTTSPublisher | None` in function `core.app.apps.advanced_chat.generate_task_pipeline.AdvancedChatAppGenerateTaskPipeline._handle_text_chunk_event` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline_core.py:457:68
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline_core.py:459:68
 ERROR Argument `SimpleNamespace` is not assignable to parameter `queue_message` with type `MessageQueueMessage | WorkflowQueueMessage | None` in function `core.app.apps.advanced_chat.generate_task_pipeline.AdvancedChatAppGenerateTaskPipeline._handle_text_chunk_event` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline_core.py:457:96
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline_core.py:459:96
 ERROR `TestAdvancedChatGenerateTaskPipeline.test_handle_output_moderation_chunk_appends_token._Moderation` is not assignable to attribute `output_moderation_handler` with type `OutputModeration | None` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline_core.py:478:66
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline_core.py:480:66
 ERROR `(**kwargs: Unknown) -> Literal['replace']` is not assignable to attribute `message_replace_to_stream_response` with type `(self: MessageCycleManager, answer: str, reason: str = '') -> MessageReplaceStreamResponse` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline_core.py:507:78
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline_core.py:509:78
 ERROR `(**kwargs: Unknown) -> Literal['filled']` is not assignable to attribute `human_input_form_filled_to_stream_response` with type `(self: WorkflowResponseConverter, *, event: QueueHumanInputFormFilledEvent, task_id: str) -> HumanInputFormFilledResponse` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline_core.py:520:92
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline_core.py:522:92
 ERROR `(**kwargs: Unknown) -> Literal['timeout']` is not assignable to attribute `human_input_form_timeout_to_stream_response` with type `(self: WorkflowResponseConverter, *, event: QueueHumanInputFormTimeoutEvent, task_id: str) -> HumanInputFormTimeoutResponse` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline_core.py:521:93
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline_core.py:523:93
 ERROR Argument `TestAdvancedChatGenerateTaskPipeline.test_save_message_strips_markdown_and_sets_usage._Session` is not assignable to parameter `session` with type `Session` in function `core.app.apps.advanced_chat.generate_task_pipeline.AdvancedChatAppGenerateTaskPipeline._save_message` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline_core.py:590:40
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline_core.py:594:40
 ERROR `() -> Literal['end']` is not assignable to attribute `_message_end_to_stream_response` with type `(self: AdvancedChatAppGenerateTaskPipeline) -> MessageEndStreamResponse` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline_core.py:598:52
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline_core.py:602:52
 ERROR `(answer: Unknown) -> Literal['safe']` is not assignable to attribute `handle_output_moderation_when_task_finished` with type `(self: BasedGenerateTaskPipeline, completion: str) -> str | None` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline_core.py:623:84
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline_core.py:629:84
 ERROR `(**kwargs: Unknown) -> Literal['replace']` is not assignable to attribute `message_replace_to_stream_response` with type `(self: MessageCycleManager, answer: str, reason: str = '') -> MessageReplaceStreamResponse` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline_core.py:624:78
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline_core.py:630:78
 ERROR `() -> Literal['end']` is not assignable to attribute `_message_end_to_stream_response` with type `(self: AdvancedChatAppGenerateTaskPipeline) -> MessageEndStreamResponse` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline_core.py:625:52
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline_core.py:631:52
 ERROR `(**kwargs: Unknown) -> Literal['failed']` is not assignable to attribute `workflow_node_finish_to_stream_response` with type `(self: WorkflowResponseConverter, *, event: QueueNodeExceptionEvent | QueueNodeFailedEvent | QueueNodeSucceededEvent, task_id: str) -> NodeFinishStreamResponse | None` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline_core.py:647:89
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline_core.py:653:89
 ERROR Argument `dict[str, dict[str, str]]` is not assignable to parameter `override_config_dict` with type `AppModelConfigDict | None` in function `core.app.apps.agent_chat.app_config_manager.AgentChatAppConfigManager.get_app_config` [bad-argument-type]
   --> tests/unit_tests/core/app/apps/agent_chat/test_agent_chat_app_config_manager.py:40:34
 ERROR Cannot index into `str` [bad-index]
@@ -3032,8 +3110,6 @@
    --> tests/unit_tests/core/app/apps/test_message_based_app_generator.py:175:9
 ERROR Object of class `ModuleType` has no attribute `TraceQueueManager` [missing-attribute]
   --> tests/unit_tests/core/app/apps/test_pause_resume.py:42:5
-ERROR Class member `_StubToolNode._run` overrides parent class `Node` in an inconsistent manner [bad-override]
-  --> tests/unit_tests/core/app/apps/test_pause_resume.py:92:9
 ERROR Argument `SimpleNamespace` is not assignable to parameter `app_model` with type `App` in function `core.app.apps.workflow.app_generator.WorkflowAppGenerator.resume` [bad-argument-type]
    --> tests/unit_tests/core/app/apps/test_pause_resume.py:226:19
 ERROR Argument `SimpleNamespace` is not assignable to parameter `workflow` with type `Workflow` in function `core.app.apps.workflow.app_generator.WorkflowAppGenerator.resume` [bad-argument-type]
@@ -3080,32 +3156,34 @@
    --> tests/unit_tests/core/app/apps/test_streaming_utils.py:127:12
 ERROR Cannot index into `str` [bad-index]
    --> tests/unit_tests/core/app/apps/test_streaming_utils.py:128:12
+ERROR No matching overload found for function `core.app.apps.workflow.app_generator.WorkflowAppGenerator.generate` called with arguments: (app_model=SimpleNamespace, workflow=SimpleNamespace, user=SimpleNamespace, args=dict[str, dict[str, str] | list[@_] | str], invoke_from=Literal['service-api'], streaming=Literal[False], call_depth=Literal[0]) [no-matching-overload]
+  --> tests/unit_tests/core/app/apps/test_workflow_app_generator.py:69:32
 ERROR Argument `SimpleNamespace` is not assignable to parameter `application_generate_entity` with type `WorkflowAppGenerateEntity` in function `core.app.apps.workflow.app_generator.WorkflowAppGenerator.resume` [bad-argument-type]
-  --> tests/unit_tests/core/app/apps/test_workflow_app_generator.py:37:37
+   --> tests/unit_tests/core/app/apps/test_workflow_app_generator.py:109:37
 ERROR Argument `tuple[Literal['layer']]` is not assignable to parameter `graph_engine_layers` with type `Sequence[GraphEngineLayer]` in function `core.app.apps.workflow.app_generator.WorkflowAppGenerator.resume` [bad-argument-type]
-  --> tests/unit_tests/core/app/apps/test_workflow_app_generator.py:41:29
+   --> tests/unit_tests/core/app/apps/test_workflow_app_generator.py:113:29
 ERROR Argument `SimpleNamespace` is not assignable to parameter `app_model` with type `App` in function `core.app.apps.workflow.app_generator.WorkflowAppGenerator._generate` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/test_workflow_app_generator.py:109:19
+   --> tests/unit_tests/core/app/apps/test_workflow_app_generator.py:181:19
 ERROR Argument `SimpleNamespace` is not assignable to parameter `application_generate_entity` with type `WorkflowAppGenerateEntity` in function `core.app.apps.workflow.app_generator.WorkflowAppGenerator._generate` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/test_workflow_app_generator.py:112:37
+   --> tests/unit_tests/core/app/apps/test_workflow_app_generator.py:184:37
 ERROR Argument `Literal['service-api']` is not assignable to parameter `invoke_from` with type `InvokeFrom` in function `core.app.apps.workflow.app_generator.WorkflowAppGenerator._generate` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/test_workflow_app_generator.py:113:21
+   --> tests/unit_tests/core/app/apps/test_workflow_app_generator.py:185:21
 ERROR Argument `tuple[Literal['base-layer']]` is not assignable to parameter `graph_engine_layers` with type `Sequence[GraphEngineLayer]` in function `core.app.apps.workflow.app_generator.WorkflowAppGenerator._generate` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/test_workflow_app_generator.py:117:29
+   --> tests/unit_tests/core/app/apps/test_workflow_app_generator.py:189:29
 ERROR Argument `SimpleNamespace` is not assignable to parameter `pause_state_config` with type `PauseStateLayerConfig | None` in function `core.app.apps.workflow.app_generator.WorkflowAppGenerator._generate` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/test_workflow_app_generator.py:119:28
+   --> tests/unit_tests/core/app/apps/test_workflow_app_generator.py:191:28
 ERROR Cannot index into `object` [bad-index]
-   --> tests/unit_tests/core/app/apps/test_workflow_app_generator.py:123:12
+   --> tests/unit_tests/core/app/apps/test_workflow_app_generator.py:195:12
 ERROR Cannot index into `object` [bad-index]
-   --> tests/unit_tests/core/app/apps/test_workflow_app_generator.py:124:12
+   --> tests/unit_tests/core/app/apps/test_workflow_app_generator.py:196:12
 ERROR Argument `SimpleNamespace` is not assignable to parameter `app_model` with type `App` in function `core.app.apps.workflow.app_generator.WorkflowAppGenerator.resume` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/test_workflow_app_generator.py:200:19
+   --> tests/unit_tests/core/app/apps/test_workflow_app_generator.py:272:19
 ERROR Argument `SimpleNamespace` is not assignable to parameter `workflow` with type `Workflow` in function `core.app.apps.workflow.app_generator.WorkflowAppGenerator.resume` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/test_workflow_app_generator.py:201:18
+   --> tests/unit_tests/core/app/apps/test_workflow_app_generator.py:273:18
 ERROR Argument `SimpleNamespace` is not assignable to parameter `application_generate_entity` with type `WorkflowAppGenerateEntity` in function `core.app.apps.workflow.app_generator.WorkflowAppGenerator.resume` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/test_workflow_app_generator.py:203:37
+   --> tests/unit_tests/core/app/apps/test_workflow_app_generator.py:275:37
 ERROR Argument `SimpleNamespace` is not assignable to parameter `pause_state_config` with type `PauseStateLayerConfig | None` in function `core.app.apps.workflow.app_generator.WorkflowAppGenerator.resume` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/test_workflow_app_generator.py:207:28
+   --> tests/unit_tests/core/app/apps/test_workflow_app_generator.py:279:28
 ERROR Argument `SimpleNamespace` is not assignable to parameter `queue_manager` with type `AppQueueManager` in function `core.app.apps.workflow_app_runner.WorkflowBasedAppRunner.__init__` [bad-argument-type]
   --> tests/unit_tests/core/app/apps/test_workflow_app_runner_core.py:47:55
 ERROR Argument `SimpleNamespace` is not assignable to parameter `queue_manager` with type `AppQueueManager` in function `core.app.apps.workflow_app_runner.WorkflowBasedAppRunner.__init__` [bad-argument-type]
@@ -3160,6 +3238,14 @@
   --> tests/unit_tests/core/app/apps/test_workflow_app_runner_notifications.py:36:51
 ERROR Argument `_DummyWorkflowEntry` is not assignable to parameter `workflow_entry` with type `WorkflowEntry` in function `core.app.apps.workflow_app_runner.WorkflowBasedAppRunner._handle_event` [bad-argument-type]
   --> tests/unit_tests/core/app/apps/test_workflow_app_runner_notifications.py:52:26
+ERROR Unexpected keyword argument `system_variables` in function `graphon.runtime.variable_pool.VariablePool.__init__` [unexpected-keyword]
+  --> tests/unit_tests/core/app/apps/test_workflow_app_runner_single_node.py:20:9
+ERROR Unexpected keyword argument `user_inputs` in function `graphon.runtime.variable_pool.VariablePool.__init__` [unexpected-keyword]
+  --> tests/unit_tests/core/app/apps/test_workflow_app_runner_single_node.py:21:9
+ERROR Unexpected keyword argument `environment_variables` in function `graphon.runtime.variable_pool.VariablePool.__init__` [unexpected-keyword]
+  --> tests/unit_tests/core/app/apps/test_workflow_app_runner_single_node.py:22:9
+ERROR Unexpected keyword argument `conversation_variables` in function `graphon.runtime.variable_pool.VariablePool.__init__` [unexpected-keyword]
+  --> tests/unit_tests/core/app/apps/test_workflow_app_runner_single_node.py:23:9
 ERROR Argument `SimpleNamespace` is not assignable to parameter `workflow_entry` with type `WorkflowEntry` in function `core.app.apps.workflow_app_runner.WorkflowBasedAppRunner._handle_event` [bad-argument-type]
   --> tests/unit_tests/core/app/apps/test_workflow_pause_events.py:84:26
 ERROR Argument `SimpleNamespace` is not assignable to parameter `application_generate_entity` with type `AdvancedChatAppGenerateEntity | WorkflowAppGenerateEntity` in function `core.app.apps.common.workflow_response_converter.WorkflowResponseConverter.__init__` [bad-argument-type]
@@ -3230,6 +3316,8 @@
   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:86:14
 ERROR Argument `(**kwargs: Unknown) -> None` is not assignable to parameter `draft_var_saver_factory` with type `DraftVariableSaverFactory` in function `core.app.apps.workflow.generate_task_pipeline.WorkflowAppGenerateTaskPipeline.__init__` [bad-argument-type]
   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:88:33
+ERROR Unexpected keyword argument `system_variables` in function `graphon.runtime.variable_pool.VariablePool.__init__` [unexpected-keyword]
+  --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:98:40
 ERROR Argument `SimpleNamespace` is not assignable to parameter `publisher` with type `AppGeneratorTTSPublisher | None` in function `core.app.apps.workflow.generate_task_pipeline.WorkflowAppGenerateTaskPipeline._listen_audio_msg` [bad-argument-type]
    --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:169:57
 ERROR `(err: Unknown) -> Unknown` is not assignable to attribute `error_to_stream_response` with type `(self: BasedGenerateTaskPipeline, e: Exception) -> ErrorStreamResponse` [bad-assignment]
@@ -3251,101 +3339,101 @@
 ERROR `(**kwargs: Unknown) -> Literal['done']` is not assignable to attribute `workflow_node_finish_to_stream_response` with type `(self: WorkflowResponseConverter, *, event: QueueNodeExceptionEvent | QueueNodeFailedEvent | QueueNodeSucceededEvent, task_id: str) -> NodeFinishStreamResponse | None` [bad-assignment]
    --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:267:89
 ERROR `(**kwargs: Unknown) -> Literal['finish']` is not assignable to attribute `workflow_finish_to_stream_response` with type `(self: WorkflowResponseConverter, *, task_id: str, workflow_id: str, status: WorkflowExecutionStatus, graph_runtime_state: GraphRuntimeState, error: str | None = None, exceptions_count: int = 0) -> WorkflowFinishStreamResponse` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:289:84
+   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:291:84
 ERROR Argument `TestWorkflowGenerateTaskPipeline.test_save_workflow_app_log_created_from._Session` is not assignable to parameter `session` with type `Session` in function `core.app.apps.workflow.generate_task_pipeline.WorkflowAppGenerateTaskPipeline._save_workflow_app_log` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:309:49
+   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:311:49
 ERROR `(**kwargs: Unknown) -> Literal['iter']` is not assignable to attribute `workflow_iteration_start_to_stream_response` with type `(self: WorkflowResponseConverter, *, task_id: str, workflow_execution_id: str, event: QueueIterationStartEvent) -> IterationNodeStartStreamResponse` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:316:93
+   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:318:93
 ERROR `(**kwargs: Unknown) -> Literal['next']` is not assignable to attribute `workflow_iteration_next_to_stream_response` with type `(self: WorkflowResponseConverter, *, task_id: str, workflow_execution_id: str, event: QueueIterationNextEvent) -> IterationNodeNextStreamResponse` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:317:92
+   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:319:92
 ERROR `(**kwargs: Unknown) -> Literal['done']` is not assignable to attribute `workflow_iteration_completed_to_stream_response` with type `(self: WorkflowResponseConverter, *, task_id: str, workflow_execution_id: str, event: QueueIterationCompletedEvent) -> IterationNodeCompletedStreamResponse` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:318:97
+   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:320:97
 ERROR `(**kwargs: Unknown) -> Literal['loop']` is not assignable to attribute `workflow_loop_start_to_stream_response` with type `(self: WorkflowResponseConverter, *, task_id: str, workflow_execution_id: str, event: QueueLoopStartEvent) -> LoopNodeStartStreamResponse` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:319:88
+   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:321:88
 ERROR `(**kwargs: Unknown) -> Literal['loop_next']` is not assignable to attribute `workflow_loop_next_to_stream_response` with type `(self: WorkflowResponseConverter, *, task_id: str, workflow_execution_id: str, event: QueueLoopNextEvent) -> LoopNodeNextStreamResponse` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:320:87
+   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:322:87
 ERROR `(**kwargs: Unknown) -> Literal['loop_done']` is not assignable to attribute `workflow_loop_completed_to_stream_response` with type `(self: WorkflowResponseConverter, *, task_id: str, workflow_execution_id: str, event: QueueLoopCompletedEvent) -> LoopNodeCompletedStreamResponse` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:321:92
+   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:323:92
 ERROR `(**kwargs: Unknown) -> Literal['filled']` is not assignable to attribute `human_input_form_filled_to_stream_response` with type `(self: WorkflowResponseConverter, *, event: QueueHumanInputFormFilledEvent, task_id: str) -> HumanInputFormFilledResponse` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:322:92
+   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:324:92
 ERROR `(**kwargs: Unknown) -> Literal['timeout']` is not assignable to attribute `human_input_form_timeout_to_stream_response` with type `(self: WorkflowResponseConverter, *, event: QueueHumanInputFormTimeoutEvent, task_id: str) -> HumanInputFormTimeoutResponse` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:323:93
+   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:325:93
 ERROR `(**kwargs: Unknown) -> Literal['log']` is not assignable to attribute `handle_agent_log` with type `(self: WorkflowResponseConverter, task_id: str, event: QueueAgentLogEvent) -> AgentLogStreamResponse` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:324:66
+   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:326:66
 ERROR `(**kwargs: Unknown) -> Iterator[Any]` is not assignable to attribute `_process_stream_response` with type `(self: WorkflowAppGenerateTaskPipeline, tts_publisher: AppGeneratorTTSPublisher | None = None, trace_manager: TraceQueueManager | None = None) -> Generator[StreamResponse]` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:416:45
+   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:418:45
 ERROR Argument `SimpleNamespace` is not assignable to parameter `workflow` with type `Workflow` in function `core.app.apps.workflow.generate_task_pipeline.WorkflowAppGenerateTaskPipeline.__init__` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:472:22
+   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:474:22
 ERROR Argument `SimpleNamespace` is not assignable to parameter `queue_manager` with type `AppQueueManager` in function `core.app.apps.workflow.generate_task_pipeline.WorkflowAppGenerateTaskPipeline.__init__` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:473:27
+   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:475:27
 ERROR Argument `(**kwargs: Unknown) -> None` is not assignable to parameter `draft_var_saver_factory` with type `DraftVariableSaverFactory` in function `core.app.apps.workflow.generate_task_pipeline.WorkflowAppGenerateTaskPipeline.__init__` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:476:37
+   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:478:37
 ERROR `(**kwargs: Unknown) -> Iterator[Any]` is not assignable to attribute `_wrapper_process_stream_response` with type `(self: WorkflowAppGenerateTaskPipeline, trace_manager: TraceQueueManager | None = None) -> Generator[StreamResponse]` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:485:53
+   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:487:53
 ERROR Object of class `tuple` has no attribute `workflow_run_id` [missing-attribute]
-   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:489:16
+   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:491:16
 ERROR `(**kwargs: Unknown) -> Iterator[Any]` is not assignable to attribute `_wrapper_process_stream_response` with type `(self: WorkflowAppGenerateTaskPipeline, trace_manager: TraceQueueManager | None = None) -> Generator[StreamResponse]` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:492:53
+   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:494:53
 ERROR Object of class `Generator` has no attribute `workflow_run_id` [missing-attribute]
-   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:514:16
+   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:516:16
 ERROR `(**kwargs: Unknown) -> Iterator[Any]` is not assignable to attribute `_process_stream_response` with type `(self: WorkflowAppGenerateTaskPipeline, tts_publisher: AppGeneratorTTSPublisher | None = None, trace_manager: TraceQueueManager | None = None) -> Generator[StreamResponse]` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:558:45
+   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:560:45
 ERROR `(**kwargs: Unknown) -> Iterator[Any]` is not assignable to attribute `_process_stream_response` with type `(self: WorkflowAppGenerateTaskPipeline, tts_publisher: AppGeneratorTTSPublisher | None = None, trace_manager: TraceQueueManager | None = None) -> Generator[StreamResponse]` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:568:45
+   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:570:45
 ERROR `(**kwargs: Unknown) -> Iterator[Any]` is not assignable to attribute `_process_stream_response` with type `(self: WorkflowAppGenerateTaskPipeline, tts_publisher: AppGeneratorTTSPublisher | None = None, trace_manager: TraceQueueManager | None = None) -> Generator[StreamResponse]` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:605:45
+   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:607:45
 ERROR `(**kwargs: Unknown) -> Literal['retry']` is not assignable to attribute `workflow_node_retry_to_stream_response` with type `(self: WorkflowResponseConverter, *, event: QueueNodeRetryEvent, task_id: str) -> NodeRetryStreamResponse | None` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:695:88
+   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:697:88
 ERROR `(**kwargs: Unknown) -> Literal['started']` is not assignable to attribute `workflow_node_start_to_stream_response` with type `(self: WorkflowResponseConverter, *, event: QueueNodeStartedEvent, task_id: str) -> NodeStartStreamResponse | None` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:700:88
+   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:702:88
 ERROR `(**kwargs: Unknown) -> Literal['failed']` is not assignable to attribute `workflow_node_finish_to_stream_response` with type `(self: WorkflowResponseConverter, *, event: QueueNodeExceptionEvent | QueueNodeFailedEvent | QueueNodeSucceededEvent, task_id: str) -> NodeFinishStreamResponse | None` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:706:89
+   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:708:89
 ERROR `(**kwargs: Unknown) -> Literal['finish']` is not assignable to attribute `workflow_finish_to_stream_response` with type `(self: WorkflowResponseConverter, *, task_id: str, workflow_id: str, status: WorkflowExecutionStatus, graph_runtime_state: GraphRuntimeState, error: str | None = None, exceptions_count: int = 0) -> WorkflowFinishStreamResponse` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:732:84
+   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:736:84
 ERROR `(**kwargs: Unknown) -> list[str]` is not assignable to attribute `workflow_pause_to_stream_response` with type `(self: WorkflowResponseConverter, *, event: QueueWorkflowPausedEvent, task_id: str, graph_runtime_state: GraphRuntimeState) -> list[StreamResponse]` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:740:83
+   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:744:83
 ERROR `(event: QueuePingEvent, **kwargs: Unknown) -> Iterator[PingStreamResponse] | Iterator[Any]` is not assignable to attribute `_handle_ping_event` with type `(self: WorkflowAppGenerateTaskPipeline, event: QueuePingEvent, **kwargs: Unknown) -> Generator[PingStreamResponse]` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:759:39
+   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:765:39
 ERROR `(event: QueueStopEvent | QueueWorkflowFailedEvent, **kwargs: Unknown) -> Iterator[StreamResponse] | Iterator[Any]` is not assignable to attribute `_handle_workflow_failed_and_stop_events` with type `(self: WorkflowAppGenerateTaskPipeline, event: QueueStopEvent | QueueWorkflowFailedEvent, *, trace_manager: TraceQueueManager | None = None, **kwargs: Unknown) -> Generator[StreamResponse]` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:762:60
+   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:768:60
 ERROR Argument `SimpleNamespace` is not assignable to parameter `event` with type `AppQueueEvent` in function `core.app.apps.workflow.generate_task_pipeline.WorkflowAppGenerateTaskPipeline._dispatch_event` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:767:46
+   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:773:46
 ERROR `() -> Iterator[MessageQueueMessage | WorkflowQueueMessage] | Iterator[Any]` is not assignable to attribute `listen` with type `(self: AppQueueManager) -> Generator[MessageQueueMessage | WorkflowQueueMessage, Unknown]` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:775:61
+   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:783:61
 ERROR `(event: QueueWorkflowStartedEvent, **kwargs: Unknown) -> Iterator[StreamResponse] | Iterator[Any]` is not assignable to attribute `_handle_workflow_started_event` with type `(self: WorkflowAppGenerateTaskPipeline, event: QueueWorkflowStartedEvent, **kwargs: Unknown) -> Generator[StreamResponse]` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:783:51
+   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:791:51
 ERROR `(event: QueueTextChunkEvent, **kwargs: Unknown) -> Iterator[StreamResponse] | Iterator[Any]` is not assignable to attribute `_handle_text_chunk_event` with type `(self: WorkflowAppGenerateTaskPipeline, event: QueueTextChunkEvent, *, tts_publisher: AppGeneratorTTSPublisher | None = None, queue_message: MessageQueueMessage | WorkflowQueueMessage | None = None, **kwargs: Unknown) -> Generator[StreamResponse]` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:784:45
+   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:792:45
 ERROR `(event: AppQueueEvent, **kwargs: Unknown) -> Iterator[StreamResponse] | Iterator[Any]` is not assignable to attribute `_dispatch_event` with type `(self: WorkflowAppGenerateTaskPipeline, event: AppQueueEvent, *, tts_publisher: AppGeneratorTTSPublisher | None = None, trace_manager: TraceQueueManager | None = None, queue_message: MessageQueueMessage | WorkflowQueueMessage | None = None) -> Generator[StreamResponse]` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:785:36
+   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:793:36
 ERROR `(event: QueueErrorEvent, **kwargs: Unknown) -> Iterator[ErrorStreamResponse] | Iterator[Any]` is not assignable to attribute `_handle_error_event` with type `(self: WorkflowAppGenerateTaskPipeline, event: QueueErrorEvent, **kwargs: Unknown) -> Generator[ErrorStreamResponse]` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:786:40
+   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:794:40
 ERROR Argument `TestWorkflowGenerateTaskPipeline.test_process_stream_response_main_match_paths_and_cleanup._Publisher` is not assignable to parameter `tts_publisher` with type `AppGeneratorTTSPublisher | None` in function `core.app.apps.workflow.generate_task_pipeline.WorkflowAppGenerateTaskPipeline._process_stream_response` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:793:74
+   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:801:74
 ERROR `() -> Iterator[MessageQueueMessage | WorkflowQueueMessage] | Iterator[Any]` is not assignable to attribute `listen` with type `(self: AppQueueManager) -> Generator[MessageQueueMessage | WorkflowQueueMessage, Unknown]` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:800:61
+   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:808:61
 ERROR `(event: QueueStopEvent | QueueWorkflowFailedEvent, **kwargs: Unknown) -> Iterator[StreamResponse] | Iterator[Any]` is not assignable to attribute `_handle_workflow_failed_and_stop_events` with type `(self: WorkflowAppGenerateTaskPipeline, event: QueueStopEvent | QueueWorkflowFailedEvent, *, trace_manager: TraceQueueManager | None = None, **kwargs: Unknown) -> Generator[StreamResponse]` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:803:60
+   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:811:60
 ERROR `() -> Iterator[MessageQueueMessage | WorkflowQueueMessage] | Iterator[Any]` is not assignable to attribute `listen` with type `(self: AppQueueManager) -> Generator[MessageQueueMessage | WorkflowQueueMessage, Unknown]` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:806:61
+   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:814:61
 ERROR `(event: QueueWorkflowPausedEvent, **kwargs: Unknown) -> Iterator[StreamResponse] | Iterator[Any]` is not assignable to attribute `_handle_workflow_paused_event` with type `(self: WorkflowAppGenerateTaskPipeline, event: QueueWorkflowPausedEvent, **kwargs: Unknown) -> Generator[StreamResponse]` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:809:50
+   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:817:50
 ERROR `() -> Iterator[MessageQueueMessage | WorkflowQueueMessage] | Iterator[Any]` is not assignable to attribute `listen` with type `(self: AppQueueManager) -> Generator[MessageQueueMessage | WorkflowQueueMessage, Unknown]` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:812:61
+   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:820:61
 ERROR `(event: QueueStopEvent | QueueWorkflowFailedEvent, **kwargs: Unknown) -> Iterator[StreamResponse] | Iterator[Any]` is not assignable to attribute `_handle_workflow_failed_and_stop_events` with type `(self: WorkflowAppGenerateTaskPipeline, event: QueueStopEvent | QueueWorkflowFailedEvent, *, trace_manager: TraceQueueManager | None = None, **kwargs: Unknown) -> Generator[StreamResponse]` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:815:60
+   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:823:60
 ERROR Argument `TestWorkflowGenerateTaskPipeline.test_save_workflow_app_log_covers_invoke_from_variants._Session` is not assignable to parameter `session` with type `Session` in function `core.app.apps.workflow.generate_task_pipeline.WorkflowAppGenerateTaskPipeline._save_workflow_app_log` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:828:49
+   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:836:49
 ERROR Object of class `object` has no attribute `created_from` [missing-attribute]
-   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:829:16
+   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:837:16
 ERROR Argument `TestWorkflowGenerateTaskPipeline.test_save_workflow_app_log_covers_invoke_from_variants._Session` is not assignable to parameter `session` with type `Session` in function `core.app.apps.workflow.generate_task_pipeline.WorkflowAppGenerateTaskPipeline._save_workflow_app_log` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:832:49
+   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:840:49
 ERROR Argument `TestWorkflowGenerateTaskPipeline.test_save_workflow_app_log_covers_invoke_from_variants._Session` is not assignable to parameter `session` with type `Session` in function `core.app.apps.workflow.generate_task_pipeline.WorkflowAppGenerateTaskPipeline._save_workflow_app_log` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:837:49
+   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:845:49
 ERROR Argument `TestWorkflowGenerateTaskPipeline.test_save_workflow_app_log_covers_invoke_from_variants._Session` is not assignable to parameter `session` with type `Session` in function `core.app.apps.workflow.generate_task_pipeline.WorkflowAppGenerateTaskPipeline._save_workflow_app_log` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:841:49
+   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:849:49
 ERROR `(**kwargs: Unknown) -> TestWorkflowGenerateTaskPipeline.test_save_output_for_event_writes_draft_variables._Saver` is not assignable to attribute `_draft_var_saver_factory` with type `DraftVariableSaverFactory` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:857:45
+   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:865:45
 ERROR `Literal['unknown']` is not assignable to attribute `stopped_by` with type `QueueStopEvent.StopBy` [bad-assignment]
   --> tests/unit_tests/core/app/entities/test_queue_entities.py:11:28
 ERROR Cannot index into `bool` [bad-index]
@@ -3443,11 +3531,11 @@
 ERROR Argument `object` is not assignable to parameter `event` with type `GraphEngineEvent` in function `core.app.layers.suspend_layer.SuspendLayer.on_event` [bad-argument-type]
   --> tests/unit_tests/core/app/layers/test_suspend_layer.py:18:24
 ERROR Argument `SimpleNamespace` is not assignable to parameter `graph_runtime_state` with type `ReadOnlyGraphRuntimeState` in function `graphon.graph_engine.layers.base.GraphEngineLayer.initialize` [bad-argument-type]
-  --> tests/unit_tests/core/app/layers/test_trigger_post_layer.py:47:30
+  --> tests/unit_tests/core/app/layers/test_trigger_post_layer.py:49:30
 ERROR Argument `SimpleNamespace` is not assignable to parameter `graph_runtime_state` with type `ReadOnlyGraphRuntimeState` in function `graphon.graph_engine.layers.base.GraphEngineLayer.initialize` [bad-argument-type]
-  --> tests/unit_tests/core/app/layers/test_trigger_post_layer.py:84:30
+  --> tests/unit_tests/core/app/layers/test_trigger_post_layer.py:88:30
 ERROR Argument `SimpleNamespace` is not assignable to parameter `graph_runtime_state` with type `ReadOnlyGraphRuntimeState` in function `graphon.graph_engine.layers.base.GraphEngineLayer.initialize` [bad-argument-type]
-   --> tests/unit_tests/core/app/layers/test_trigger_post_layer.py:104:30
+   --> tests/unit_tests/core/app/layers/test_trigger_post_layer.py:110:30
 ERROR Argument `SimpleNamespace` is not assignable to parameter `application_generate_entity` with type `AppGenerateEntity` in function `core.app.task_pipeline.based_generate_task_pipeline.BasedGenerateTaskPipeline.__init__` [bad-argument-type]
   --> tests/unit_tests/core/app/task_pipeline/test_based_generate_task_pipeline.py:23:41
 ERROR Argument `SimpleNamespace` is not assignable to parameter `queue_manager` with type `AppQueueManager` in function `core.app.task_pipeline.easy_ui_based_generate_task_pipeline.EasyUIBasedGenerateTaskPipeline.__init__` [bad-argument-type]
@@ -3789,45 +3877,45 @@
 ERROR Argument `SimpleNamespace` is not assignable to parameter `node` with type `Node[Unknown]` in function `core.app.workflow.layers.observability.ObservabilityLayer.on_node_run_end` [bad-argument-type]
    --> tests/unit_tests/core/app/workflow/test_observability_layer_extra.py:208:31
 ERROR Argument `_RepoRecorder` is not assignable to parameter `workflow_execution_repository` with type `WorkflowExecutionRepository` in function `core.app.workflow.layers.persistence.WorkflowPersistenceLayer.__init__` [bad-argument-type]
-  --> tests/unit_tests/core/app/workflow/test_persistence_layer.py:93:39
+  --> tests/unit_tests/core/app/workflow/test_persistence_layer.py:97:39
 ERROR Argument `_RepoRecorder` is not assignable to parameter `workflow_node_execution_repository` with type `WorkflowNodeExecutionRepository` in function `core.app.workflow.layers.persistence.WorkflowPersistenceLayer.__init__` [bad-argument-type]
-  --> tests/unit_tests/core/app/workflow/test_persistence_layer.py:94:44
+  --> tests/unit_tests/core/app/workflow/test_persistence_layer.py:98:44
 ERROR Argument `object | None` is not assignable to parameter `trace_manager` with type `TraceQueueManager | None` in function `core.app.workflow.layers.persistence.WorkflowPersistenceLayer.__init__` [bad-argument-type]
-  --> tests/unit_tests/core/app/workflow/test_persistence_layer.py:95:23
+  --> tests/unit_tests/core/app/workflow/test_persistence_layer.py:99:23
 ERROR Argument `None` is not assignable to parameter `command_channel` with type `CommandChannel` in function `graphon.graph_engine.layers.base.GraphEngineLayer.initialize` [bad-argument-type]
-  --> tests/unit_tests/core/app/workflow/test_persistence_layer.py:97:55
+   --> tests/unit_tests/core/app/workflow/test_persistenc\n\n... (truncated) ...

github-actions · 2026-05-07T02:41:24Z

Pyrefly Diff

base → PR

--- /tmp/pyrefly_base.txt	2026-05-07 02:40:49.311303968 +0000
+++ /tmp/pyrefly_pr.txt	2026-05-07 02:40:36.113215899 +0000
@@ -28,6 +28,8 @@
    --> core/app/task_pipeline/easy_ui_based_generate_task_pipeline.py:321:29
 ERROR Argument `list[Unknown]` is not assignable to parameter `value` with type `SQLCoreOperations[dict[str, Any]] | dict[str, Any]` in function `sqlalchemy.orm.base.Mapped.__set__` [bad-argument-type]
    --> core/app/task_pipeline/easy_ui_based_generate_task_pipeline.py:388:27
+ERROR Object of class `QuestionClassifierNode` has no attribute `model_instance` [missing-attribute]
+   --> core/app/workflow/layers/llm_quota.py:119:38
 ERROR No matching overload found for function `core.model_manager.ModelInstance.invoke_llm` called with arguments: (prompt_messages=list[UserPromptMessage], model_parameters=dict[str, int], stream=Literal[False]) [no-matching-overload]
    --> core/llm_generator/llm_generator.py:113:60
 ERROR No matching overload found for function `core.model_manager.ModelInstance.invoke_llm` called with arguments: (prompt_messages=list[UserPromptMessage], model_parameters=dict[str, float | int] | dict[str, object] | dict[Unknown, Unknown], stop=list[str] | list[Unknown], stream=Literal[False]) [no-matching-overload]
@@ -48,6 +50,10 @@
    --> core/llm_generator/llm_generator.py:653:60
 ERROR Class member `OpsTraceProviderConfigMap.__getitem__` overrides parent class `UserDict` in an inconsistent manner [bad-override-param-name]
    --> core/ops/ops_trace_manager.py:206:9
+ERROR Class member `PluginModelRuntime.invoke_llm` overrides parent class `ModelRuntime` in an inconsistent manner [bad-override]
+   --> core/plugin/impl/model_runtime.py:198:9
+ERROR Cannot instantiate `PluginModelRuntime` because the following members are abstract: `invoke_llm_with_structured_output` [bad-instantiation]
+   --> core/plugin/impl/model_runtime_factory.py:116:30
 ERROR Argument `Unknown | None` is not assignable to parameter `result_object` with type `dict[str, Any]` in function `WaterCrawlProvider._structure_data` [bad-argument-type]
    --> core/rag/extractor/watercrawl/provider.py:108:37
 ERROR Object of class `BaseOxmlElement` has no attribute `body` [missing-attribute]
@@ -632,18 +638,82 @@
    --> tests/integration_tests/services/test_workflow_draft_variable_service.py:651:21
 ERROR Object of class `TestWorkflowDraftVariableServiceResetVariable` has no attribute `_test_user_id` [missing-attribute]
    --> tests/integration_tests/services/test_workflow_draft_variable_service.py:817:21
+ERROR Unexpected keyword argument `system_variables` in function `graphon.runtime.variable_pool.VariablePool.__init__` [unexpected-keyword]
+  --> tests/integration_tests/workflow/nodes/test_code.py:49:9
+ERROR Unexpected keyword argument `user_inputs` in function `graphon.runtime.variable_pool.VariablePool.__init__` [unexpected-keyword]
+  --> tests/integration_tests/workflow/nodes/test_code.py:50:9
+ERROR Unexpected keyword argument `environment_variables` in function `graphon.runtime.variable_pool.VariablePool.__init__` [unexpected-keyword]
+  --> tests/integration_tests/workflow/nodes/test_code.py:51:9
+ERROR Unexpected keyword argument `conversation_variables` in function `graphon.runtime.variable_pool.VariablePool.__init__` [unexpected-keyword]
+  --> tests/integration_tests/workflow/nodes/test_code.py:52:9
+ERROR Unexpected keyword argument `system_variables` in function `graphon.runtime.variable_pool.VariablePool.__init__` [unexpected-keyword]
+  --> tests/integration_tests/workflow/nodes/test_http.py:59:9
+ERROR Unexpected keyword argument `user_inputs` in function `graphon.runtime.variable_pool.VariablePool.__init__` [unexpected-keyword]
+  --> tests/integration_tests/workflow/nodes/test_http.py:60:9
+ERROR Unexpected keyword argument `environment_variables` in function `graphon.runtime.variable_pool.VariablePool.__init__` [unexpected-keyword]
+  --> tests/integration_tests/workflow/nodes/test_http.py:61:9
+ERROR Unexpected keyword argument `conversation_variables` in function `graphon.runtime.variable_pool.VariablePool.__init__` [unexpected-keyword]
+  --> tests/integration_tests/workflow/nodes/test_http.py:62:9
 ERROR Argument `SSRFProxy` is not assignable to parameter `http_client` with type `HttpClientProtocol | None` in function `graphon.nodes.http_request.node.HttpRequestNode.__init__` [bad-argument-type]
   --> tests/integration_tests/workflow/nodes/test_http.py:83:21
-ERROR Argument `type[ToolFileManager]` is not assignable to parameter `tool_file_manager_factory` with type `() -> ToolFileManagerProtocol` in function `graphon.nodes.http_request.node.HttpRequestNode.__init__` [bad-argument-type]
+ERROR Argument `type[ToolFileManager]` is not assignable to parameter `tool_file_manager_factory` with type `(() -> ToolFileManagerProtocol) | None` in function `graphon.nodes.http_request.node.HttpRequestNode.__init__` [bad-argument-type]
   --> tests/integration_tests/workflow/nodes/test_http.py:84:35
+ERROR Unexpected keyword argument `system_variables` in function `graphon.runtime.variable_pool.VariablePool.__init__` [unexpected-keyword]
+   --> tests/integration_tests/workflow/nodes/test_http.py:208:9
+ERROR Unexpected keyword argument `user_inputs` in function `graphon.runtime.variable_pool.VariablePool.__init__` [unexpected-keyword]
+   --> tests/integration_tests/workflow/nodes/test_http.py:209:9
+ERROR Unexpected keyword argument `environment_variables` in function `graphon.runtime.variable_pool.VariablePool.__init__` [unexpected-keyword]
+   --> tests/integration_tests/workflow/nodes/test_http.py:210:9
+ERROR Unexpected keyword argument `conversation_variables` in function `graphon.runtime.variable_pool.VariablePool.__init__` [unexpected-keyword]
+   --> tests/integration_tests/workflow/nodes/test_http.py:211:9
 ERROR Argument `SSRFProxy` is not assignable to parameter `http_client` with type `HttpClientProtocol` in function `graphon.nodes.http_request.executor.Executor.__init__` [bad-argument-type]
    --> tests/integration_tests/workflow/nodes/test_http.py:242:25
+ERROR Unexpected keyword argument `system_variables` in function `graphon.runtime.variable_pool.VariablePool.__init__` [unexpected-keyword]
+   --> tests/integration_tests/workflow/nodes/test_http.py:706:9
+ERROR Unexpected keyword argument `user_inputs` in function `graphon.runtime.variable_pool.VariablePool.__init__` [unexpected-keyword]
+   --> tests/integration_tests/workflow/nodes/test_http.py:707:9
+ERROR Unexpected keyword argument `environment_variables` in function `graphon.runtime.variable_pool.VariablePool.__init__` [unexpected-keyword]
+   --> tests/integration_tests/workflow/nodes/test_http.py:708:9
+ERROR Unexpected keyword argument `conversation_variables` in function `graphon.runtime.variable_pool.VariablePool.__init__` [unexpected-keyword]
+   --> tests/integration_tests/workflow/nodes/test_http.py:709:9
 ERROR Argument `SSRFProxy` is not assignable to parameter `http_client` with type `HttpClientProtocol | None` in function `graphon.nodes.http_request.node.HttpRequestNode.__init__` [bad-argument-type]
    --> tests/integration_tests/workflow/nodes/test_http.py:731:21
-ERROR Argument `type[ToolFileManager]` is not assignable to parameter `tool_file_manager_factory` with type `() -> ToolFileManagerProtocol` in function `graphon.nodes.http_request.node.HttpRequestNode.__init__` [bad-argument-type]
+ERROR Argument `type[ToolFileManager]` is not assignable to parameter `tool_file_manager_factory` with type `(() -> ToolFileManagerProtocol) | None` in function `graphon.nodes.http_request.node.HttpRequestNode.__init__` [bad-argument-type]
    --> tests/integration_tests/workflow/nodes/test_http.py:732:35
+ERROR Unexpected keyword argument `system_variables` in function `graphon.runtime.variable_pool.VariablePool.__init__` [unexpected-keyword]
+  --> tests/integration_tests/workflow/nodes/test_llm.py:57:9
+ERROR Unexpected keyword argument `user_inputs` in function `graphon.runtime.variable_pool.VariablePool.__init__` [unexpected-keyword]
+  --> tests/integration_tests/workflow/nodes/test_llm.py:65:9
+ERROR Unexpected keyword argument `environment_variables` in function `graphon.runtime.variable_pool.VariablePool.__init__` [unexpected-keyword]
+  --> tests/integration_tests/workflow/nodes/test_llm.py:66:9
+ERROR Unexpected keyword argument `conversation_variables` in function `graphon.runtime.variable_pool.VariablePool.__init__` [unexpected-keyword]
+  --> tests/integration_tests/workflow/nodes/test_llm.py:67:9
+ERROR Unexpected keyword argument `system_variables` in function `graphon.runtime.variable_pool.VariablePool.__init__` [unexpected-keyword]
+  --> tests/integration_tests/workflow/nodes/test_parameter_extractor.py:60:9
+ERROR Unexpected keyword argument `user_inputs` in function `graphon.runtime.variable_pool.VariablePool.__init__` [unexpected-keyword]
+  --> tests/integration_tests/workflow/nodes/test_parameter_extractor.py:63:9
+ERROR Unexpected keyword argument `environment_variables` in function `graphon.runtime.variable_pool.VariablePool.__init__` [unexpected-keyword]
+  --> tests/integration_tests/workflow/nodes/test_parameter_extractor.py:64:9
+ERROR Unexpected keyword argument `conversation_variables` in function `graphon.runtime.variable_pool.VariablePool.__init__` [unexpected-keyword]
+  --> tests/integration_tests/workflow/nodes/test_parameter_extractor.py:65:9
+ERROR Unexpected keyword argument `system_variables` in function `graphon.runtime.variable_pool.VariablePool.__init__` [unexpected-keyword]
+  --> tests/integration_tests/workflow/nodes/test_template_transform.py:70:9
+ERROR Unexpected keyword argument `user_inputs` in function `graphon.runtime.variable_pool.VariablePool.__init__` [unexpected-keyword]
+  --> tests/integration_tests/workflow/nodes/test_template_transform.py:71:9
+ERROR Unexpected keyword argument `environment_variables` in function `graphon.runtime.variable_pool.VariablePool.__init__` [unexpected-keyword]
+  --> tests/integration_tests/workflow/nodes/test_template_transform.py:72:9
+ERROR Unexpected keyword argument `conversation_variables` in function `graphon.runtime.variable_pool.VariablePool.__init__` [unexpected-keyword]
+  --> tests/integration_tests/workflow/nodes/test_template_transform.py:73:9
 ERROR Argument `_SimpleJinja2Renderer` is not assignable to parameter `jinja2_template_renderer` with type `Jinja2TemplateRenderer` in function `graphon.nodes.template_transform.template_transform_node.TemplateTransformNode.__init__` [bad-argument-type]
   --> tests/integration_tests/workflow/nodes/test_template_transform.py:94:34
+ERROR Unexpected keyword argument `system_variables` in function `graphon.runtime.variable_pool.VariablePool.__init__` [unexpected-keyword]
+  --> tests/integration_tests/workflow/nodes/test_tool.py:45:9
+ERROR Unexpected keyword argument `user_inputs` in function `graphon.runtime.variable_pool.VariablePool.__init__` [unexpected-keyword]
+  --> tests/integration_tests/workflow/nodes/test_tool.py:46:9
+ERROR Unexpected keyword argument `environment_variables` in function `graphon.runtime.variable_pool.VariablePool.__init__` [unexpected-keyword]
+  --> tests/integration_tests/workflow/nodes/test_tool.py:47:9
+ERROR Unexpected keyword argument `conversation_variables` in function `graphon.runtime.variable_pool.VariablePool.__init__` [unexpected-keyword]
+  --> tests/integration_tests/workflow/nodes/test_tool.py:48:9
 ERROR `dict[str, str | None]` is not assignable to attribute `env` with type `dict[str, str]` [bad-assignment]
    --> tests/test_containers_integration_tests/conftest.py:204:39
 ERROR Argument value `Literal[0]` violates Pydantic `ge` constraint `Literal[1]` for field `page` [bad-argument-type]
@@ -685,23 +755,23 @@
 ERROR Attribute `workflow_run_service` of class `TestPauseStatePersistenceLayerTestContainers` is a read-only descriptor with no `__set__` and cannot be set [read-only]
    --> tests/test_containers_integration_tests/core/app/layers/test_pause_state_persist_layer.py:157:9
 ERROR Argument `dict[str, str]` is not assignable to parameter `outputs` with type `dict[str, object] | None` in function `TestPauseStatePersistenceLayerTestContainers._create_graph_runtime_state` [bad-argument-type]
-   --> tests/test_containers_integration_tests/core/app/layers/test_pause_state_persist_layer.py:310:21
+   --> tests/test_containers_integration_tests/core/app/layers/test_pause_state_persist_layer.py:312:21
 ERROR Argument `dict[tuple[str, str], dict[str, str] | str]` is not assignable to parameter `variables` with type `dict[tuple[str, str], object] | None` in function `TestPauseStatePersistenceLayerTestContainers._create_graph_runtime_state` [bad-argument-type]
-   --> tests/test_containers_integration_tests/core/app/layers/test_pause_state_persist_layer.py:313:23
+   --> tests/test_containers_integration_tests/core/app/layers/test_pause_state_persist_layer.py:315:23
 ERROR Argument `dict[tuple[str, str], dict[str, str] | list[int] | str]` is not assignable to parameter `variables` with type `dict[tuple[str, str], object] | None` in function `TestPauseStatePersistenceLayerTestContainers._create_graph_runtime_state` [bad-argument-type]
-   --> tests/test_containers_integration_tests/core/app/layers/test_pause_state_persist_layer.py:377:23
+   --> tests/test_containers_integration_tests/core/app/layers/test_pause_state_persist_layer.py:379:23
 ERROR Object of class `FixtureFunctionDefinition` has no attribute `_workflow_run_repo` [missing-attribute]
-   --> tests/test_containers_integration_tests/core/app/layers/test_pause_state_persist_layer.py:389:24
+   --> tests/test_containers_integration_tests/core/app/layers/test_pause_state_persist_layer.py:391:24
 ERROR Object of class `AdvancedChatAppGenerateEntity` has no attribute `workflow_execution_id` [missing-attribute]
-   --> tests/test_containers_integration_tests/core/app/layers/test_pause_state_persist_layer.py:403:16
+   --> tests/test_containers_integration_tests/core/app/layers/test_pause_state_persist_layer.py:405:16
 ERROR Argument `dict[str, LiteralString]` is not assignable to parameter `outputs` with type `dict[str, object] | None` in function `TestPauseStatePersistenceLayerTestContainers._create_graph_runtime_state` [bad-argument-type]
-   --> tests/test_containers_integration_tests/core/app/layers/test_pause_state_persist_layer.py:444:21
+   --> tests/test_containers_integration_tests/core/app/layers/test_pause_state_persist_layer.py:446:21
 ERROR Object of class `AdvancedChatAppGenerateEntity` has no attribute `workflow_execution_id` [missing-attribute]
-   --> tests/test_containers_integration_tests/core/app/layers/test_pause_state_persist_layer.py:468:16
+   --> tests/test_containers_integration_tests/core/app/layers/test_pause_state_persist_layer.py:470:16
 ERROR Object of class `FixtureFunctionDefinition` has no attribute `_workflow_run_repo` [missing-attribute]
-   --> tests/test_containers_integration_tests/core/app/layers/test_pause_state_persist_layer.py:530:24
+   --> tests/test_containers_integration_tests/core/app/layers/test_pause_state_persist_layer.py:532:24
 ERROR Object of class `AdvancedChatAppGenerateEntity` has no attribute `workflow_execution_id` [missing-attribute]
-   --> tests/test_containers_integration_tests/core/app/layers/test_pause_state_persist_layer.py:533:16
+   --> tests/test_containers_integration_tests/core/app/layers/test_pause_state_persist_layer.py:535:16
 ERROR Object of class `NoneType` has no attribute `id` [missing-attribute]
   --> tests/test_containers_integration_tests/core/rag/retrieval/test_dataset_retrieval_integration.py:36:23
 ERROR Object of class `NoneType` has no attribute `id` [missing-attribute]
@@ -762,6 +832,12 @@
    --> tests/test_containers_integration_tests/core/rag/retrieval/test_dataset_retrieval_integration.py:574:23
 ERROR Object of class `NoneType` has no attribute `id` [missing-attribute]
    --> tests/test_containers_integration_tests/core/rag/retrieval/test_dataset_retrieval_integration.py:584:23
+ERROR Unexpected keyword argument `system_variables` in function `graphon.runtime.variable_pool.VariablePool.__init__` [unexpected-keyword]
+  --> tests/test_containers_integration_tests/core/workflow/test_human_input_resume_node_execution.py:70:9
+ERROR Unexpected keyword argument `user_inputs` in function `graphon.runtime.variable_pool.VariablePool.__init__` [unexpected-keyword]
+  --> tests/test_containers_integration_tests/core/workflow/test_human_input_resume_node_execution.py:76:9
+ERROR Unexpected keyword argument `conversation_variables` in function `graphon.runtime.variable_pool.VariablePool.__init__` [unexpected-keyword]
+  --> tests/test_containers_integration_tests/core/workflow/test_human_input_resume_node_execution.py:77:9
 ERROR Argument `Connection | Engine` is not assignable to parameter `session_factory` with type `Engine | sessionmaker[Unknown]` in function `core.repositories.sqlalchemy_workflow_execution_repository.SQLAlchemyWorkflowExecutionRepository.__init__` [bad-argument-type]
    --> tests/test_containers_integration_tests/core/workflow/test_human_input_resume_node_execution.py:264:29
 ERROR Argument `Connection | Engine` is not assignable to parameter `session_factory` with type `Engine | sessionmaker[Unknown]` in function `core.repositories.sqlalchemy_workflow_node_execution_repository.SQLAlchemyWorkflowNodeExecutionRepository.__init__` [bad-argument-type]
@@ -2700,6 +2776,8 @@
   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline_core.py:99:23
 ERROR Argument `(**kwargs: Unknown) -> None` is not assignable to parameter `draft_var_saver_factory` with type `DraftVariableSaverFactory` in function `core.app.apps.advanced_chat.generate_task_pipeline.AdvancedChatAppGenerateTaskPipeline.__init__` [bad-argument-type]
    --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline_core.py:105:33
+ERROR Unexpected keyword argument `system_variables` in function `graphon.runtime.variable_pool.VariablePool.__init__` [unexpected-keyword]
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline_core.py:135:40
 ERROR `SimpleNamespace` is not assignable to attribute `_message_cycle_manager` with type `MessageCycleManager` [bad-assignment]
    --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline_core.py:183:43
 ERROR Argument `SimpleNamespace` is not assignable to parameter `event` with type `QueueTextChunkEvent` in function `core.app.apps.advanced_chat.generate_task_pipeline.AdvancedChatAppGenerateTaskPipeline._handle_text_chunk_event` [bad-argument-type]
@@ -2735,43 +2813,43 @@
 ERROR `(**kwargs: Unknown) -> Literal['loop_done']` is not assignable to attribute `workflow_loop_completed_to_stream_response` with type `(self: WorkflowResponseConverter, *, task_id: str, workflow_execution_id: str, event: QueueLoopCompletedEvent) -> LoopNodeCompletedStreamResponse` [bad-assignment]
    --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline_core.py:313:92
 ERROR `(**kwargs: Unknown) -> Literal['finish']` is not assignable to attribute `workflow_finish_to_stream_response` with type `(self: WorkflowResponseConverter, *, task_id: str, workflow_id: str, status: WorkflowExecutionStatus, graph_runtime_state: GraphRuntimeState, error: str | None = None, exceptions_count: int = 0) -> WorkflowFinishStreamResponse` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline_core.py:378:84
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline_core.py:380:84
 ERROR `(**kwargs: Unknown) -> list[str]` is not assignable to attribute `workflow_pause_to_stream_response` with type `(self: WorkflowResponseConverter, *, event: QueueWorkflowPausedEvent, task_id: str, graph_runtime_state: GraphRuntimeState) -> list[StreamResponse]` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline_core.py:379:83
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline_core.py:381:83
 ERROR `(err: Unknown) -> Unknown` is not assignable to attribute `error_to_stream_response` with type `(self: BasedGenerateTaskPipeline, e: Exception) -> ErrorStreamResponse` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline_core.py:384:65
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline_core.py:386:65
 ERROR `(**kwargs: Unknown) -> SimpleNamespace` is not assignable to attribute `_get_message` with type `(self: AdvancedChatAppGenerateTaskPipeline, *, session: Session) -> Message` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline_core.py:385:33
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline_core.py:387:33
 ERROR `(**kwargs: Unknown) -> Literal['node_finish']` is not assignable to attribute `workflow_node_finish_to_stream_response` with type `(self: WorkflowResponseConverter, *, event: QueueNodeExceptionEvent | QueueNodeFailedEvent | QueueNodeSucceededEvent, task_id: str) -> NodeFinishStreamResponse | None` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline_core.py:416:89
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline_core.py:418:89
 ERROR `SimpleNamespace` is not assignable to attribute `_message_cycle_manager` with type `MessageCycleManager` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline_core.py:451:43
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline_core.py:453:43
 ERROR Argument `SimpleNamespace` is not assignable to parameter `event` with type `QueueTextChunkEvent` in function `core.app.apps.advanced_chat.generate_task_pipeline.AdvancedChatAppGenerateTaskPipeline._handle_text_chunk_event` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline_core.py:457:47
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline_core.py:459:47
 ERROR Argument `TestAdvancedChatGenerateTaskPipeline.test_handle_text_chunk_event_tracks_streaming_metrics._Publisher` is not assignable to parameter `tts_publisher` with type `AppGeneratorTTSPublisher | None` in function `core.app.apps.advanced_chat.generate_task_pipeline.AdvancedChatAppGenerateTaskPipeline._handle_text_chunk_event` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline_core.py:457:68
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline_core.py:459:68
 ERROR Argument `SimpleNamespace` is not assignable to parameter `queue_message` with type `MessageQueueMessage | WorkflowQueueMessage | None` in function `core.app.apps.advanced_chat.generate_task_pipeline.AdvancedChatAppGenerateTaskPipeline._handle_text_chunk_event` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline_core.py:457:96
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline_core.py:459:96
 ERROR `TestAdvancedChatGenerateTaskPipeline.test_handle_output_moderation_chunk_appends_token._Moderation` is not assignable to attribute `output_moderation_handler` with type `OutputModeration | None` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline_core.py:478:66
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline_core.py:480:66
 ERROR `(**kwargs: Unknown) -> Literal['replace']` is not assignable to attribute `message_replace_to_stream_response` with type `(self: MessageCycleManager, answer: str, reason: str = '') -> MessageReplaceStreamResponse` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline_core.py:507:78
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline_core.py:509:78
 ERROR `(**kwargs: Unknown) -> Literal['filled']` is not assignable to attribute `human_input_form_filled_to_stream_response` with type `(self: WorkflowResponseConverter, *, event: QueueHumanInputFormFilledEvent, task_id: str) -> HumanInputFormFilledResponse` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline_core.py:520:92
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline_core.py:522:92
 ERROR `(**kwargs: Unknown) -> Literal['timeout']` is not assignable to attribute `human_input_form_timeout_to_stream_response` with type `(self: WorkflowResponseConverter, *, event: QueueHumanInputFormTimeoutEvent, task_id: str) -> HumanInputFormTimeoutResponse` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline_core.py:521:93
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline_core.py:523:93
 ERROR Argument `TestAdvancedChatGenerateTaskPipeline.test_save_message_strips_markdown_and_sets_usage._Session` is not assignable to parameter `session` with type `Session` in function `core.app.apps.advanced_chat.generate_task_pipeline.AdvancedChatAppGenerateTaskPipeline._save_message` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline_core.py:590:40
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline_core.py:594:40
 ERROR `() -> Literal['end']` is not assignable to attribute `_message_end_to_stream_response` with type `(self: AdvancedChatAppGenerateTaskPipeline) -> MessageEndStreamResponse` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline_core.py:598:52
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline_core.py:602:52
 ERROR `(answer: Unknown) -> Literal['safe']` is not assignable to attribute `handle_output_moderation_when_task_finished` with type `(self: BasedGenerateTaskPipeline, completion: str) -> str | None` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline_core.py:623:84
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline_core.py:629:84
 ERROR `(**kwargs: Unknown) -> Literal['replace']` is not assignable to attribute `message_replace_to_stream_response` with type `(self: MessageCycleManager, answer: str, reason: str = '') -> MessageReplaceStreamResponse` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline_core.py:624:78
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline_core.py:630:78
 ERROR `() -> Literal['end']` is not assignable to attribute `_message_end_to_stream_response` with type `(self: AdvancedChatAppGenerateTaskPipeline) -> MessageEndStreamResponse` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline_core.py:625:52
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline_core.py:631:52
 ERROR `(**kwargs: Unknown) -> Literal['failed']` is not assignable to attribute `workflow_node_finish_to_stream_response` with type `(self: WorkflowResponseConverter, *, event: QueueNodeExceptionEvent | QueueNodeFailedEvent | QueueNodeSucceededEvent, task_id: str) -> NodeFinishStreamResponse | None` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline_core.py:647:89
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline_core.py:653:89
 ERROR Argument `dict[str, dict[str, str]]` is not assignable to parameter `override_config_dict` with type `AppModelConfigDict | None` in function `core.app.apps.agent_chat.app_config_manager.AgentChatAppConfigManager.get_app_config` [bad-argument-type]
   --> tests/unit_tests/core/app/apps/agent_chat/test_agent_chat_app_config_manager.py:40:34
 ERROR Cannot index into `str` [bad-index]
@@ -3006,8 +3084,6 @@
    --> tests/unit_tests/core/app/apps/test_message_based_app_generator.py:175:9
 ERROR Object of class `ModuleType` has no attribute `TraceQueueManager` [missing-attribute]
   --> tests/unit_tests/core/app/apps/test_pause_resume.py:42:5
-ERROR Class member `_StubToolNode._run` overrides parent class `Node` in an inconsistent manner [bad-override]
-  --> tests/unit_tests/core/app/apps/test_pause_resume.py:92:9
 ERROR Argument `SimpleNamespace` is not assignable to parameter `app_model` with type `App` in function `core.app.apps.workflow.app_generator.WorkflowAppGenerator.resume` [bad-argument-type]
    --> tests/unit_tests/core/app/apps/test_pause_resume.py:226:19
 ERROR Argument `SimpleNamespace` is not assignable to parameter `workflow` with type `Workflow` in function `core.app.apps.workflow.app_generator.WorkflowAppGenerator.resume` [bad-argument-type]
@@ -3054,32 +3130,34 @@
    --> tests/unit_tests/core/app/apps/test_streaming_utils.py:127:12
 ERROR Cannot index into `str` [bad-index]
    --> tests/unit_tests/core/app/apps/test_streaming_utils.py:128:12
+ERROR No matching overload found for function `core.app.apps.workflow.app_generator.WorkflowAppGenerator.generate` called with arguments: (app_model=SimpleNamespace, workflow=SimpleNamespace, user=SimpleNamespace, args=dict[str, list[@_] | str | dict[str, str]], invoke_from=Literal['service-api'], streaming=Literal[False], call_depth=Literal[0]) [no-matching-overload]
+  --> tests/unit_tests/core/app/apps/test_workflow_app_generator.py:69:32
 ERROR Argument `SimpleNamespace` is not assignable to parameter `application_generate_entity` with type `WorkflowAppGenerateEntity` in function `core.app.apps.workflow.app_generator.WorkflowAppGenerator.resume` [bad-argument-type]
-  --> tests/unit_tests/core/app/apps/test_workflow_app_generator.py:37:37
+   --> tests/unit_tests/core/app/apps/test_workflow_app_generator.py:109:37
 ERROR Argument `tuple[Literal['layer']]` is not assignable to parameter `graph_engine_layers` with type `Sequence[GraphEngineLayer]` in function `core.app.apps.workflow.app_generator.WorkflowAppGenerator.resume` [bad-argument-type]
-  --> tests/unit_tests/core/app/apps/test_workflow_app_generator.py:41:29
+   --> tests/unit_tests/core/app/apps/test_workflow_app_generator.py:113:29
 ERROR Argument `SimpleNamespace` is not assignable to parameter `app_model` with type `App` in function `core.app.apps.workflow.app_generator.WorkflowAppGenerator._generate` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/test_workflow_app_generator.py:109:19
+   --> tests/unit_tests/core/app/apps/test_workflow_app_generator.py:181:19
 ERROR Argument `SimpleNamespace` is not assignable to parameter `application_generate_entity` with type `WorkflowAppGenerateEntity` in function `core.app.apps.workflow.app_generator.WorkflowAppGenerator._generate` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/test_workflow_app_generator.py:112:37
+   --> tests/unit_tests/core/app/apps/test_workflow_app_generator.py:184:37
 ERROR Argument `Literal['service-api']` is not assignable to parameter `invoke_from` with type `InvokeFrom` in function `core.app.apps.workflow.app_generator.WorkflowAppGenerator._generate` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/test_workflow_app_generator.py:113:21
+   --> tests/unit_tests/core/app/apps/test_workflow_app_generator.py:185:21
 ERROR Argument `tuple[Literal['base-layer']]` is not assignable to parameter `graph_engine_layers` with type `Sequence[GraphEngineLayer]` in function `core.app.apps.workflow.app_generator.WorkflowAppGenerator._generate` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/test_workflow_app_generator.py:117:29
+   --> tests/unit_tests/core/app/apps/test_workflow_app_generator.py:189:29
 ERROR Argument `SimpleNamespace` is not assignable to parameter `pause_state_config` with type `PauseStateLayerConfig | None` in function `core.app.apps.workflow.app_generator.WorkflowAppGenerator._generate` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/test_workflow_app_generator.py:119:28
+   --> tests/unit_tests/core/app/apps/test_workflow_app_generator.py:191:28
 ERROR Cannot index into `object` [bad-index]
-   --> tests/unit_tests/core/app/apps/test_workflow_app_generator.py:123:12
+   --> tests/unit_tests/core/app/apps/test_workflow_app_generator.py:195:12
 ERROR Cannot index into `object` [bad-index]
-   --> tests/unit_tests/core/app/apps/test_workflow_app_generator.py:124:12
+   --> tests/unit_tests/core/app/apps/test_workflow_app_generator.py:196:12
 ERROR Argument `SimpleNamespace` is not assignable to parameter `app_model` with type `App` in function `core.app.apps.workflow.app_generator.WorkflowAppGenerator.resume` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/test_workflow_app_generator.py:200:19
+   --> tests/unit_tests/core/app/apps/test_workflow_app_generator.py:272:19
 ERROR Argument `SimpleNamespace` is not assignable to parameter `workflow` with type `Workflow` in function `core.app.apps.workflow.app_generator.WorkflowAppGenerator.resume` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/test_workflow_app_generator.py:201:18
+   --> tests/unit_tests/core/app/apps/test_workflow_app_generator.py:273:18
 ERROR Argument `SimpleNamespace` is not assignable to parameter `application_generate_entity` with type `WorkflowAppGenerateEntity` in function `core.app.apps.workflow.app_generator.WorkflowAppGenerator.resume` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/test_workflow_app_generator.py:203:37
+   --> tests/unit_tests/core/app/apps/test_workflow_app_generator.py:275:37
 ERROR Argument `SimpleNamespace` is not assignable to parameter `pause_state_config` with type `PauseStateLayerConfig | None` in function `core.app.apps.workflow.app_generator.WorkflowAppGenerator.resume` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/test_workflow_app_generator.py:207:28
+   --> tests/unit_tests/core/app/apps/test_workflow_app_generator.py:279:28
 ERROR Argument `SimpleNamespace` is not assignable to parameter `queue_manager` with type `AppQueueManager` in function `core.app.apps.workflow_app_runner.WorkflowBasedAppRunner.__init__` [bad-argument-type]
   --> tests/unit_tests/core/app/apps/test_workflow_app_runner_core.py:47:55
 ERROR Argument `SimpleNamespace` is not assignable to parameter `queue_manager` with type `AppQueueManager` in function `core.app.apps.workflow_app_runner.WorkflowBasedAppRunner.__init__` [bad-argument-type]
@@ -3134,6 +3212,14 @@
   --> tests/unit_tests/core/app/apps/test_workflow_app_runner_notifications.py:36:51
 ERROR Argument `_DummyWorkflowEntry` is not assignable to parameter `workflow_entry` with type `WorkflowEntry` in function `core.app.apps.workflow_app_runner.WorkflowBasedAppRunner._handle_event` [bad-argument-type]
   --> tests/unit_tests/core/app/apps/test_workflow_app_runner_notifications.py:52:26
+ERROR Unexpected keyword argument `system_variables` in function `graphon.runtime.variable_pool.VariablePool.__init__` [unexpected-keyword]
+  --> tests/unit_tests/core/app/apps/test_workflow_app_runner_single_node.py:20:9
+ERROR Unexpected keyword argument `user_inputs` in function `graphon.runtime.variable_pool.VariablePool.__init__` [unexpected-keyword]
+  --> tests/unit_tests/core/app/apps/test_workflow_app_runner_single_node.py:21:9
+ERROR Unexpected keyword argument `environment_variables` in function `graphon.runtime.variable_pool.VariablePool.__init__` [unexpected-keyword]
+  --> tests/unit_tests/core/app/apps/test_workflow_app_runner_single_node.py:22:9
+ERROR Unexpected keyword argument `conversation_variables` in function `graphon.runtime.variable_pool.VariablePool.__init__` [unexpected-keyword]
+  --> tests/unit_tests/core/app/apps/test_workflow_app_runner_single_node.py:23:9
 ERROR Argument `SimpleNamespace` is not assignable to parameter `workflow_entry` with type `WorkflowEntry` in function `core.app.apps.workflow_app_runner.WorkflowBasedAppRunner._handle_event` [bad-argument-type]
   --> tests/unit_tests/core/app/apps/test_workflow_pause_events.py:84:26
 ERROR Argument `SimpleNamespace` is not assignable to parameter `application_generate_entity` with type `AdvancedChatAppGenerateEntity | WorkflowAppGenerateEntity` in function `core.app.apps.common.workflow_response_converter.WorkflowResponseConverter.__init__` [bad-argument-type]
@@ -3206,6 +3292,8 @@
   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:86:14
 ERROR Argument `(**kwargs: Unknown) -> None` is not assignable to parameter `draft_var_saver_factory` with type `DraftVariableSaverFactory` in function `core.app.apps.workflow.generate_task_pipeline.WorkflowAppGenerateTaskPipeline.__init__` [bad-argument-type]
   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:88:33
+ERROR Unexpected keyword argument `system_variables` in function `graphon.runtime.variable_pool.VariablePool.__init__` [unexpected-keyword]
+  --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:98:40
 ERROR Argument `SimpleNamespace` is not assignable to parameter `publisher` with type `AppGeneratorTTSPublisher | None` in function `core.app.apps.workflow.generate_task_pipeline.WorkflowAppGenerateTaskPipeline._listen_audio_msg` [bad-argument-type]
    --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:169:57
 ERROR `(err: Unknown) -> Unknown` is not assignable to attribute `error_to_stream_response` with type `(self: BasedGenerateTaskPipeline, e: Exception) -> ErrorStreamResponse` [bad-assignment]
@@ -3227,101 +3315,101 @@
 ERROR `(**kwargs: Unknown) -> Literal['done']` is not assignable to attribute `workflow_node_finish_to_stream_response` with type `(self: WorkflowResponseConverter, *, event: QueueNodeExceptionEvent | QueueNodeFailedEvent | QueueNodeSucceededEvent, task_id: str) -> NodeFinishStreamResponse | None` [bad-assignment]
    --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:267:89
 ERROR `(**kwargs: Unknown) -> Literal['finish']` is not assignable to attribute `workflow_finish_to_stream_response` with type `(self: WorkflowResponseConverter, *, task_id: str, workflow_id: str, status: WorkflowExecutionStatus, graph_runtime_state: GraphRuntimeState, error: str | None = None, exceptions_count: int = 0) -> WorkflowFinishStreamResponse` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:289:84
+   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:291:84
 ERROR Argument `TestWorkflowGenerateTaskPipeline.test_save_workflow_app_log_created_from._Session` is not assignable to parameter `session` with type `Session` in function `core.app.apps.workflow.generate_task_pipeline.WorkflowAppGenerateTaskPipeline._save_workflow_app_log` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:309:49
+   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:311:49
 ERROR `(**kwargs: Unknown) -> Literal['iter']` is not assignable to attribute `workflow_iteration_start_to_stream_response` with type `(self: WorkflowResponseConverter, *, task_id: str, workflow_execution_id: str, event: QueueIterationStartEvent) -> IterationNodeStartStreamResponse` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:316:93
+   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:318:93
 ERROR `(**kwargs: Unknown) -> Literal['next']` is not assignable to attribute `workflow_iteration_next_to_stream_response` with type `(self: WorkflowResponseConverter, *, task_id: str, workflow_execution_id: str, event: QueueIterationNextEvent) -> IterationNodeNextStreamResponse` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:317:92
+   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:319:92
 ERROR `(**kwargs: Unknown) -> Literal['done']` is not assignable to attribute `workflow_iteration_completed_to_stream_response` with type `(self: WorkflowResponseConverter, *, task_id: str, workflow_execution_id: str, event: QueueIterationCompletedEvent) -> IterationNodeCompletedStreamResponse` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:318:97
+   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:320:97
 ERROR `(**kwargs: Unknown) -> Literal['loop']` is not assignable to attribute `workflow_loop_start_to_stream_response` with type `(self: WorkflowResponseConverter, *, task_id: str, workflow_execution_id: str, event: QueueLoopStartEvent) -> LoopNodeStartStreamResponse` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:319:88
+   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:321:88
 ERROR `(**kwargs: Unknown) -> Literal['loop_next']` is not assignable to attribute `workflow_loop_next_to_stream_response` with type `(self: WorkflowResponseConverter, *, task_id: str, workflow_execution_id: str, event: QueueLoopNextEvent) -> LoopNodeNextStreamResponse` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:320:87
+   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:322:87
 ERROR `(**kwargs: Unknown) -> Literal['loop_done']` is not assignable to attribute `workflow_loop_completed_to_stream_response` with type `(self: WorkflowResponseConverter, *, task_id: str, workflow_execution_id: str, event: QueueLoopCompletedEvent) -> LoopNodeCompletedStreamResponse` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:321:92
+   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:323:92
 ERROR `(**kwargs: Unknown) -> Literal['filled']` is not assignable to attribute `human_input_form_filled_to_stream_response` with type `(self: WorkflowResponseConverter, *, event: QueueHumanInputFormFilledEvent, task_id: str) -> HumanInputFormFilledResponse` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:322:92
+   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:324:92
 ERROR `(**kwargs: Unknown) -> Literal['timeout']` is not assignable to attribute `human_input_form_timeout_to_stream_response` with type `(self: WorkflowResponseConverter, *, event: QueueHumanInputFormTimeoutEvent, task_id: str) -> HumanInputFormTimeoutResponse` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:323:93
+   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:325:93
 ERROR `(**kwargs: Unknown) -> Literal['log']` is not assignable to attribute `handle_agent_log` with type `(self: WorkflowResponseConverter, task_id: str, event: QueueAgentLogEvent) -> AgentLogStreamResponse` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:324:66
+   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:326:66
 ERROR `(**kwargs: Unknown) -> Iterator[Any]` is not assignable to attribute `_process_stream_response` with type `(self: WorkflowAppGenerateTaskPipeline, tts_publisher: AppGeneratorTTSPublisher | None = None, trace_manager: TraceQueueManager | None = None) -> Generator[StreamResponse]` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:416:45
+   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:418:45
 ERROR Argument `SimpleNamespace` is not assignable to parameter `workflow` with type `Workflow` in function `core.app.apps.workflow.generate_task_pipeline.WorkflowAppGenerateTaskPipeline.__init__` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:472:22
+   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:474:22
 ERROR Argument `SimpleNamespace` is not assignable to parameter `queue_manager` with type `AppQueueManager` in function `core.app.apps.workflow.generate_task_pipeline.WorkflowAppGenerateTaskPipeline.__init__` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:473:27
+   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:475:27
 ERROR Argument `(**kwargs: Unknown) -> None` is not assignable to parameter `draft_var_saver_factory` with type `DraftVariableSaverFactory` in function `core.app.apps.workflow.generate_task_pipeline.WorkflowAppGenerateTaskPipeline.__init__` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:476:37
+   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:478:37
 ERROR `(**kwargs: Unknown) -> Iterator[Any]` is not assignable to attribute `_wrapper_process_stream_response` with type `(self: WorkflowAppGenerateTaskPipeline, trace_manager: TraceQueueManager | None = None) -> Generator[StreamResponse]` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:485:53
+   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:487:53
 ERROR Object of class `tuple` has no attribute `workflow_run_id` [missing-attribute]
-   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:489:16
+   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:491:16
 ERROR `(**kwargs: Unknown) -> Iterator[Any]` is not assignable to attribute `_wrapper_process_stream_response` with type `(self: WorkflowAppGenerateTaskPipeline, trace_manager: TraceQueueManager | None = None) -> Generator[StreamResponse]` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:492:53
+   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:494:53
 ERROR Object of class `Generator` has no attribute `workflow_run_id` [missing-attribute]
-   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:514:16
+   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:516:16
 ERROR `(**kwargs: Unknown) -> Iterator[Any]` is not assignable to attribute `_process_stream_response` with type `(self: WorkflowAppGenerateTaskPipeline, tts_publisher: AppGeneratorTTSPublisher | None = None, trace_manager: TraceQueueManager | None = None) -> Generator[StreamResponse]` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:558:45
+   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:560:45
 ERROR `(**kwargs: Unknown) -> Iterator[Any]` is not assignable to attribute `_process_stream_response` with type `(self: WorkflowAppGenerateTaskPipeline, tts_publisher: AppGeneratorTTSPublisher | None = None, trace_manager: TraceQueueManager | None = None) -> Generator[StreamResponse]` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:568:45
+   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:570:45
 ERROR `(**kwargs: Unknown) -> Iterator[Any]` is not assignable to attribute `_process_stream_response` with type `(self: WorkflowAppGenerateTaskPipeline, tts_publisher: AppGeneratorTTSPublisher | None = None, trace_manager: TraceQueueManager | None = None) -> Generator[StreamResponse]` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:605:45
+   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:607:45
 ERROR `(**kwargs: Unknown) -> Literal['retry']` is not assignable to attribute `workflow_node_retry_to_stream_response` with type `(self: WorkflowResponseConverter, *, event: QueueNodeRetryEvent, task_id: str) -> NodeRetryStreamResponse | None` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:695:88
+   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:697:88
 ERROR `(**kwargs: Unknown) -> Literal['started']` is not assignable to attribute `workflow_node_start_to_stream_response` with type `(self: WorkflowResponseConverter, *, event: QueueNodeStartedEvent, task_id: str) -> NodeStartStreamResponse | None` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:700:88
+   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:702:88
 ERROR `(**kwargs: Unknown) -> Literal['failed']` is not assignable to attribute `workflow_node_finish_to_stream_response` with type `(self: WorkflowResponseConverter, *, event: QueueNodeExceptionEvent | QueueNodeFailedEvent | QueueNodeSucceededEvent, task_id: str) -> NodeFinishStreamResponse | None` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:706:89
+   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:708:89
 ERROR `(**kwargs: Unknown) -> Literal['finish']` is not assignable to attribute `workflow_finish_to_stream_response` with type `(self: WorkflowResponseConverter, *, task_id: str, workflow_id: str, status: WorkflowExecutionStatus, graph_runtime_state: GraphRuntimeState, error: str | None = None, exceptions_count: int = 0) -> WorkflowFinishStreamResponse` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:732:84
+   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:736:84
 ERROR `(**kwargs: Unknown) -> list[str]` is not assignable to attribute `workflow_pause_to_stream_response` with type `(self: WorkflowResponseConverter, *, event: QueueWorkflowPausedEvent, task_id: str, graph_runtime_state: GraphRuntimeState) -> list[StreamResponse]` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:740:83
+   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:744:83
 ERROR `(event: QueuePingEvent, **kwargs: Unknown) -> Iterator[PingStreamResponse] | Iterator[Any]` is not assignable to attribute `_handle_ping_event` with type `(self: WorkflowAppGenerateTaskPipeline, event: QueuePingEvent, **kwargs: Unknown) -> Generator[PingStreamResponse]` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:759:39
+   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:765:39
 ERROR `(event: QueueStopEvent | QueueWorkflowFailedEvent, **kwargs: Unknown) -> Iterator[StreamResponse] | Iterator[Any]` is not assignable to attribute `_handle_workflow_failed_and_stop_events` with type `(self: WorkflowAppGenerateTaskPipeline, event: QueueStopEvent | QueueWorkflowFailedEvent, *, trace_manager: TraceQueueManager | None = None, **kwargs: Unknown) -> Generator[StreamResponse]` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:762:60
+   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:768:60
 ERROR Argument `SimpleNamespace` is not assignable to parameter `event` with type `AppQueueEvent` in function `core.app.apps.workflow.generate_task_pipeline.WorkflowAppGenerateTaskPipeline._dispatch_event` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:767:46
+   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:773:46
 ERROR `() -> Iterator[MessageQueueMessage | WorkflowQueueMessage] | Iterator[Any]` is not assignable to attribute `listen` with type `(self: AppQueueManager) -> Generator[MessageQueueMessage | WorkflowQueueMessage, Unknown]` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:775:61
+   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:783:61
 ERROR `(event: QueueWorkflowStartedEvent, **kwargs: Unknown) -> Iterator[StreamResponse] | Iterator[Any]` is not assignable to attribute `_handle_workflow_started_event` with type `(self: WorkflowAppGenerateTaskPipeline, event: QueueWorkflowStartedEvent, **kwargs: Unknown) -> Generator[StreamResponse]` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:783:51
+   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:791:51
 ERROR `(event: QueueTextChunkEvent, **kwargs: Unknown) -> Iterator[StreamResponse] | Iterator[Any]` is not assignable to attribute `_handle_text_chunk_event` with type `(self: WorkflowAppGenerateTaskPipeline, event: QueueTextChunkEvent, *, tts_publisher: AppGeneratorTTSPublisher | None = None, queue_message: MessageQueueMessage | WorkflowQueueMessage | None = None, **kwargs: Unknown) -> Generator[StreamResponse]` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:784:45
+   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:792:45
 ERROR `(event: AppQueueEvent, **kwargs: Unknown) -> Iterator[StreamResponse] | Iterator[Any]` is not assignable to attribute `_dispatch_event` with type `(self: WorkflowAppGenerateTaskPipeline, event: AppQueueEvent, *, tts_publisher: AppGeneratorTTSPublisher | None = None, trace_manager: TraceQueueManager | None = None, queue_message: MessageQueueMessage | WorkflowQueueMessage | None = None) -> Generator[StreamResponse]` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:785:36
+   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:793:36
 ERROR `(event: QueueErrorEvent, **kwargs: Unknown) -> Iterator[ErrorStreamResponse] | Iterator[Any]` is not assignable to attribute `_handle_error_event` with type `(self: WorkflowAppGenerateTaskPipeline, event: QueueErrorEvent, **kwargs: Unknown) -> Generator[ErrorStreamResponse]` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:786:40
+   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:794:40
 ERROR Argument `TestWorkflowGenerateTaskPipeline.test_process_stream_response_main_match_paths_and_cleanup._Publisher` is not assignable to parameter `tts_publisher` with type `AppGeneratorTTSPublisher | None` in function `core.app.apps.workflow.generate_task_pipeline.WorkflowAppGenerateTaskPipeline._process_stream_response` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:793:74
+   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:801:74
 ERROR `() -> Iterator[MessageQueueMessage | WorkflowQueueMessage] | Iterator[Any]` is not assignable to attribute `listen` with type `(self: AppQueueManager) -> Generator[MessageQueueMessage | WorkflowQueueMessage, Unknown]` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:800:61
+   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:808:61
 ERROR `(event: QueueStopEvent | QueueWorkflowFailedEvent, **kwargs: Unknown) -> Iterator[StreamResponse] | Iterator[Any]` is not assignable to attribute `_handle_workflow_failed_and_stop_events` with type `(self: WorkflowAppGenerateTaskPipeline, event: QueueStopEvent | QueueWorkflowFailedEvent, *, trace_manager: TraceQueueManager | None = None, **kwargs: Unknown) -> Generator[StreamResponse]` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:803:60
+   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:811:60
 ERROR `() -> Iterator[MessageQueueMessage | WorkflowQueueMessage] | Iterator[Any]` is not assignable to attribute `listen` with type `(self: AppQueueManager) -> Generator[MessageQueueMessage | WorkflowQueueMessage, Unknown]` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:806:61
+   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:814:61
 ERROR `(event: QueueWorkflowPausedEvent, **kwargs: Unknown) -> Iterator[StreamResponse] | Iterator[Any]` is not assignable to attribute `_handle_workflow_paused_event` with type `(self: WorkflowAppGenerateTaskPipeline, event: QueueWorkflowPausedEvent, **kwargs: Unknown) -> Generator[StreamResponse]` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:809:50
+   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:817:50
 ERROR `() -> Iterator[MessageQueueMessage | WorkflowQueueMessage] | Iterator[Any]` is not assignable to attribute `listen` with type `(self: AppQueueManager) -> Generator[MessageQueueMessage | WorkflowQueueMessage, Unknown]` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:812:61
+   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:820:61
 ERROR `(event: QueueStopEvent | QueueWorkflowFailedEvent, **kwargs: Unknown) -> Iterator[StreamResponse] | Iterator[Any]` is not assignable to attribute `_handle_workflow_failed_and_stop_events` with type `(self: WorkflowAppGenerateTaskPipeline, event: QueueStopEvent | QueueWorkflowFailedEvent, *, trace_manager: TraceQueueManager | None = None, **kwargs: Unknown) -> Generator[StreamResponse]` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:815:60
+   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:823:60
 ERROR Argument `TestWorkflowGenerateTaskPipeline.test_save_workflow_app_log_covers_invoke_from_variants._Session` is not assignable to parameter `session` with type `Session` in function `core.app.apps.workflow.generate_task_pipeline.WorkflowAppGenerateTaskPipeline._save_workflow_app_log` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:828:49
+   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:836:49
 ERROR Object of class `object` has no attribute `created_from` [missing-attribute]
-   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:829:16
+   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:837:16
 ERROR Argument `TestWorkflowGenerateTaskPipeline.test_save_workflow_app_log_covers_invoke_from_variants._Session` is not assignable to parameter `session` with type `Session` in function `core.app.apps.workflow.generate_task_pipeline.WorkflowAppGenerateTaskPipeline._save_workflow_app_log` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:832:49
+   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:840:49
 ERROR Argument `TestWorkflowGenerateTaskPipeline.test_save_workflow_app_log_covers_invoke_from_variants._Session` is not assignable to parameter `session` with type `Session` in function `core.app.apps.workflow.generate_task_pipeline.WorkflowAppGenerateTaskPipeline._save_workflow_app_log` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:837:49
+   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:845:49
 ERROR Argument `TestWorkflowGenerateTaskPipeline.test_save_workflow_app_log_covers_invoke_from_variants._Session` is not assignable to parameter `session` with type `Session` in function `core.app.apps.workflow.generate_task_pipeline.WorkflowAppGenerateTaskPipeline._save_workflow_app_log` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:841:49
+   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:849:49
 ERROR `(**kwargs: Unknown) -> TestWorkflowGenerateTaskPipeline.test_save_output_for_event_writes_draft_variables._Saver` is not assignable to attribute `_draft_var_saver_factory` with type `DraftVariableSaverFactory` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:857:45
+   --> tests/unit_tests/core/app/apps/workflow/test_generate_task_pipeline_core.py:865:45
 ERROR `Literal['unknown']` is not assignable to attribute `stopped_by` with type `QueueStopEvent.StopBy` [bad-assignment]
   --> tests/unit_tests/core/app/entities/test_queue_entities.py:11:28
 ERROR Cannot index into `bool` [bad-index]
@@ -3419,11 +3507,11 @@
 ERROR Argument `object` is not assignable to parameter `event` with type `GraphEngineEvent` in function `core.app.layers.suspend_layer.SuspendLayer.on_event` [bad-argument-type]
   --> tests/unit_tests/core/app/layers/test_suspend_layer.py:18:24
 ERROR Argument `SimpleNamespace` is not assignable to parameter `graph_runtime_state` with type `ReadOnlyGraphRuntimeState` in function `graphon.graph_engine.layers.base.GraphEngineLayer.initialize` [bad-argument-type]
-  --> tests/unit_tests/core/app/layers/test_trigger_post_layer.py:47:30
+  --> tests/unit_tests/core/app/layers/test_trigger_post_layer.py:49:30
 ERROR Argument `SimpleNamespace` is not assignable to parameter `graph_runtime_state` with type `ReadOnlyGraphRuntimeState` in function `graphon.graph_engine.layers.base.GraphEngineLayer.initialize` [bad-argument-type]
-  --> tests/unit_tests/core/app/layers/test_trigger_post_layer.py:84:30
+  --> tests/unit_tests/core/app/layers/test_trigger_post_layer.py:88:30
 ERROR Argument `SimpleNamespace` is not assignable to parameter `graph_runtime_state` with type `ReadOnlyGraphRuntimeState` in function `graphon.graph_engine.layers.base.GraphEngineLayer.initialize` [bad-argument-type]
-   --> tests/unit_tests/core/app/layers/test_trigger_post_layer.py:104:30
+   --> tests/unit_tests/core/app/layers/test_trigger_post_layer.py:110:30
 ERROR Argument `SimpleNamespace` is not assignable to parameter `application_generate_entity` with type `AppGenerateEntity` in function `core.app.task_pipeline.based_generate_task_pipeline.BasedGenerateTaskPipeline.__init__` [bad-argument-type]
   --> tests/unit_tests/core/app/task_pipeline/test_based_generate_task_pipeline.py:23:41
 ERROR Argument `SimpleNamespace` is not assignable to parameter `queue_manager` with type `AppQueueManager` in function `core.app.task_pipeline.easy_ui_based_generate_task_pipeline.EasyUIBasedGenerateTaskPipeline.__init__` [bad-argument-type]
@@ -3765,45 +3853,45 @@
 ERROR Argument `SimpleNamespace` is not assignable to parameter `node` with type `Node[Unknown]` in function `core.app.workflow.layers.observability.ObservabilityLayer.on_node_run_end` [bad-argument-type]
    --> tests/unit_tests/core/app/workflow/test_observability_layer_extra.py:208:31
 ERROR Argument `_RepoRecorder` is not assignable to parameter `workflow_execution_repository` with type `WorkflowExecutionRepository` in function `core.app.workflow.layers.persistence.WorkflowPersistenceLayer.__init__` [bad-argument-type]
-  --> tests/unit_tests/core/app/workflow/test_persistence_layer.py:93:39
+  --> tests/unit_tests/core/app/workflow/test_persistence_layer.py:97:39
 ERROR Argument `_RepoRecorder` is not assignable to parameter `workflow_node_execution_repository` with type `WorkflowNodeExecutionRepository` in function `core.app.workflow.layers.persistence.WorkflowPersistenceLayer.__init__` [bad-argument-type]
-  --> tests/unit_tests/core/app/workflow/test_persistence_layer.py:94:44
+  --> tests/unit_tests/core/app/workflow/test_persistence_layer.py:98:44
 ERROR Argument `object | None` is not assignable to parameter `trace_manager` with type `TraceQueueManager | None` in function `core.app.workflow.layers.persistence.WorkflowPersistenceLayer.__init__` [bad-argument-type]
-  --> tests/unit_tests/core/app/workflow/test_persistence_layer.py:95:23
+  --> tests/unit_tests/core/app/workflow/test_persistence_layer.py:99:23
 ERROR Argument `None` is not assignable to parameter `command_channel` with type `CommandChannel` in function `graphon.graph_engine.layers.base.GraphEngineLayer.initialize` [bad-argument-type]
-  --> tests/unit_tests/core/app/workflow/test_persistence_layer.py:97:55
+   --> tests/unit_tests/core/app/workflow/test_persistence_layer.py:101:55
 ERROR `object` is not assignable to attribute `_workflow_execution` with type `WorkflowExecution | None` [bad-assignment]
-   --> tests/unit_tests/core/app/workflow/test_persistence_layer.py:105:37
+   --> tests/unit_tests/core/app/workflow/test_\n\n... (truncated) ...

github-actions · 2026-05-11T03:09:54Z

Pyrefly Type Coverage

Metric	Base	PR	Delta
Type coverage	0.00%	43.65%	+43.65%
Strict coverage	0.00%	43.17%	+43.17%
Typed symbols	0	22,022	+22,022
Untyped symbols	0	28,745	+28,745
Modules	0	2549	+2,549

Blackoutta · 2026-05-11T03:14:32Z

update

now requires to merge first: chore(api): upgrade graphon to v0.3.1 #35987, otherwise unit tests related to this feature will fail

Blackoutta · 2026-05-11T03:16:32Z

Phoenix Trace Feature Summary

Date: 2026-05-07

Background and Goals

This feature fixes the hierarchy and session display issues for Dify workflow tracing in Phoenix. The original problems were concentrated in three areas:

Top-level workflow traces could easily become orphan roots, which could result in rootSpan = null in Phoenix session queries.
Nested workflows were split into separate sessions and could not be grouped by the business context of the outer workflow or chatflow.
A child workflow triggered through workflow-as-tool could enter the same trace/session, but still could not be reliably attached back to the outer tool span.

The final design preserves Dify's existing tracing contract: upstream code remains responsible for trace payload generation, workflow execution context, and asynchronous dispatch. The Phoenix provider only fills in what Phoenix/OpenTelemetry needs for display: the canonical root, session.id, span hierarchy, and a provider-local parent span bridge. Later review also moved the retry contract back into core.ops, so the generic trace task does not depend on Phoenix-provider-specific symbols.

Overall Design

Trace Generation and Dispatch Flow

sequenceDiagram
    participant Runtime as Workflow Runtime
    participant Tool as Workflow-as-Tool
    participant Generator as WorkflowAppGenerator
    participant Persistence as Workflow Persistence
    participant Task as ops_trace_task
    participant Phoenix as Phoenix Provider
    participant Redis as Redis Parent Span Bridge
    participant UI as Phoenix UI

    Runtime->>Tool: Execute tool node and inject outer workflow run plus node execution
    Tool->>Generator: Private binding generates parent_trace_context
    Generator->>Persistence: Write parent_trace_context into extras
    Persistence->>Task: TraceTask metadata carries parent context
    Task->>Phoenix: Dispatch trace payload
    Phoenix->>Phoenix: Resolve canonical root, session.id, and node parent
    Phoenix->>Redis: Tool node span publishes traceparent carrier
    Phoenix->>Redis: Nested workflow queries carrier by parent_node_execution_id
    alt parent carrier exists
        Phoenix->>UI: Attach child workflow span to outer tool span
    else parent carrier is not available yet
        Phoenix-->>Task: PendingTraceParentContextError
        Task->>Task: Bounded retry while preserving payload and enterprise dispatch state
        Task->>Phoenix: Retry dispatch
    end

Phoenix Internal Hierarchy Rules

flowchart TD
    A[Trace payload] --> B{Has upstream root context?}
    B -->|Yes| C[Reuse resolved_parent_context]
    B -->|No| D[Top-level workflow/chatflow creates canonical root]

    C --> E[Resolve session.id]
    D --> E
    E --> F{Session source}
    F -->|conversation_id| G[Chatflow or existing conversation session]
    F -->|parent_workflow_run_id| H[Nested workflow inherits outer workflow session]
    F -->|workflow_run_id| I[Top-level workflow session]

    G --> J[Build node span parent]
    H --> J
    I --> J

    J --> K{Parent rule}
    K -->|Explicit predecessor span| L[Attach to predecessor node]
    K -->|Graph parent span| M[Attach to graph-structure parent node]
    K -->|Loop/iteration structured parent| N[Attach to structure node]
    K -->|Ambiguous parallel or missing parent| O[Fallback to workflow root span]

Key Design Decisions

Decision	Conclusion	Reason
V1 boundary	Keep hierarchy and session fallback inside the Phoenix provider for now, without changing `ops_trace_manager`, `trace_entity`, or the enterprise telemetry contract.	This keeps the scope controlled, reuses the existing upstream parent context, and avoids conflicting with the still-evolving core tracing contract.
`session.id` semantics	Top-level workflows use `workflow_run_id`, chatflows use `conversation_id`, and nested workflows inherit the outer session.	`session.id` is a Dify business tracing semantic, not a Phoenix-specific display field. Phoenix Sessions simply consumes that attribute as a product capability.
Canonical root	Top-level workflow/chatflow spans must be true roots, and synthetic parents should not be created for roots.	Orphan roots make it hard for the Phoenix session view to locate the root span, which is the core risk behind `rootSpan = null`.
Node parent rules	Fallback in order: explicit predecessor, graph parent, structured parent, workflow root. Do not infer parentage broadly from execution order.	The rules are explainable and testable. They behave consistently for serial flows, branches, loops, and iterations while avoiding incorrect attachment in parallel-like scenarios.
V1 scope	Cover regular workflows, chatflow messages, nested workflows, workflow-as-tool, and loop/iteration body nodes. Complex concurrent merges and full cross-provider abstraction are out of scope for now.	These scenarios currently provide the highest debugging value in the Phoenix UI. Complex concurrency needs stronger upstream semantics.
Nested workflow session rollout	First connect `outer_workflow_run_id` and session inheritance, then handle precise parent tool span attachment.	Session merging and parent span restoration have different levels of difficulty, so separating them delivers visible value faster.
Parent trace context propagation	Workflow-as-tool parent context uses a private runtime binding instead of public `runtime_parameters`.	Review found that public parameters could overwrite user inputs with the same name. A private binding isolates internal trace metadata from user inputs.
Parent tool span bridge	The Phoenix provider stores the tool node span carrier in Redis, and the child workflow reads it by `parent_node_execution_id`.	Trace dispatch is asynchronous, so the child task can run before the parent tool span exists. The Redis bridge is the smallest current provider-local coordination mechanism.
Pending parent retry	When a child workflow cannot find the parent carrier, it raises the core-layer `PendingTraceParentContextError`; Celery then performs a bounded retry while preserving the payload.	A missing carrier on the first attempt should not silently attach the child to a synthetic root. Bounded retry improves the chance of correct attachment while containing the failure surface.
Provider-neutral retry contract	`ops_trace_task` catches the `RetryableTraceDispatchError` base class, with Phoenix pending-parent as one subclass.	The generic task should not import the Phoenix provider or encode Phoenix semantics in its naming. Future providers can reuse the same retry abstraction.
Enterprise trace idempotency	The retry payload persists enterprise dispatch state, so a successful enterprise dispatch is not repeated on retry.	Pending-parent retry should only retry provider dispatch; it should not duplicate enterprise telemetry.
Test file cleanup	Merge the two same-named `test_arize_phoenix_trace.py` files and keep a single provider test entry point.	This avoids confusion in pytest nodes, review diffs, and maintenance entry points.

Iteration Summary

Started from the prototype and Phoenix session behavior, then confirmed that the problem was not a single missing field. The missing piece was unified semantics for root context, session.id, and node parent rules.
After freezing the Phoenix root/session/hierarchy expectations, implemented canonical root, session fallback, and the workflow node parent helper inside the provider.
Found that workflow-as-tool lacked nested workflow parent context, so the backend call chain was extended: trace_id_helper, workflow tool runtime, WorkflowAppGenerator.generate(), and workflow persistence now pass through parent_trace_context together.
Found next that the nested workflow could enter the same trace/session but still could not attach to the outer tool span, so a Phoenix-local Redis parent span bridge and pending-parent retry were added.
After code review, tightened the boundaries: internal trace metadata moved to a private binding, the generic task now depends on a provider-neutral retry contract, and the retry payload records enterprise dispatch state.
Finally, merged duplicate Phoenix provider test files, preserving the coverage while reducing documentation and test-entry noise.

Test Coverage

Area	Main Scenarios
Phoenix root/session	Top-level workflow uses a canonical root; root span naming uses the workflow run id and handles blank fallback; message traces keep the conversation session; nested workflows inherit the parent workflow session; nested workflows with `conversation_id` keep their own conversation session while reusing the parent root context.
Phoenix node hierarchy	Start node attaches to workflow span; serial node attaches to resolved predecessor; graph parent fallback; loop/iteration body node attaches to the enclosing structure; duplicate body nodes do not overwrite parent; ambiguous parallel-like predecessor falls back to workflow span; graph node title takes precedence over execution title.
Phoenix parent span bridge	Tool node span context is published to Redis; nested workflow uses the published parent node context; missing or malformed carriers are rejected; span is cleaned up when parent context publishing fails; missing parent carrier raises the pending-parent retry signal.
Workflow-as-tool context	Extracts `parent_workflow_run_id` and `parent_node_execution_id` from runtime; omits context when runtime is incomplete; supports explicitly clearing parent context; user input with the old internal key name is not overwritten.
Backend propagation	`WorkflowAppGenerator.generate()` writes parent context into extras; workflow persistence continues passing parent context through graph succeeded/failed trace tasks.
Trace dispatch retry	Retryable dispatch failure preserves the payload and schedules bounded retry; retry scheduling failure is treated as terminal failure; payload is deleted after retries are exhausted; payload is deleted after successful dispatch.
Enterprise trace idempotency	First dispatch writes enterprise state into the retry payload; retry skips enterprise traces that have already been sent successfully, avoiding duplicate reporting.
Provider boundary	`ops_trace_task` uses the `RetryableTraceDispatchError` base class for retry behavior and does not directly depend on the Phoenix provider; Phoenix tests still verify the concrete `PendingTraceParentContextError`.
Existing trace provider behavior	Existing Arize/Phoenix provider behavior remains covered, including tracer setup, config validation, span status, metadata serialization, and message/tool/moderation/dataset/generate-name traces.

The final focused verification command recorded for this feature was:

uv run --project api pytest api/providers/trace/trace-arize-phoenix/tests/unit_tests/arize_phoenix_trace/test_arize_phoenix_trace.py

Result: 67 passed.

Main Code Impact

api/core/ops/exceptions.py: Adds the provider-neutral trace dispatch retry exception contract.
api/core/helper/trace_id_helper.py, api/core/tools/workflow_as_tool/tool.py, api/core/workflow/node_runtime.py: Generate and carry workflow-as-tool parent trace context.
api/core/app/apps/workflow/app_generator.py, api/core/app/workflow/layers/persistence.py: Write parent trace context into workflow extras and trace task metadata.
api/providers/trace/trace-arize-phoenix/src/dify_trace_arize_phoenix/arize_phoenix_trace.py: Implements Phoenix root/session/hierarchy behavior, parent span bridge, and span naming.
api/tasks/ops_trace_task.py: Implements provider-neutral bounded retry and enterprise dispatch idempotency protection.
api/tests/unit_tests/** and api/providers/trace/trace-arize-phoenix/tests/unit_tests/**: Cover the behavior above.

Future Migration Direction

The current Phoenix-local helper is a transitional design. If the upstream tracing builder later exposes a stable provider-agnostic root/session/node-parent/parent-span contract, the following logic can gradually move out or be removed:

Session fallback inside the Phoenix provider.
Workflow node hierarchy reconstruction inside the Phoenix provider.
Redis parent span bridge.
Phoenix-specific trigger conditions in pending-parent retry.

Adapt the backend Graphon integration to the v0.3.0 breaking changes. Migrate provider factory and runtime usage, switch workflow node construction to the new data payload API, and refresh backend tests for the updated VariablePool and node behaviors.

This reverts commit 9ec083d.

Blackoutta · 2026-05-11T07:26:15Z

latest live demo:
https://github.com/user-attachments/assets/9bb4834d-7e57-4b56-9f47-ecc3b062cf88

Blackoutta requested review from QuantumGhost, crazywoola and laipz8200 as code owners April 27, 2026 12:43

dosubot Bot added size:XXL This PR changes 1000+ lines, ignoring generated files. 🌊 feat:workflow Workflow related stuff. labels Apr 27, 2026

Blackoutta force-pushed the feature/improve-phoenix-squashed branch from 533a207 to 5e8df5b Compare April 28, 2026 01:43

Blackoutta requested review from Mairuis, Stream29 and Yeuoly as code owners April 28, 2026 01:43

dosubot Bot added size:XL This PR changes 500-999 lines, ignoring generated files. and removed size:XXL This PR changes 1000+ lines, ignoring generated files. labels Apr 28, 2026

This was referenced Apr 28, 2026

fix: preserve tool runtime compatibility langgenius/graphon#105

Merged

feat: try to see the hierarchy in phonenix trace rather than all single events #35027

Closed

Blackoutta self-assigned this Apr 29, 2026

Blackoutta force-pushed the feature/improve-phoenix-squashed branch from 5e8df5b to d610170 Compare May 7, 2026 02:39

Blackoutta force-pushed the feature/improve-phoenix-squashed branch from d610170 to 0d7b4f9 Compare May 11, 2026 03:07

Blackoutta force-pushed the feature/improve-phoenix-squashed branch from 7f76347 to 6a24380 Compare May 11, 2026 06:26

Blackoutta enabled auto-merge May 11, 2026 06:31

Blackoutta and others added 4 commits May 11, 2026 14:54

feat: improve phoenix workflow tracing

7e35127

[autofix.ci] apply automated fixes

f80c7e1

docs: consolidate phoenix trace notes

535f35e

Blackoutta added 6 commits May 11, 2026 14:54

Revert "chore(api): upgrade graphon to v0.3.0"

b674332

This reverts commit 9ec083d.

fix: errors found by ruff check

d78c9c7

docs: delete feature summary doc(updated to the issue thread)

56fdfe1

test: align workflow tool trace context expectation

7b2eb36

refactor: type parent trace context

dc2b4ca

fix: preserve parent workflow trace fallback

545645b

Blackoutta force-pushed the feature/improve-phoenix-squashed branch from 6a24380 to 545645b Compare May 11, 2026 06:54

laipz8200 added 3 commits May 11, 2026 16:07

fix(api): harden phoenix trace retry handling

133bf2c

chore(docker): expose ops trace retry env vars

bbf1ea8

chore(docker): document root env example scope

d0a0093

laipz8200 approved these changes May 11, 2026

View reviewed changes

dosubot Bot added the lgtm This PR has been approved by a maintainer label May 11, 2026

Blackoutta added this pull request to the merge queue May 11, 2026

Merged via the queue into langgenius:main with commit 7fc40e6 May 11, 2026
28 checks passed

Blackoutta deleted the feature/improve-phoenix-squashed branch May 11, 2026 08:46

dosubot Bot mentioned this pull request May 12, 2026

Phoenix tracing is lost after resuming HITL workflows #36063

Closed

6 tasks

dosubot Bot mentioned this pull request May 20, 2026

Phoenix tracing misses loop/iteration grouping and handled node error visibility #36387

Closed

5 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: improve phoenix workflow tracing#35605

feat: improve phoenix workflow tracing#35605
Blackoutta merged 13 commits into
langgenius:mainfrom
Blackoutta:feature/improve-phoenix-squashed

Blackoutta commented Apr 27, 2026 •

edited

Loading

Uh oh!

Blackoutta commented Apr 27, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Apr 27, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Apr 27, 2026

Uh oh!

github-actions Bot commented Apr 28, 2026

Uh oh!

github-actions Bot commented May 7, 2026

Uh oh!

github-actions Bot commented May 11, 2026 •

edited

Loading

Uh oh!

Blackoutta commented May 11, 2026

Uh oh!

Blackoutta commented May 11, 2026 •

edited

Loading

Uh oh!

Blackoutta commented May 11, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

Blackoutta commented Apr 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Cross-repo dependency

Reviewer notes

Screenshots

Verification

Checklist

Uh oh!

Blackoutta commented Apr 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Apr 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pyrefly Diff

Uh oh!

github-actions Bot commented Apr 27, 2026

Pyrefly Diff

Uh oh!

github-actions Bot commented Apr 28, 2026

Pyrefly Diff

Uh oh!

github-actions Bot commented May 7, 2026

Pyrefly Diff

Uh oh!

github-actions Bot commented May 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pyrefly Type Coverage

Uh oh!

Blackoutta commented May 11, 2026

update

Uh oh!

Blackoutta commented May 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Phoenix Trace Feature Summary

Background and Goals

Overall Design

Trace Generation and Dispatch Flow

Phoenix Internal Hierarchy Rules

Key Design Decisions

Iteration Summary

Test Coverage

Main Code Impact

Future Migration Direction

Uh oh!

Blackoutta commented May 11, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Blackoutta commented Apr 27, 2026 •

edited

Loading

Blackoutta commented Apr 27, 2026 •

edited

Loading

github-actions Bot commented Apr 27, 2026 •

edited

Loading

github-actions Bot commented May 11, 2026 •

edited

Loading

Blackoutta commented May 11, 2026 •

edited

Loading