Investigate: <include> tags in step outputs bypass preprocessing

[jamesdlevine]

Summary

When agentic orchestrators process templates, preprocess() runs on the template before format() substitutes context values. This means any <include> tags that end up in step outputs (e.g., step1_output, step2_output) will bypass preprocessing when substituted into subsequent step templates.

This affects all agentic orchestrators and potentially other consumers of preprocess().

Current Flow

1. preprocess() runs on the template (expands <include> directives)
2. format() substitutes context values (including stepN_output)
3. Context values are NOT preprocessed

Potential Issue

If an <include> tag ends up in a step's output:

• It gets substituted into later templates via {stepN_output}
• Substitution happens AFTER preprocess
• The <include> tag passes through unexpanded to the LLM

How this could happen:

1. An early step's <include> wasn't expanded (e.g., file not found) and the LLM echoed it back
2. The LLM generated an <include> tag in its response

Scope

• agentic_change_orchestrator.py
• agentic_test_orchestrator.py (if applicable)
• Any other agentic orchestrators
• Other consumers of preprocess() that use context substitution after preprocessing

Options to Consider

1. Preprocess context values before adding them to the context dict
2. Post-format scan for <include> tags and warn/expand
3. Accept the limitation - document that step outputs shouldn't contain <include> tags
4. Centralized solution in preprocess or a wrapper function used by all orchestrators

Related

This was identified during work on the template preprocessing fix for agentic_change_orchestrator.py (branch: fix/change-orchestrator-template-preprocessing).

Action

1. Investigate whether this is a real problem in practice
2. Audit all agentic orchestrators and preprocess consumers
3. Determine the appropriate fix if needed