Circular include cycle detection missing in non-recursive preprocess path

[Serhan-Asad]

Bug Description

Circular <include> tags cause an infinite loop when using the default (non-recursive) preprocess path. The process hangs forever, spamming "Processing XML include" messages until manually killed.

Reproduction

mkdir /tmp/circ-test && cd /tmp/circ-test && git init
echo '<include>b.txt</include>' > a.txt
echo '<include>a.txt</include>' > b.txt
echo '<include>a.txt</include>' > circular_python.prompt
timeout 10 pdd preprocess circular_python.prompt
# Process hangs, spams "Processing XML include" until timeout kills it

Root Cause

process_include_tags() in preprocess.py has a while prev_text != current_text convergence loop (line 295). The _seen set for cycle detection is only populated inside the if recursive: block (lines 272-274). In the non-recursive path, _seen is never updated, so the cycle check at line 226 (if resolved in _seen) never triggers.

The same issue exists in process_backtick_includes() (line 200-205).

Context

PR #528 (merged, fix for #521) added cycle detection for the recursive path. This is a follow-up — the non-recursive (default) path was not covered by that fix.

Expected Behavior

pdd preprocess should detect the circular include and raise a ValueError("Circular include detected: ...") instead of looping forever. Same behavior as the recursive path.

[Serhan-Asad]

Step 1: Duplicate Check

Status: Related to #526 (closed prematurely) — proceeding as new issue

Search Performed

• Searched for: "circular include", "infinite loop preprocess", "cycle detection"
• Issues reviewed: 5 (Circular <include> tags silently produce corrupted output (no cycle detection) #521, Circular <include> tags WITHOUT --recursive cause infinite loop (process never terminates) #526, Sync fingerprint ignores <include> dependencies — stale code after included file changes #522, IndexError crash when pytest collection/execution fails (empty test_results) #450, Circular include cycle detection missing in non-recursive preprocess path #553)

Findings

#521 (CLOSED): Circular <include> tags with --recursive — silent corruption via RecursionError swallowed by broad except Exception. Fixed by PR #528, which added _seen set tracking to the recursive path. Different code path — recursive mode only.

#526 (CLOSED): Circular <include> tags WITHOUT --recursive — infinite loop, process never terminates. This describes the exact same bug as #553. However, #526 was closed with the comment "Closing as duplicate of #521 — same root cause. PR #528 covers both XML and backtick includes."

The closure of #526 was premature. PR #528 did NOT fix the non-recursive path. Code verification confirms:

• process_include_tags() line 272-274: _seen is only populated inside if recursive: — the non-recursive path skips this entirely
• process_backtick_includes() line 179-181: Same pattern — _seen only updated when recursive=True
• The while loops at lines 295 and 202 have no cycle guard in non-recursive mode

The bug described in #526 (and now #553) was never actually fixed. #553 should proceed as the canonical issue for this bug.

---

Proceeding to Step 2: Documentation Check

[Serhan-Asad]

Step 2: Documentation Check

Status: Confirmed Bug

Documentation Reviewed

• docs/prompting_guide.md — <include> tag processing, recursive vs non-recursive behavior
• README.md — general project docs
• pdd/preprocess.py — inline code comments, docstrings, _seen set logic
• tests/test_circular_includes.py — existing test expectations
• PR Add failing tests for circular include cycle detection (#521) #528 commit history — scope of the recursive-path fix

Findings

No documentation describes non-recursive cycle detection as unsupported. In fact, the code clearly intends to detect cycles in both paths:

1. Both process_include_tags() and process_backtick_includes() contain if resolved in _seen: raise ValueError("Circular include detected...") checks that apply regardless of the recursive flag.

2. The bug is purely an implementation gap: _seen is only populated inside the if recursive: block, so the check always evaluates to False in the non-recursive while loop (lines 293–303 and 200–204).

3. Existing tests confirm this is a bug. tests/test_circular_includes.py explicitly documents: "The preprocessor has no cycle detection — circular includes recurse ~82 times until Python's recursion limit, then the broad except Exception swallows the RecursionError and returns corrupted output with exit code 0." Test cases for the non-recursive path exist and are expected to fail.

4. The ValueError message and exception handling prove original intent was to catch cycles in all paths — the non-recursive path simply wasn't updated when PR Add failing tests for circular include cycle detection (#521) #528 added the _seen tracking for the recursive path.

Verdict: This is a confirmed bug where behavior contradicts the code's own intended design. The cycle detection infrastructure exists but is not wired up for the non-recursive (default) path.

---

Proceeding to Step 3: Triage

[Serhan-Asad]

Step 3: Triage

Status: Proceed

Summary

All four triage criteria are satisfied:

1. Expected behavior — Clear: pdd preprocess should raise ValueError("Circular include detected: ...") when circular <include> tags are encountered, regardless of whether --recursive is used.

2. Actual behavior — Clear: The process hangs in an infinite while prev_text != current_text loop, spamming "Processing XML include" messages until manually killed. This happens because _seen is never populated in the non-recursive path (preprocess.py:272-274 only runs inside if recursive:), so the cycle check at line 226 never triggers.

3. Reproduction steps — Provided and specific: Create two files with mutual <include> tags, create a .prompt file including one of them, run pdd preprocess. The issue also identifies the same gap in process_backtick_includes() (lines 200-205).

4. Code area — Precisely identified: pdd/preprocess.py, two functions:

   • process_include_tags() — while loop at line 295, _seen update at lines 272-274 (recursive-only)
   • process_backtick_includes() — while loop at line 202, _seen update at lines 178-179 (recursive-only)

Steps 1 and 2 confirmed this is not a duplicate (#521/#528 only fixed the recursive path) and that existing tests already document the broken non-recursive behavior.

---

Proceeding to Step 4: Reproduction

[Serhan-Asad]

Step 4: Reproduction

Status: Reproduced

Environment

• macOS Darwin 24.3.0 (Apple Silicon)
• Python 3.13.2
• pdd v0.0.157 (installed from promptdriven/pdd main branch at commit f9df4b8f)

Steps Taken

1. Set up test files with mutual circular <include> references:

   mkdir /tmp/circ-test && cd /tmp/circ-test && git init
   echo '<include>b.txt</include>' > a.txt
   echo '<include>a.txt</include>' > b.txt
   echo '<include>a.txt</include>' > circular_python.prompt

2. Ran pdd preprocess circular_python.prompt (non-recursive, the default):

   • Process entered infinite loop, spamming Processing XML include: ./a.txt / Processing XML include: ./b.txt alternating endlessly
   • Killed after 5 seconds — had already produced 51+ lines of output and showed no sign of stopping
   • Exit code: -9 (SIGKILL from forced termination)

3. Ran pdd preprocess --recursive circular_python.prompt (recursive mode):

   • Correctly detected the cycle: Error during preprocessing: Circular include detected: a.txt is already in the include chain
   • Exited with code 1 as expected

4. Called process_include_tags() directly from Python to confirm at function level:

   • process_include_tags(text, recursive=False) → infinite loop (did not return within 3 seconds, had to SIGALRM)
   • process_include_tags(text, recursive=True) → correctly raised ValueError("Circular include detected: ...")

Observed Behavior

Non-recursive (default) — BROKEN:

Processing XML include: ./a.txt
Processing XML include: ./b.txt
Processing XML include: ./a.txt
Processing XML include: ./b.txt
Processing XML include: ./a.txt
Processing XML include: ./b.txt
... (repeats forever, process never terminates)

Recursive — CORRECT:

Processing XML include: ./a.txt
Processing XML include: ./b.txt
Error during preprocessing: Circular include detected: a.txt is already in the include chain

Affected Files

• pdd/preprocess.py:272-274 — _seen is only populated inside if recursive: block in process_include_tags()
• pdd/preprocess.py:179-181 — Same issue in process_backtick_includes()
• pdd/preprocess.py:295 — The while prev_text != current_text convergence loop runs forever because substitution always changes the text (A→B→A→B...)
• tests/test_circular_includes.py — Existing tests only cover recursive=True; no test cases for the non-recursive path

Notes

• The infinite loop mechanism is clear: each iteration of the while loop replaces <include>a.txt</include> with <include>b.txt</include> content (which contains <include>a.txt</include>), so prev_text != current_text is always True — the text oscillates between two states
• The _seen check at line 226 exists but is a no-op in non-recursive mode because _seen is never populated (lines 272-274 are inside if recursive:)
• Both process_include_tags() and process_backtick_includes() have the same structural bug
• This is confirmed as a follow-up to PR Add failing tests for circular include cycle detection (#521) #528 which fixed cycle detection for recursive mode only

---

Proceeding to Step 5: Root Cause Analysis

[Serhan-Asad]

Step 5: Root Cause Analysis

Root Cause Identified: Yes

Summary

The _seen set used for circular include cycle detection in preprocess.py is only populated inside the if recursive: code path (lines 272-274 for XML includes, lines 179-181 for backtick includes). In the default non-recursive mode, the convergence loop (while prev_text != current_text at line 295) repeatedly calls re.sub to expand <include> tags, but never adds expanded files to _seen. This means the cycle check at line 226 (if resolved in _seen) always sees an empty set and never triggers. When file A includes file B and B includes A, the text oscillates between two states forever — the loop never converges.

Technical Details

• Location: pdd/preprocess.py:272-274 (XML includes), pdd/preprocess.py:179-181 (backtick includes)

• Problematic Code (XML includes):

# Line 226 — cycle check (always passes because _seen is empty)
if resolved in _seen:
    raise ValueError(f"Circular include detected: {file_path} is already in the include chain")

# ...

# Lines 272-274 — _seen is ONLY populated here, inside `if recursive:`
if recursive:
    child_seen = _seen | {resolved}
    content = preprocess(content, recursive=True, double_curly_brackets=False, _seen=child_seen)

# ...

# Lines 293-302 — convergence loop (no cycle guard)
prev_text = ""
current_text = text
while prev_text != current_text:  # Never converges with circular includes
    prev_text = current_text
    current_text = re.sub(pattern, replace_include_with_spans, current_text, flags=re.DOTALL)

• Why it fails: The convergence loop pattern while prev_text != current_text assumes repeated expansion will reach a fixed point. With circular includes (A→B→A), the text oscillates: iteration 1 produces <include>b.txt</include>\n, iteration 2 produces <include>a.txt</include>\n\n, iteration 3 produces <include>b.txt</include>\n\n\n, etc. Each iteration differs from the previous (trailing newlines accumulate), so the loop never terminates. This is the text-processing analog of a 2-cycle in fixed-point iteration — a well-known divergence pattern.

• Same bug in backtick includes: process_backtick_includes() has the identical structure — _seen populated only at line 179-181 inside if recursive:, convergence loop at lines 200-204 with no guard.

External Research

• Search terms: "circular include detection convergence loop", "template engine cycle detection", "fixed-point iteration oscillation"
• Sources checked: GitHub issues for Go text/template, Jinja2, Handlebars; Wikipedia (fixed-point iteration, include guards); GCC preprocessor internals; clang-tidy docs
• Relevant findings:
  • Go text/template uses a max depth counter of 100,000 (no explicit cycle detection) — golang/go#21914
  • Jinja2 has no cycle detection at all; relies on Python's RecursionError — pallets/jinja#1032
  • Handlebars.java blocks circular partials by default using ancestor tracking
  • C preprocessor uses include guards (idempotency) plus #pragma once — a structural prevention approach
  • Best practice: Track files expanded per iteration; if a file reappears in a subsequent iteration, it's circular. This is the "ancestor set" approach from directed graph cycle detection.

Repository History

• Relevant commits:

  • 0804e2a3b (2025-09-01, Greg Tanaka) — Original convergence loop created, no cycle detection at all
  • b371bf1a2 (2025-10-25, Jiamin) — Added if recursive: branch, still no cycle detection
  • 0d5e6587c (2026-02-14, Serhan) — Added _seen parameter and cycle detection, but ONLY inside if recursive: blocks
  • 45df29ec (2026-02-15, Serhan, branch fix/issue-526) — Attempted proper fix with per-iteration tracking (_previously_expanded / _current_iteration), never merged to main

• Related PRs:

  • PR Add failing tests for circular include cycle detection (#521) #528 (merged 2026-02-15) — Added failing tests for circular includes + the recursive-path fix. Did not cover the non-recursive convergence loop.
  • Branch fix/issue-526 (not merged) — Contains a correct per-iteration tracking approach that should inform this fix.

• Similar past issues:

  • Circular <include> tags silently produce corrupted output (no cycle detection) #521 (closed) — Circular includes in recursive mode → silent corruption. Fixed by PR Add failing tests for circular include cycle detection (#521) #528.
  • Circular <include> tags WITHOUT --recursive cause infinite loop (process never terminates) #526 (closed prematurely as duplicate of Circular <include> tags silently produce corrupted output (no cycle detection) #521) — Described this exact non-recursive bug. Should not have been closed.
  • Circular include cycle detection missing in non-recursive preprocess path #553 (this issue) — Correctly identifies the remaining gap.

Experiments Performed

1. Oscillation verification: Patched re.sub with a counter. Confirmed the convergence loop oscillates indefinitely between <include>a.txt</include> and <include>b.txt</include>, with trailing newlines accumulating each iteration (each iteration's text is unique and longer than the previous).

2. Recursive mode verification: Same test with recursive=True correctly raises ValueError("Circular include detected: ...") — the _seen set works when populated.

3. Previous fix review: The fix/issue-526 branch (commit 45df29ec) uses per-iteration file tracking — _previously_expanded accumulates files across iterations, _current_iteration tracks files in the current pass. If a file appears in _previously_expanded, it's circular. This approach is correct for the convergence loop pattern and should be the basis for the fix.

Fix Location

• File(s) to modify: pdd/preprocess.py
• Why this location: The root cause IS in preprocess.py — the _seen set is defined and populated here, and the convergence loops that lack protection are here. This is not a symptom location; it's where the cycle detection logic lives.
• Existing pattern: The recursive=True path already correctly uses _seen to detect cycles (lines 272-274). The fix/issue-526 branch demonstrates the correct pattern for the convergence loop: track files expanded per iteration and check against previously expanded files.
• Fix approach: In both process_include_tags() and process_backtick_includes():
  1. Add _seen.add(resolved) (or equivalent per-iteration tracking) in the non-recursive path, so the cycle check at lines 174/226 actually triggers
  2. Alternatively, use the fix/issue-526 pattern: maintain _previously_expanded and _current_iteration sets around the convergence loop, raising ValueError when a file reappears
  3. As defense-in-depth, consider adding a max iteration guard on the convergence loop (e.g., 50 iterations) as a safety net

---

Proceeding to Step 5.5: Prompt Classification

[Serhan-Asad]

Step 5.5: Prompt Classification

Classification: Code Bug

DEFECT_TYPE: code

Analysis

This is a code bug — the implementation does not correctly carry out its own design intent. No dedicated prompt file exists for the preprocess module, so the classification is based on the code's evident intent vs. its actual behavior.

The code in process_include_tags() and process_backtick_includes() already has all the machinery for circular include detection:

• A _seen: Optional[set] parameter (lines 164, 217)
• A cycle check: if resolved in _seen: raise ValueError("Circular include detected: ...") (lines 174, 226)

However, _seen is only populated inside if recursive: blocks (lines 179-181, 272-274). In the default non-recursive path, _seen stays empty, making the cycle check dead code. The convergence loop (while prev_text != current_text) then oscillates forever.

Evidence

• Code design specifies: Cycle detection via _seen set — present in both function signatures and both cycle checks
• Code implements: _seen only populated in if recursive: blocks — non-recursive path has no cycle guard
• User expects: ValueError("Circular include detected: ...") in both modes — matches the code's design intent

Conclusion

The code does not correctly implement its own design. The _seen tracking was added by PR #528 for the recursive path but omitted from the non-recursive convergence loop. This is an incomplete implementation, not a prompt defect. Proceeding with test generation.

---

Proceeding to Step 6: Test Plan

[Serhan-Asad]

Step 6: Test Plan

Existing Test Coverage

• Test file: tests/test_circular_includes.py
  • Current coverage: Tests circular <include> and backtick includes for A→B→A, self-referencing, three-file cycle, diamond (non-circular), and non-circular chain — all with recursive=True only
  • Gap: Zero tests for circular includes with recursive=False (the default, non-recursive path). Every single circular test passes recursive=True to preprocess().
• Test file: tests/test_e2e_issue_521_circular_includes.py
  • Current coverage: E2E CLI tests for circular includes — all with --recursive flag only
  • Gap: No CLI test for the default (non-recursive) path with circular includes.
• Test file: tests/test_preprocess.py
  • Current coverage: Normal (non-circular) includes with recursive=False — working correctly
  • Gap: No circular include scenarios tested in this file

Proposed Tests

All tests will be added to tests/test_circular_includes.py alongside the existing recursive-mode tests. Tests must use pytest.mark.timeout(10) to prevent infinite loops from hanging the test suite (since the bug manifests as an infinite loop, not a wrong result).

Test 1: test_circular_xml_includes_non_recursive_must_error

• Input: A→B→A mutual <include> cycle, preprocess(..., recursive=False)
• Expected: Raises ValueError("Circular include detected: ...")
• Actual (before fix): Infinite loop — while prev_text != current_text never terminates because _seen is never populated

Test 2: test_circular_backtick_includes_non_recursive_must_error

• Input: Mutual backtick cycle (```<x.txt>``` ↔ ```<y.txt>```), preprocess(..., recursive=False)
• Expected: Raises ValueError("Circular include detected: ...")
• Actual (before fix): Infinite loop in process_backtick_includes() convergence loop (line 202)

Test 3: test_self_referencing_include_non_recursive_must_error

• Input: A file that includes itself via <include>self.txt</include>, recursive=False
• Expected: Raises ValueError("Circular include detected: ...")
• Actual (before fix): Infinite loop — self-reference causes text to expand each iteration

Test 4: test_three_file_cycle_non_recursive_must_error

• Input: A→B→C→A three-file cycle, recursive=False
• Expected: Raises ValueError("Circular include detected: ...")
• Actual (before fix): Infinite loop

Test 5: test_non_circular_includes_non_recursive_still_work (regression guard)

• Input: Non-circular include chain A→B (no cycle), recursive=False
• Expected: Returns resolved content with both files' content — no exception
• Purpose: Ensures the cycle detection fix doesn't falsely flag legitimate (non-circular) includes

Test 6: test_diamond_includes_non_recursive_not_falsely_flagged (regression guard)

• Input: Diamond pattern A→{B,C}→D (shared dependency, no cycle), recursive=False
• Expected: D's content appears twice (via B and via C) — no exception
• Purpose: Diamond includes are NOT cycles; the fix must not break them

Test Location

• File: tests/test_circular_includes.py (append to existing file)
• Framework: pytest with unittest.mock (matching existing pattern)
• Class: New class TestCircularIncludesNonRecursive alongside existing TestCircularIncludes

Notes

• All circular-detection tests MUST use @pytest.mark.timeout(10) — without it, a failing test would hang the CI runner indefinitely (the bug is an infinite loop, not a wrong value)
• The existing _make_mock_open() helper and file-map pattern from the existing tests will be reused
• Tests assert specifically for ValueError with "Circular include detected" message, matching the existing error format in preprocess.py lines 175 and 227
• The CI command pytest tests/ -m "not integration and not e2e and not real" --timeout=60 provides a global timeout safety net, but per-test @pytest.mark.timeout(10) gives faster failure feedback

---

Proceeding to Step 7: Generate Test

[Serhan-Asad]

Step 7: Generated Test

Test File

tests/test_circular_includes.py (appended TestCircularIncludesNonRecursive class)

Test Code

class TestCircularIncludesNonRecursive:
    """Issue #553: Circular includes must be detected in the non-recursive (default) path.

    PR #528 added cycle detection for recursive=True, but the non-recursive path
    (which uses a while-loop for convergence) never populates the _seen set,
    so circular includes cause an infinite loop instead of raising ValueError.
    """

    @pytest.mark.timeout(10)
    def test_circular_xml_includes_non_recursive_must_error(self):
        """A→B→A circular XML include with recursive=False must raise ValueError, not loop forever."""
        file_map = {
            'a.txt': 'Hello\n<include>b.txt</include>',
            'b.txt': 'World\n<include>a.txt</include>',
        }
        with patch('builtins.open', mock_open()) as m:
            m.side_effect = _make_mock_open(file_map)
            with pytest.raises(ValueError, match="Circular include detected"):
                preprocess('<include>a.txt</include>', recursive=False, double_curly_brackets=False)

    @pytest.mark.timeout(10)
    def test_circular_backtick_includes_non_recursive_must_error(self):
        """A→B→A circular backtick include with recursive=False must raise ValueError."""
        file_map = {
            'x.txt': 'Foo\n```<y.txt>```',
            'y.txt': 'Bar\n```<x.txt>```',
        }
        with patch('builtins.open', mock_open()) as m:
            m.side_effect = _make_mock_open(file_map)
            with pytest.raises(ValueError, match="Circular include detected"):
                preprocess('```<x.txt>```', recursive=False, double_curly_brackets=False)

    @pytest.mark.timeout(10)
    def test_self_referencing_include_non_recursive_must_error(self):
        """A file that includes itself must be detected in non-recursive mode."""
        file_map = {'self.txt': 'Content\n<include>self.txt</include>'}
        with patch('builtins.open', mock_open()) as m:
            m.side_effect = _make_mock_open(file_map)
            with pytest.raises(ValueError, match="Circular include detected"):
                preprocess('<include>self.txt</include>', recursive=False, double_curly_brackets=False)

    @pytest.mark.timeout(10)
    def test_three_file_cycle_non_recursive_must_error(self):
        """A→B→C→A three-file cycle must be detected in non-recursive mode."""
        file_map = {
            'a.txt': 'AAA\n<include>b.txt</include>',
            'b.txt': 'BBB\n<include>c.txt</include>',
            'c.txt': 'CCC\n<include>a.txt</include>',
        }
        with patch('builtins.open', mock_open()) as m:
            m.side_effect = _make_mock_open(file_map)
            with pytest.raises(ValueError, match="Circular include detected"):
                preprocess('<include>a.txt</include>', recursive=False, double_curly_brackets=False)

    def test_non_circular_includes_non_recursive_still_work(self):
        """Non-circular includes (A→B→C, no cycle) must still work in non-recursive mode."""
        file_map = {
            'top.txt': 'Top\n<include>mid.txt</include>',
            'mid.txt': 'Mid\n<include>leaf.txt</include>',
            'leaf.txt': 'Leaf',
        }
        with patch('builtins.open', mock_open()) as m:
            m.side_effect = _make_mock_open(file_map)
            result = preprocess('<include>top.txt</include>', recursive=False, double_curly_brackets=False)
            assert 'Top' in result
            assert 'Mid' in result
            assert 'Leaf' in result

    def test_diamond_includes_non_recursive_not_falsely_flagged(self):
        """Diamond pattern (A→B, A→C, B→D, C→D) is NOT circular and must work."""
        file_map = {
            'a.txt': '<include>b.txt</include>\n<include>c.txt</include>',
            'b.txt': 'B\n<include>d.txt</include>',
            'c.txt': 'C\n<include>d.txt</include>',
            'd.txt': 'Shared',
        }
        with patch('builtins.open', mock_open()) as m:
            m.side_effect = _make_mock_open(file_map)
            result = preprocess('<include>a.txt</include>', recursive=False, double_curly_brackets=False)
            assert 'B' in result
            assert 'C' in result
            assert result.count('Shared') == 2

What This Test Verifies

4 failing tests (detect the bug):

• Tests 1-4 call preprocess() with recursive=False (the default) and circular <include> / backtick-include patterns
• On buggy code: the while prev_text != current_text convergence loop in process_include_tags() / process_backtick_includes() oscillates forever because _seen is never populated → timeout after 10s → test FAILS
• On fixed code: ValueError("Circular include detected: ...") is raised → test PASSES

2 regression guards (pass on both buggy and fixed code):

• Test 5: A→B→C linear chain converges naturally in the while-loop
• Test 6: Diamond pattern (A→B→D, A→C→D) converges — D included twice is not circular

Test Results on Current (Buggy) Code

FAILED  test_circular_xml_includes_non_recursive_must_error     - Timeout >10s (infinite loop)
FAILED  test_circular_backtick_includes_non_recursive_must_error - Timeout >10s (infinite loop)
FAILED  test_self_referencing_include_non_recursive_must_error   - Timeout >10s (infinite loop)
FAILED  test_three_file_cycle_non_recursive_must_error           - Timeout >10s (infinite loop)
PASSED  test_non_circular_includes_non_recursive_still_work
PASSED  test_diamond_includes_non_recursive_not_falsely_flagged

All 6 original TestCircularIncludes (recursive=True) tests still pass.

Running the Test

pytest tests/test_circular_includes.py::TestCircularIncludesNonRecursive -v --timeout=15

---

Proceeding to Step 8: Verification

[Serhan-Asad]

Step 8: Verification Complete

Test Execution

$ pytest tests/test_circular_includes.py::TestCircularIncludesNonRecursive -v --timeout=15

tests/test_circular_includes.py::TestCircularIncludesNonRecursive::test_circular_xml_includes_non_recursive_must_error FAILED [ 16%]
tests/test_circular_includes.py::TestCircularIncludesNonRecursive::test_circular_backtick_includes_non_recursive_must_error FAILED [ 33%]
tests/test_circular_includes.py::TestCircularIncludesNonRecursive::test_self_referencing_include_non_recursive_must_error FAILED [ 50%]
tests/test_circular_includes.py::TestCircularIncludesNonRecursive::test_three_file_cycle_non_recursive_must_error FAILED [ 66%]
tests/test_circular_includes.py::TestCircularIncludesNonRecursive::test_non_circular_includes_non_recursive_still_work PASSED [ 83%]
tests/test_circular_includes.py::TestCircularIncludesNonRecursive::test_diamond_includes_non_recursive_not_falsely_flagged PASSED [100%]

4 failed, 2 passed in 40.39s

All 4 circular-include tests fail with Failed: Timeout (>10.0s) from pytest-timeout, confirming the infinite loop. Stdout shows repeated "Processing XML include" messages cycling through the include chain (e.g., ./a.txt → ./b.txt → ./a.txt → ...) until the timeout kills the test.

The 2 regression guard tests pass, confirming non-circular includes and diamond patterns work correctly in non-recursive mode.

All 6 existing recursive-path tests (TestCircularIncludes) continue to pass (6 passed in 0.27s).

Verification Status

PASS: Test correctly detects the bug

All 4 circular-include tests fail for exactly the right reason: the non-recursive convergence loop in process_include_tags() (and process_backtick_includes()) never terminates because _seen is not populated, so the cycle check is dead code and the while prev_text != current_text loop oscillates forever.

Summary

Step	Result
Duplicate Check	Confirmed not a duplicate — #521/#528 only fixed recursive path
Documentation	No documented limitation — code intends cycle detection in both paths
Triage	Proceed — clear expected vs actual behavior, precise code location
Reproduction	Reproduced — infinite loop at CLI and function level, recursive mode works correctly
Root Cause	_seen only populated inside if recursive: blocks (lines 272-274, 179-181)
Prompt Classification	Code Bug (DEFECT_TYPE: code) — no prompt file involved
Test Plan	6 tests planned: 4 circular (must error) + 2 regression guards
Test Generation	6 tests written, all behave as expected against buggy code
Verification	PASS — 4 circular tests fail (timeout/infinite loop), 2 guards pass

Bug Details

• Location: pdd/preprocess.py:272-274 (XML includes) and pdd/preprocess.py:179-181 (backtick includes)
• Root Cause: _seen set is only populated inside if recursive: blocks, making the cycle check at line 226 dead code in the default non-recursive path; the convergence loop at line 295 (while prev_text != current_text) oscillates forever with circular includes
• Test File: tests/test_circular_includes.py (class TestCircularIncludesNonRecursive)

Next Steps

1. Fix the bug by populating _seen in the non-recursive path (move _seen.add(resolved) outside the if recursive: block, or add per-iteration tracking in the while loop)
2. Run the test to confirm the fix: pytest tests/test_circular_includes.py::TestCircularIncludesNonRecursive -v --timeout=15
3. Run full test suite to check for regressions: pytest tests/ -m "not integration and not e2e and not real" --timeout=60
4. Submit PR with fix and test

---

Proceeding to Step 9: E2E Tests

[Serhan-Asad]

Step 9: E2E Test

Environment Discovery

• Repository language: Python
• Test framework: pytest (pytest-timeout, pytest-xdist, pytest-mock)
• Existing E2E tests found: Yes — 40+ files matching test_e2e_*.py, including test_e2e_issue_521_circular_includes.py (closely related, tests recursive path only)
• Test configuration: pytest.ini — markers: real, integration, slow, e2e; timeout: 60s default
• Fixtures/helpers used: tmp_path (pytest builtin), conftest.py with preserve_pdd_path and restore_standard_streams autouse fixtures
• Environment variables required: None for this test (preprocess is local-only, no LLM calls)
• Auth/credentials: Not applicable (preprocess is offline)

E2E Test File

tests/test_e2e_issue_553_circular_includes_non_recursive.py

Test Type

CLI (subprocess) — runs pdd --force preprocess <file> as a real subprocess with a 10-second timeout. This is a true end-to-end test: real files on disk, real CLI invocation, no mocking.

What This Test Verifies

When a user runs pdd preprocess (default, non-recursive mode) on prompt files with circular <include> tags, the CLI must:

1. Detect the cycle and exit with non-zero exit code within seconds
2. Print "Circular include detected" in the output
3. NOT hang forever in an infinite loop

Test Code

"""
E2E Test for Issue #553: Circular <include> tags cause infinite loop in non-recursive preprocess.
"""

import subprocess
import sys
import pytest
from pathlib import Path

CLI_TIMEOUT = 10

def _run_pdd_preprocess(prompt_file: str, cwd: str, extra_args: list = None):
    cmd = [sys.executable, "-m", "pdd.cli", "--force", "preprocess", prompt_file]
    if extra_args:
        cmd.extend(extra_args)
    return subprocess.run(cmd, capture_output=True, text=True, cwd=cwd, timeout=CLI_TIMEOUT)

class TestIssue553CircularIncludesNonRecursiveE2E:

    def test_mutual_circular_includes_default_mode(self, tmp_path):
        """A→B→A circular include, default mode — must error, not hang."""
        a_file = tmp_path / "a_python.prompt"
        b_file = tmp_path / "b_python.prompt"
        a_file.write_text("Hello\n<include>b_python.prompt</include>\n")
        b_file.write_text("World\n<include>a_python.prompt</include>\n")
        try:
            result = _run_pdd_preprocess(str(a_file), cwd=str(tmp_path))
        except subprocess.TimeoutExpired:
            pytest.fail(f"pdd preprocess hung for {CLI_TIMEOUT}s — infinite loop (bug #553)")
        assert result.returncode != 0
        assert "circular include" in (result.stdout + result.stderr).lower()

    def test_self_referencing_include_default_mode(self, tmp_path):
        """Self-include must be detected in non-recursive mode."""
        # ... (self-referencing file, same pattern)

    def test_three_file_cycle_default_mode(self, tmp_path):
        """A→B→C→A cycle must be detected in non-recursive mode."""
        # ... (3-file chain, same pattern)

    def test_backtick_circular_includes_default_mode(self, tmp_path):
        """Backtick-style circular includes must also be detected."""
        # ... (backtick syntax, same pattern)

    def test_non_circular_includes_still_work(self, tmp_path):
        """Regression guard: A→B→C (no cycle) must still work."""
        # ... asserts exit code 0, output contains all content

    def test_recursive_mode_still_detects_cycle(self, tmp_path):
        """Regression guard: --recursive mode still catches cycles."""
        # ... asserts exit code != 0

Test Execution Result

$ pytest tests/test_e2e_issue_553_circular_includes_non_recursive.py -v --timeout=120

tests/...::test_mutual_circular_includes_default_mode   FAILED  [16%]
tests/...::test_self_referencing_include_default_mode    FAILED  [33%]
tests/...::test_three_file_cycle_default_mode            FAILED  [50%]
tests/...::test_backtick_circular_includes_default_mode  FAILED  [66%]
tests/...::test_non_circular_includes_still_work         PASSED  [83%]
tests/...::test_recursive_mode_still_detects_cycle       PASSED  [100%]

Failure reasons (all 4 circular tests):
  subprocess.TimeoutExpired: Command timed out after 10 seconds
  → pytest.fail("pdd preprocess hung for 10s — infinite loop detected (bug #553)")

4 failed, 2 passed in 44.47s

All 4 circular-include tests fail on the buggy code because the subprocess hangs forever (infinite loop in the non-recursive convergence while loop). The 2 regression guards pass: non-circular includes work, and --recursive mode correctly detects cycles.

E2E Test Status

PASS: E2E test correctly detects the bug

Test	Result	Reason
test_mutual_circular_includes_default_mode	FAILED (timeout)	Infinite loop — bug confirmed
test_self_referencing_include_default_mode	FAILED (timeout)	Infinite loop — bug confirmed
test_three_file_cycle_default_mode	FAILED (timeout)	Infinite loop — bug confirmed
test_backtick_circular_includes_default_mode	FAILED (timeout)	Infinite loop — bug confirmed
test_non_circular_includes_still_work	PASSED	Regression guard OK
test_recursive_mode_still_detects_cycle	PASSED	Regression guard OK

---

Proceeding to Step 10: Create Draft PR

[Serhan-Asad]

Step 10: Draft PR Created

Pull Request

PR #554: test: add failing tests for circular include detection in non-recursive mode (#553)

Branch

fix/issue-553

What's Included

• Failing unit tests at tests/test_circular_includes.py (6 new tests in TestCircularIncludesNonRecursive)
• Failing E2E tests at tests/test_e2e_issue_553_circular_includes_non_recursive.py (6 subprocess-based CLI tests)
• Commits: 1

Test Summary

Type	Circular Tests (should fail)	Regression Guards (should pass)
Unit	4 FAILED (timeout — infinite loop)	2 PASSED
E2E	4 FAILED (timeout — infinite loop)	2 PASSED

Next Steps for Maintainers

1. Review the failing tests to understand the expected behavior
2. Implement the fix at the identified location (preprocess.py — extend _seen tracking into non-recursive path)
3. Verify both unit and E2E tests pass with your fix
4. Run full test suite to check for regressions
5. Mark the PR as ready for review

PDD Fix Command

To auto-fix this bug using PDD:

pdd fix https://github.com/promptdriven/pdd/issues/553

Tip: Use --protect-tests if the tests are known to be correct, or --max-cycles N to limit fix attempts.

---

Investigation complete. A draft PR with failing tests has been created and linked to this issue.

[Serhan-Asad]

Step 2: E2E Test Check (Cycle 1)

Status: 4 Tests Failing

E2E Tests Run

• ✗
• ✗
• ✗
• ✗
• ✓
• ✓

Failures

---

Proceeding to Step 3: Root Cause Analysis

[Serhan-Asad]

Step 2: E2E Test Check (Cycle 1)

Status: 4 Tests Failing

E2E Tests Run

• tests/test_e2e_issue_553_circular_includes_non_recursive.py::TestIssue553CircularIncludesNonRecursiveE2E::test_mutual_circular_includes_default_mode ✗
• tests/test_e2e_issue_553_circular_includes_non_recursive.py::TestIssue553CircularIncludesNonRecursiveE2E::test_self_referencing_include_default_mode ✗
• tests/test_e2e_issue_553_circular_includes_non_recursive.py::TestIssue553CircularIncludesNonRecursiveE2E::test_three_file_cycle_default_mode ✗
• tests/test_e2e_issue_553_circular_includes_non_recursive.py::TestIssue553CircularIncludesNonRecursiveE2E::test_backtick_circular_includes_default_mode ✗
• tests/test_e2e_issue_553_circular_includes_non_recursive.py::TestIssue553CircularIncludesNonRecursiveE2E::test_non_circular_includes_still_work ✓
• tests/test_e2e_issue_553_circular_includes_non_recursive.py::TestIssue553CircularIncludesNonRecursiveE2E::test_recursive_mode_still_detects_cycle ✓

Failures

test_mutual_circular_includes_default_mode

subprocess.TimeoutExpired: Command '['/Library/Frameworks/Python.framework/Versions/3.13/bin/python3', '-m', 'pdd.cli', '--force', 'preprocess', '/private/var/folders/m7/03m2hb3j4_3df05ypw6v1rvm0000gn/T/pytest-of-serhanasad/pytest-100/test_mutual_circular_includes_0/a_python.prompt']' timed out after 10 seconds
Failed: pdd preprocess hung for 10s — infinite loop (bug #553)

test_self_referencing_include_default_mode

subprocess.TimeoutExpired: Command '['/Library/Frameworks/Python.framework/Versions/3.13/bin/python3', '-m', 'pdd.cli', '--force', 'preprocess', '/private/var/folders/m7/03m2hb3j4_3df05ypw6v1rvm0000gn/T/pytest-of-serhanasad/pytest-100/test_self_referencing_include_0/self_python.prompt']' timed out after 10 seconds
Failed: pdd preprocess hung for 10s — infinite loop (bug #553)

test_three_file_cycle_default_mode

subprocess.TimeoutExpired: Command '['/Library/Frameworks/Python.framework/Versions/3.13/bin/python3', '-m', 'pdd.cli', '--force', 'preprocess', '/private/var/folders/m7/03m2hb3j4_3df05ypw6v1rvm0000gn/T/pytest-of-serhanasad/pytest-100/test_three_file_cycle_default_0/a_python.prompt']' timed out after 10 seconds
Failed: pdd preprocess hung for 10s — infinite loop (bug #553)

test_backtick_circular_includes_default_mode

subprocess.TimeoutExpired: Command '['/Library/Frameworks/Python.framework/Versions/3.13/bin/python3', '-m', 'pdd.cli', '--force', 'preprocess', '/private/var/folders/m7/03m2hb3j4_3df05ypw6v1rvm0000gn/T/pytest-of-serhanasad/pytest-100/test_backtick_circular_include0/x_python.prompt']' timed out after 10 seconds
Failed: pdd preprocess hung for 10s — infinite loop (bug #553)

---

Proceeding to Step 3: Root Cause Analysis