Groq structured output path mutates shared formatted_messages, corrupting fallback models

[Serhan-Asad]

Bug Description

When a Groq model is selected as the first candidate and structured output (output_pydantic) is requested, the Groq code path in llm_invoke.py mutates the shared formatted_messages list in-place. If the Groq call fails and falls back to another model, the fallback model receives corrupted messages containing Groq's JSON schema instruction as an injected system message.

Root Cause

llm_invoke.py:1966 passes formatted_messages by reference (not copy):

litellm_kwargs: Dict[str, Any] = {
    "model": model_name_litellm,
    "messages": formatted_messages,  # <-- REFERENCE to shared list
}

Then llm_invoke.py:2125-2129 mutates the list in-place:

messages_list = litellm_kwargs.get("messages", [])  # same reference
if messages_list and messages_list[0].get("role") == "system":
    messages_list[0]["content"] = schema_instruction + "\n\n" + messages_list[0]["content"]
else:
    messages_list.insert(0, {"role": "system", "content": schema_instruction})

Since messages_list, litellm_kwargs["messages"], and formatted_messages all point to the same list object, the .insert(0, ...) mutates formatted_messages permanently. Every subsequent model candidate in the fallback loop receives the corrupted messages.

Reproduction

import os, sys, json
sys.path.insert(0, "/path/to/pdd")
os.environ["GROQ_API_KEY"] = "your-key"

from pydantic import BaseModel
import litellm
from pdd.llm_invoke import llm_invoke

# Add groq model with high Elo to ~/.pdd/llm_model.csv:
# Groq,groq/nonexistent-model,0.01,0.01,1500,,GROQ_API_KEY,0,True,none,

class SimpleResult(BaseModel):
    answer: str
    confidence: float

# Monkey-patch to inspect messages
_orig = litellm.completion
def spy(**kw):
    model = kw.get("model")
    msgs = kw.get("messages", [])
    for m in msgs:
        if "You must respond with valid JSON" in m.get("content", ""):
            print(f"BUG: {model} received Groq's JSON schema instruction!")
    return _orig(**kw)
litellm.completion = spy

result = llm_invoke(
    messages=[{"role": "user", "content": "What is 2 + 2?"}],
    strength=1.0, temperature=0.0, time=0.0,
    output_pydantic=SimpleResult,
)

Output:

BUG: groq/nonexistent-model received Groq's JSON schema instruction!
BUG: gpt-4o-mini received Groq's JSON schema instruction!    <-- CORRUPTED

Impact

• Fallback models receive redundant/conflicting JSON schema instructions (once in system message, once in response_format API parameter)
• Extra token cost on every fallback attempt (177 tokens vs ~20 in test case)
• Potential for confused model responses from conflicting instructions
• Silent corruption — no error is raised, results may just be subtly wrong
• Every subsequent model in the fallback chain inherits the corruption

Suggested Fix

Deep copy formatted_messages when building litellm_kwargs at line 1966:

import copy

litellm_kwargs: Dict[str, Any] = {
    "model": model_name_litellm,
    "messages": copy.deepcopy(formatted_messages),  # isolate per-model
}

Or more targeted: deep copy only in the Groq path before mutating.

Environment

• PDD version: 0.0.145
• File: pdd/llm_invoke.py, lines 1966, 2125-2129
• Affects any command using output_pydantic with Groq as a candidate model

[Serhan-Asad]

Step 1: Duplicate Check

Status: No duplicates found

Search Performed

• Searched for: "Groq structured output", "formatted_messages mutate", "fallback corrupted messages", "llm_invoke shared reference", "Groq fallback", "output_pydantic", "deepcopy messages", "schema_instruction", "in-place mutation fallback", "JSON schema instruction system message", "litellm_kwargs messages copy", title keywords (mutate, corrupt, messages)
• Issues reviewed: 8 candidates examined in detail

Findings

No existing issue covers this bug. Two related issues in llm_invoke.py were reviewed but have different root causes:

• Race Condition in LLM Cost Tracking Causes Data Corruption #375 (Open) — Race condition in _LAST_CALLBACK_DATA global dict for cost tracking in concurrent server mode. Different root cause: shared global state across threads, not message list mutation in fallback loop.
• Cost under-reporting: retry calls overwrite original call cost in _LAST_CALLBACK_DATA #509 (Closed) — Cost under-reporting when retry calls overwrite _LAST_CALLBACK_DATA["cost"]. Different root cause: cost overwrite on retry, not message corruption across model candidates.

Issue #562 is unique: it describes an in-place mutation of formatted_messages in the Groq structured output code path (lines 2125-2129) that corrupts the shared message list for all subsequent fallback model candidates. This is a distinct bug pattern (aliased mutable data structure in a loop) unrelated to cost tracking or threading issues.

---

Proceeding to Step 2: Documentation Check

[Serhan-Asad]

Step 2: Documentation Check

Status: Confirmed Bug

Documentation Reviewed

• README.md — No mention of message mutation, Groq structured output behavior, or fallback corruption
• CONTRIBUTING.md — No warnings about shared message state or mutation constraints
• CHANGELOG.md (v0.0.150–v0.0.157) — Related fixes (multi-credential handling, cost tracking) but no entry for this issue
• pdd/llm_invoke.py docstring (lines 1727–1769) — Documents fallback model iteration and structured output support but does not mention mutation side effects
• pdd/llm_invoke.py inline comments (lines 2110–2130) — Comment says "Groq has issues with tool-based structured output - use JSON mode with schema in prompt" but does not acknowledge or warn about fallback corruption
• No KNOWN_ISSUES file exists in the repository
• No test files cover Groq structured output with fallback models (no *groq* or *structured* test files found)

Findings

The reported behavior contradicts the intended design of the fallback model system. The code is structured so that formatted_messages is shared across all candidate models in the fallback loop (lines 1941–2960). The Groq structured output path at lines 2125–2130 mutates this shared list in-place via .insert() and direct content string modification, permanently corrupting messages for all subsequent fallback candidates.

Key evidence this is a bug, not expected behavior:

1. No documentation warns about it — None of the documentation files (README, CONTRIBUTING, CHANGELOG, docstrings, inline comments) mention that Groq structured output may corrupt fallback model messages
2. No tests validate it — No unit or integration tests cover Groq structured output with fallback, suggesting this side effect was not anticipated
3. The fallback loop design assumes immutability — The outer loop iterates candidate_models and rebuilds litellm_kwargs each iteration (line 1966), but reuses the same formatted_messages reference, implying the messages should remain unchanged between iterations
4. The Groq comment describes a workaround, not intentional mutation — The comment "Groq has issues with tool-based structured output" describes why JSON mode is used, not why message corruption is acceptable
5. No deep copy or mutation guard exists — There is no .deepcopy(), no list cloning, and no comment warning maintainers not to mutate the shared list

The fix suggested in the issue (deep copy at line 1966 or before the Groq mutation block) is appropriate and low-risk.

---

Proceeding to Step 3: Triage

[Serhan-Asad]

Step 3: Triage

Status: Proceed

Summary

This issue has all the information needed to attempt reproduction:

• Expected behavior is clear: when a Groq model fails and falls back to another model, the fallback should receive the original, unmodified formatted_messages.
• Actual behavior is clear: the Groq structured output code path mutates formatted_messages in-place (inserting a JSON schema system message), and since litellm_kwargs["messages"] is a reference to the same list object, the mutation persists across all subsequent fallback candidates.
• Root cause is verified in source code:
  • Line 1966: "messages": formatted_messages passes by reference (not copy)
  • Lines 2125-2129: .insert(0, ...) and content prepending mutate the shared list in-place
• Reproduction path is provided with a concrete script, and the bug is also reproducible via unit test by verifying list identity before/after the Groq path executes.

No additional information is needed from the issue author.

---

Proceeding to Step 4: Reproduction

[Serhan-Asad]

Step 4: Reproduction

Status: Reproduced

Environment

• macOS Darwin 24.3.0
• Python 3.12.7
• pytest 8.3.5
• PDD repo: promptdriven/pdd @ commit d9ab3809

Steps Taken

1. Verified affected code — Confirmed the shared reference at llm_invoke.py:1966 and the in-place mutation at lines 2125-2129 match the issue description exactly.

2. Wrote 3 targeted reproduction tests that directly exercise the buggy code path without requiring API keys or network calls:

   • test_groq_structured_output_mutates_shared_messages — Proves litellm_kwargs["messages"] is the same object as formatted_messages (identity check via is), and that the Groq .insert(0, ...) mutates the shared list.
   • test_groq_mutation_corrupts_fallback_model — Simulates the full fallback scenario: Groq model mutates messages, then a second model candidate receives the corrupted list with the injected system message.
   • test_groq_mutation_with_existing_system_message — Covers the branch at line 2126-2127 where an existing system message's content is mutated in-place (dict mutation, not just list mutation).

3. Ran tests — All 3 pass, confirming the bug:

tests/test_groq_message_mutation_bug.py::test_groq_structured_output_mutates_shared_messages PASSED
tests/test_groq_message_mutation_bug.py::test_groq_mutation_corrupts_fallback_model PASSED
tests/test_groq_message_mutation_bug.py::test_groq_mutation_with_existing_system_message PASSED

3 passed in 0.33s

Observed Behavior

The bug manifests in two forms:

Form 1 — No existing system message (line 2129): .insert(0, {...}) adds a new system message to formatted_messages. Subsequent models receive [groq_schema_msg, user_msg] instead of [user_msg].

Form 2 — Existing system message (line 2127): The dict's content key is overwritten with schema_instruction + "\n\n" + original_content. Since dict mutation affects all references, subsequent models see the corrupted system message with Groq's JSON schema prepended.

Both forms permanently corrupt formatted_messages for every remaining model in the fallback loop.

Affected Files

• pdd/llm_invoke.py:1966 — formatted_messages passed by reference into litellm_kwargs
• pdd/llm_invoke.py:2125-2129 — Groq structured output workaround mutates the shared list/dicts in-place

Notes

• The reproduction does not require any API keys or network calls — the bug is a pure Python reference/mutation issue that can be verified with object identity checks.
• The bug affects both mutation paths: list .insert() (no system message) and dict content overwrite (existing system message).
• Test file: tests/test_groq_message_mutation_bug.py

---

Proceeding to Step 5: Root Cause Analysis

[Serhan-Asad]

Step 5: Root Cause Analysis

Root Cause Identified: Yes

Summary

When the fallback loop in llm_invoke() iterates through candidate models, line 1966 passes formatted_messages by reference into each iteration's litellm_kwargs. The Groq structured output workaround (lines 2125-2129) then mutates this shared reference in-place via two mechanisms: (1) messages_list.insert(0, ...) adds a system message to the shared list, and (2) messages_list[0]["content"] = schema_instruction + "\n\n" + ... overwrites dict values in-place. Since messages_list, litellm_kwargs["messages"], and formatted_messages all point to the same Python objects, every subsequent model candidate — and even the GPT-5 Responses API path at line 2251 — receives corrupted messages containing Groq's JSON schema instruction.

Technical Details

• Primary Location: pdd/llm_invoke.py:1966 — shared reference passed into per-model litellm_kwargs
• Mutation Location: pdd/llm_invoke.py:2125-2129 — Groq workaround mutates shared objects

Problematic Code (line 1966 — shared reference):

litellm_kwargs: Dict[str, Any] = {
    "model": model_name_litellm,
    "messages": formatted_messages,  # <-- REFERENCE to shared list
}

Mutation Point 1 — List-level (line 2129, no existing system message):

messages_list = litellm_kwargs.get("messages", [])  # same object as formatted_messages
messages_list.insert(0, {"role": "system", "content": schema_instruction})

Mutation Point 2 — Dict-level (line 2127, existing system message):

messages_list[0]["content"] = schema_instruction + "\n\n" + messages_list[0]["content"]

• Why it fails: Python passes mutable objects (lists, dicts) by reference. litellm_kwargs.get("messages") returns the same list object as formatted_messages. The .insert() call mutates the list in-place, and the dict content overwrite mutates the dict in-place. Both are visible to all references.

• Additional impact: Line 2251 also reads formatted_messages directly for the GPT-5 Responses API path. If Groq was tried first and failed, this path would also see corrupted messages.

External Research

• Search terms: "litellm groq structured output json_object messages mutation fallback bug"
• Sources checked: GitHub issues (litellm, openai-agents-python, strands-agents), LiteLLM docs, Groq docs
• Relevant findings:
  • BerriAI/litellm#15761 — Groq tool call and structured response issues confirming why PDD uses the json_object workaround
  • openai/openai-agents-python#2140 — Groq doesn't support tool_choice: "json_tool_call", confirming the need for the workaround
  • The Groq workaround itself is sound — the bug is specifically that it mutates shared state instead of a local copy

Repository History

• Introducing commit: 1f4de919e (Greg Tanaka, 2025-12-18) — "Bump version" commit that added the Groq structured output workaround to llm_invoke.py
• Parent commit for litellm_kwargs reference: 0804e2a3b (Greg Tanaka, 2025-09-01) — original litellm_kwargs construction that passes formatted_messages by reference
• Related PRs: No standalone PR for the Groq workaround was found; it was bundled into the version bump commit
• Similar past issues:
  • Issue Race Condition in LLM Cost Tracking Causes Data Corruption #375 — Race condition in _LAST_CALLBACK_DATA (threading, not message mutation — different root cause)
  • Issue Cost under-reporting: retry calls overwrite original call cost in _LAST_CALLBACK_DATA #509 — Cost under-reporting on retries (cost overwrite, not message corruption — different root cause)
  • No prior issues about message mutation in the fallback loop

Experiments Performed

1. Object identity verification: Confirmed litellm_kwargs.get("messages") returns the exact same object (id() match) as formatted_messages — all three variables point to a single list.

2. List-level mutation (no system message): When formatted_messages has no system message, .insert(0, ...) adds a Groq schema system message to the shared list. Subsequent models see len(formatted_messages) increase by 1 and find the Groq instruction at index 0.

3. Dict-level mutation (existing system message): When formatted_messages[0] already has role: "system", the content overwrite messages_list[0]["content"] = ... mutates the shared dict in-place. This is worse because even a shallow copy (list[:] or list.copy()) would NOT prevent it — inner dicts are still shared references.

4. Shallow copy insufficiency: Verified that formatted_messages[:] (shallow copy) still allows dict-level mutation. Only copy.deepcopy() prevents both mutation vectors.

5. GPT-5 Responses API contamination: Verified that line 2251's use of formatted_messages would also see corrupted messages if Groq was attempted first.

6. Fix verification: Confirmed copy.deepcopy(formatted_messages) at line 1966 completely isolates each model candidate's messages from the shared state, preventing both list-level and dict-level mutation.

Fix Location

• File to modify: pdd/llm_invoke.py
• Line 1966 — the root cause (shared reference passed into per-model kwargs)
• Why this location: This is where the shared reference is created. Fixing here (rather than in the Groq path) protects against any future provider-specific workarounds that might also mutate messages. The Groq code at lines 2125-2129 is where symptoms appear, but the root cause is the assumption that formatted_messages won't be mutated by provider-specific code paths.
• Existing pattern: The LM Studio path (lines 2090-2108) avoids this bug only because it modifies litellm_kwargs["response_format"] (a separate key) rather than mutating the messages list. There is no existing deepcopy pattern in the codebase — copy is not imported anywhere in pdd/.
• Fix approach: Add import copy at file top, change line 1966 from "messages": formatted_messages to "messages": copy.deepcopy(formatted_messages). This isolates each model candidate's messages, costing ~negligible overhead for typical message sizes (a few dicts). Alternative: targeted deepcopy only in the Groq path before mutation, but the broader fix is more robust.

---

Proceeding to Step 5.5: Prompt Classification

[Serhan-Asad]

Step 5.5: Prompt Classification

Classification: Code Bug

DEFECT_TYPE: code

Analysis

This is a code implementation bug, not a prompt specification defect. The Groq structured output workaround at llm_invoke.py:2125-2129 inadvertently mutates the shared formatted_messages list in-place due to Python reference semantics, corrupting messages for all subsequent fallback model candidates.

No llm_invoke_python.prompt file exists in this repository (the prompt likely resides in pdd_cap). However, this classification does not depend on reading the prompt — the defect is unambiguously an implementation error:

Evidence

• Prompt specifies (inferred from design): The fallback loop iterates over model candidates, rebuilding litellm_kwargs each iteration. This design clearly intends each model candidate to receive clean, independent state. Any reasonable prompt specification would require "try models in order, falling back on failure" without cross-contamination.

• Code implements: Line 1966 assigns formatted_messages by reference into litellm_kwargs["messages"]. The Groq workaround at lines 2125-2129 then mutates this shared reference via .insert(0, ...) (list-level) and content overwrite (dict-level), permanently corrupting formatted_messages for all subsequent loop iterations.

• User expects: Each fallback model receives the original, unmodified messages — which matches the intended design. The mutation is an unintended side effect of the Groq workaround added in commit 1f4de919e (Dec 18, 2025), not a deliberate design choice.

Why Not a Prompt Defect

1. Implementation-level issue: This is a Python reference semantics bug (pass-by-reference vs deep copy), not a requirements-level issue
2. Unintended side effect: The Groq workaround was added to fix structured output for Groq models; the mutation of shared state was accidental, not specified
3. Design contradicts behavior: The loop rebuilds litellm_kwargs each iteration, demonstrating the intent to isolate per-model state — the bug violates this intent
4. No specification for mutation: No prompt would specify "corrupt messages across fallback iterations"

Conclusion

The code does not correctly implement the fallback isolation that the design clearly intends. Proceeding with test generation.

---

Proceeding to Step 6: Test Plan

[Serhan-Asad]

Step 6: Test Plan

Existing Test Coverage

• Test file: tests/test_llm_invoke.py — Main unit tests for llm_invoke(). Covers input validation, model selection by strength, prompt formatting, Pydantic output parsing, debug suppression, and basic invocation flow. Uses mock_load_models fixture with namedtuple-based DataFrame and mocks litellm.completion.
• Test file: tests/test_llm_invoke_retry_cost.py — Tests for issue Cost under-reporting: retry calls overwrite original call cost in _LAST_CALLBACK_DATA #509 (retry cost accumulation). Uses _select_model_candidates mock + litellm mock pattern. Closest pattern to what we need (tests fallback/retry behavior).
• Test file: tests/test_llm_invoke_nested_schema.py — Tests for _add_additional_properties_false and _ensure_all_properties_required helper functions.
• Test file: tests/test_llm_invoke_vertex_retry.py — Tests for Vertex AI retry credential handling.
• Gap: No tests cover the Groq structured output code path (lines 2110-2133). No tests verify that formatted_messages remains unmodified after the Groq workaround executes. No tests verify message integrity across fallback model candidates in the for-loop.

Proposed Tests

Test 1: test_groq_structured_output_does_not_mutate_formatted_messages

• Input: Two model candidates: Groq model (first, fails with exception) + OpenAI model (second, succeeds). output_pydantic set to a Pydantic model. Messages list with a single user message [{"role": "user", "content": "What is 2+2?"}].
• Expected: The second model (OpenAI) receives the original messages — [{"role": "user", "content": "What is 2+2?"}] — without any Groq JSON schema instruction prepended.
• Actual (before fix): The second model receives corrupted messages with an extra system message containing "You must respond with valid JSON matching this schema..." prepended by the Groq code path at line 2129.

Test 2: test_groq_structured_output_does_not_mutate_existing_system_message

• Input: Same as Test 1 but messages start with a system message: [{"role": "system", "content": "You are helpful."}, {"role": "user", "content": "Hi"}]. Groq model fails, falls back to second model.
• Expected: The fallback model receives the original system message content "You are helpful." unmodified.
• Actual (before fix): The fallback model receives a system message with "You must respond with valid JSON...\n\nYou are helpful." because line 2127 overwrites the dict's content key in-place via the shared reference.

Test 3: test_groq_structured_output_no_schema_in_non_groq_model

• Input: Same two-model setup. After Groq fails, capture the litellm.completion kwargs for the second (non-Groq) call.
• Expected: No message in the second call's messages list contains the substring "You must respond with valid JSON matching this schema".
• Actual (before fix): The first message in the second call contains the Groq-specific schema instruction.

Test 4: test_groq_structured_output_messages_list_length_preserved

• Input: Messages with only a user message (no system message). Two candidates: Groq (fails) + fallback.
• Expected: The fallback model's messages list has length 1 (just the user message).
• Actual (before fix): The fallback model's messages list has length 2 because line 2129 inserts a new system message via .insert(0, ...) into the shared list.

Test 5: test_multiple_groq_fallbacks_no_cumulative_corruption

• Input: Three model candidates: Groq pdd install_completion: install_completion() got unexpected keyword argument 'quiet' #1 (fails), Groq LiteLLM: AttributeError: 'Cache' object has no attribute 'cache' #2 (fails), non-Groq Local: How to prevent pdd from trying LLMs where no API Key exists? #3 (succeeds). output_pydantic set.
• Expected: The third model receives the original messages unmodified. Schema instructions should not accumulate.
• Actual (before fix): The third model receives messages with two Groq schema instructions prepended (one from each failed Groq attempt), doubling the corruption.

Test Location

• File: tests/test_llm_invoke_groq_message_mutation.py (new file)
• Framework: pytest with unittest.mock (patch, MagicMock, patch.dict)
• Rationale: New file follows the project convention of separate test files per issue/feature (e.g., test_llm_invoke_retry_cost.py for Cost under-reporting: retry calls overwrite original call cost in _LAST_CALLBACK_DATA #509, test_llm_invoke_nested_schema.py for schema issues, test_llm_invoke_vertex_retry.py for [core-dump] generate failed on Darwin #185). The bug is self-contained and the tests require specific Groq-focused fixtures.

Notes

Test Pattern: Follow test_llm_invoke_retry_cost.py pattern:

• Mock _select_model_candidates to return specific model dicts (Groq first, OpenAI second)
• Mock _load_model_data to return a DataFrame
• Mock _ensure_api_key to return True
• Mock litellm.completion with a side_effect that raises on the first call (Groq) and succeeds on the second (fallback)
• Use patch.dict(os.environ, {"PDD_FORCE_LOCAL": "1", "GROQ_API_KEY": "test", "OPENAI_API_KEY": "test"})

Key assertions: Capture litellm.completion call args for each invocation and inspect the messages kwarg. The second call's messages must match the original input exactly (same length, same content, no schema instruction strings).

Model dict fields (matching CSV columns):

groq_model = {
    "provider": "Groq", "model": "groq/llama-70b",
    "input": 0.01, "output": 0.01, "coding_arena_elo": 1500,
    "structured_output": True, "base_url": "", "api_key": "GROQ_API_KEY",
    "max_tokens": "", "max_completion_tokens": "",
    "reasoning_type": "none", "max_reasoning_tokens": 0,
}
openai_model = {
    "provider": "OpenAI", "model": "gpt-4o-mini",
    "input": 0.02, "output": 0.03, "coding_arena_elo": 1400,
    "structured_output": True, "base_url": "", "api_key": "OPENAI_API_KEY",
    "max_tokens": "", "max_completion_tokens": "",
    "reasoning_type": "none", "max_reasoning_tokens": 0,
}

No real API calls needed: Tests are pure unit tests mocking litellm.completion. The Groq call should raise (e.g., openai.APIConnectionError) to trigger the fallback loop.

---

Proceeding to Step 7: Generate Test

[Serhan-Asad]

Step 7: Generated Test

Test File

tests/test_llm_invoke_groq_message_mutation.py

Test Code

"""Tests for issue #562: Groq structured output path mutates shared formatted_messages,
corrupting fallback models.

When a Groq model is the first candidate and structured output (output_pydantic) is
requested, the Groq code path in llm_invoke.py mutates the shared formatted_messages
list in-place. If the Groq call fails and falls back to another model, the fallback
model receives corrupted messages containing Groq's JSON schema instruction.

Root cause: llm_invoke.py:1966 passes formatted_messages by reference (not copy) into
litellm_kwargs, and lines 2125-2129 mutate the list in-place via .insert() and dict
content overwrite.
"""

import copy
import json
import os
import pytest
import pandas as pd
from unittest.mock import patch, MagicMock
from pydantic import BaseModel

import pdd.llm_invoke as _llm_mod


# --- Pydantic model for structured output ---

class SimpleResult(BaseModel):
    answer: str
    confidence: float


# --- Helpers (following test_llm_invoke_retry_cost.py conventions) ---

def _make_groq_model():
    """Create a mock Groq model dict with structured_output enabled."""
    return {
        "provider": "Groq",
        "model": "groq/llama-3.3-70b-versatile",
        "input": 0.59,
        "output": 0.79,
        "coding_arena_elo": 1200,
        "structured_output": True,
        "base_url": "",
        "api_key": "GROQ_API_KEY",
        "max_tokens": "",
        "max_completion_tokens": "",
        "reasoning_type": "none",
        "max_reasoning_tokens": 0,
    }


def _make_openai_model():
    """Create a mock OpenAI fallback model dict."""
    return {
        "provider": "OpenAI",
        "model": "gpt-4o-mini",
        "input": 0.15,
        "output": 0.60,
        "coding_arena_elo": 1100,
        "structured_output": True,
        "base_url": "",
        "api_key": "OPENAI_API_KEY",
        "max_tokens": "",
        "max_completion_tokens": "",
        "reasoning_type": "none",
        "max_reasoning_tokens": 0,
    }


def _make_response(content, prompt_tokens=100, completion_tokens=50):
    """Create a mock LiteLLM response."""
    resp = MagicMock()
    choice = MagicMock()
    choice.message.content = content
    choice.finish_reason = "stop"
    resp.choices = [choice]
    usage = MagicMock()
    usage.prompt_tokens = prompt_tokens
    usage.completion_tokens = completion_tokens
    resp.usage = usage
    return resp


# Groq's JSON schema instruction string (substring used for detection)
GROQ_SCHEMA_MARKER = "You must respond with valid JSON matching this schema"


@pytest.fixture(autouse=True)
def reset_callback_data():
    """Reset _llm_mod._LAST_CALLBACK_DATA before each test."""
    _llm_mod._LAST_CALLBACK_DATA["cost"] = 0.0
    _llm_mod._LAST_CALLBACK_DATA["input_tokens"] = 0
    _llm_mod._LAST_CALLBACK_DATA["output_tokens"] = 0
    _llm_mod._LAST_CALLBACK_DATA["finish_reason"] = None
    yield


class TestGroqMessageMutation:
    """Tests that the Groq structured output path does not corrupt shared messages."""

    @patch.dict(os.environ, {"PDD_FORCE_LOCAL": "1", "GROQ_API_KEY": "test-groq-key", "OPENAI_API_KEY": "test-openai-key"})
    @patch("pdd.llm_invoke._ensure_api_key", return_value=True)
    @patch("pdd.llm_invoke._select_model_candidates")
    @patch("pdd.llm_invoke._load_model_data")
    @patch("pdd.llm_invoke.litellm")
    def test_groq_structured_output_does_not_mutate_formatted_messages(
        self, mock_litellm, mock_load, mock_select, mock_ensure
    ):
        """Test 1: Groq structured output path must not mutate formatted_messages
        for fallback models (.insert() mutation vector)."""
        groq_model = _make_groq_model()
        openai_model = _make_openai_model()
        mock_load.return_value = pd.DataFrame([groq_model, openai_model])
        mock_select.return_value = [groq_model, openai_model]
        mock_litellm.cache = None
        mock_litellm.drop_params = True

        captured_calls = []

        def completion_side_effect(**kwargs):
            captured_calls.append(copy.deepcopy(kwargs))
            model = kwargs.get("model", "")
            if "groq/" in model:
                raise Exception("Groq API error: model not available")
            result = json.dumps({"answer": "4", "confidence": 0.99})
            resp = _make_response(content=result)
            _llm_mod._LAST_CALLBACK_DATA["cost"] = 0.001
            _llm_mod._LAST_CALLBACK_DATA["input_tokens"] = resp.usage.prompt_tokens
            _llm_mod._LAST_CALLBACK_DATA["output_tokens"] = resp.usage.completion_tokens
            return resp

        mock_litellm.completion = MagicMock(side_effect=completion_side_effect)

        from pdd.llm_invoke import llm_invoke
        result = llm_invoke(
            messages=[{"role": "user", "content": "What is 2 + 2?"}],
            strength=0.5, temperature=0.0, time=0.0,
            output_pydantic=SimpleResult, use_cloud=False,
        )

        assert len(captured_calls) >= 2
        fallback_messages = captured_calls[1]["messages"]
        for msg in fallback_messages:
            content = msg.get("content", "")
            assert GROQ_SCHEMA_MARKER not in content

    # Test 2: Existing system message content must not be overwritten
    # Test 3: No schema instruction in non-Groq model messages
    # Test 4: Messages list length must be preserved
    # Test 5: Multiple Groq failures must not accumulate schema instructions
    # (See full file for all 5 tests)

What This Test Verifies

5 test cases that detect the bug from two different mutation vectors:

Test	What it checks	Why it fails on buggy code
Test 1	Fallback model messages don't contain Groq schema instruction	.insert(0, ...) adds system message to shared formatted_messages list
Test 2	Existing system message content is preserved for fallback	messages_list[0]["content"] = schema + "\n\n" + ... overwrites dict in-place
Test 3	No Groq schema marker string in any fallback message	Both mutation vectors inject "You must respond with valid JSON..."
Test 4	Fallback messages list has original length (1, not 2)	.insert(0, ...) permanently grows the shared list
Test 5	Multiple Groq failures don't accumulate schema instructions	Each failed Groq candidate adds another schema instruction to the shared list

All 5 tests FAIL on the current buggy code:

FAILED test_groq_structured_output_does_not_mutate_formatted_messages - Fallback received Groq's JSON schema instruction
FAILED test_groq_structured_output_does_not_mutate_existing_system_message - System message content was overwritten
FAILED test_groq_structured_output_no_schema_in_non_groq_model - Non-Groq model received schema instruction
FAILED test_groq_structured_output_messages_list_length_preserved - Messages list grew from 1 to 2
FAILED test_multiple_groq_fallbacks_no_cumulative_corruption - Schema instructions accumulated across failed Groq candidates

Running the Test

pytest tests/test_llm_invoke_groq_message_mutation.py -v

---

Proceeding to Step 8: Verification

[Serhan-Asad]

Step 8: Verification Complete

Test Execution

============================= test session starts ==============================
platform darwin -- Python 3.12.7, pytest-8.3.5, pluggy-1.6.0
collected 5 items

tests/test_llm_invoke_groq_message_mutation.py::TestGroqMessageMutation::test_groq_structured_output_does_not_mutate_formatted_messages FAILED [ 20%]
tests/test_llm_invoke_groq_message_mutation.py::TestGroqMessageMutation::test_groq_structured_output_does_not_mutate_existing_system_message FAILED [ 40%]
tests/test_llm_invoke_groq_message_mutation.py::TestGroqMessageMutation::test_groq_structured_output_no_schema_in_non_groq_model FAILED [ 60%]
tests/test_llm_invoke_groq_message_mutation.py::TestGroqMessageMutation::test_groq_structured_output_messages_list_length_preserved FAILED [ 80%]
tests/test_llm_invoke_groq_message_mutation.py::TestGroqMessageMutation::test_multiple_groq_fallbacks_no_cumulative_corruption FAILED [100%]

FAILURES:
Test 1: "BUG: Fallback model received Groq's JSON schema instruction in message
         with role='system'. The Groq code path mutated the shared formatted_messages list."

Test 2: "BUG: Fallback model's system message was mutated.
         Expected: 'You are a helpful math tutor.',
         Got: 'You must respond with valid JSON matching this schema:...'
         The Groq code path overwrote the dict's content in-place."

Test 3: "BUG: Non-Groq fallback model received Groq's JSON schema instruction."

Test 4: "BUG: Fallback model received 2 messages, expected 1.
         The Groq code path inserted an extra system message via .insert(0, ...)
         into the shared formatted_messages list."

Test 5: "BUG: Final fallback model received 1 Groq schema instruction(s).
         Schema instructions accumulated across multiple failed Groq candidates."

5 failed in ~4s

Verification Status

PASS: Test correctly detects the bug

All 5 tests fail for the right reason — the Groq structured output path at llm_invoke.py:2125-2129 mutates the shared formatted_messages list in-place, corrupting messages for fallback models. Each test targets a different facet of the mutation:

Test	What It Checks	Failure Confirms
Test 1	.insert() mutation vector	Groq's schema system message leaks into fallback
Test 2	Dict content overwrite vector	Existing system message content is permanently mutated
Test 3	Schema marker in any message	Schema instruction string appears in non-Groq model's messages
Test 4	Message list length preservation	Extra system message injected via .insert(0, ...)
Test 5	Cumulative corruption across multiple Groq failures	Schema instructions accumulate with each failed Groq candidate

Summary

Step	Result
Duplicate Check	No duplicates found
Documentation	No documentation covers this behavior — confirmed unintentional
Triage	Proceed — all info available for reproduction
Reproduction	Reproduced — pure Python reference/mutation issue
Root Cause	formatted_messages passed by reference at line 1966, mutated in-place at lines 2125-2129
Prompt Classification	Code Bug (DEFECT_TYPE: code)
Test Plan	5 test cases covering both mutation vectors + cumulative corruption
Test Generation	5 tests written in tests/test_llm_invoke_groq_message_mutation.py
Verification	All 5 tests FAIL on buggy code, confirming bug detection

Bug Details

• Location: pdd/llm_invoke.py:1966 (shared reference) and pdd/llm_invoke.py:2125-2129 (in-place mutation)
• Root Cause: The Groq structured output workaround mutates the shared formatted_messages list in-place via .insert(0, ...) and dict content overwrite, corrupting all subsequent fallback model candidates
• Test File: tests/test_llm_invoke_groq_message_mutation.py
• Fix: copy.deepcopy(formatted_messages) at line 1966 to isolate each model candidate's messages

Next Steps

1. Fix the bug at the identified location (llm_invoke.py:1966)
2. Run the test to confirm the fix (pytest tests/test_llm_invoke_groq_message_mutation.py -v)
3. Run full test suite to check for regressions (pytest tests/ -m "not integration and not e2e and not real" --timeout=60)
4. Submit PR with fix and test

---

Proceeding to Step 9: E2E Tests

[Serhan-Asad]

Step 3: Triage

Status: Proceed

Summary

This issue has all the information needed to attempt reproduction:

• Expected behavior is clear: when a Groq model fails and falls back to another model, the fallback should receive the original, unmodified formatted_messages.
• Actual behavior is clear: the Groq structured output code path mutates formatted_messages in-place (inserting a JSON schema system message), and since litellm_kwargs["messages"] is a reference to the same list object, the mutation persists across all subsequent fallback candidates.
• Root cause is verified in source code:
  • Line 1966: "messages": formatted_messages passes by reference (not copy)
  • Lines 2125-2129: .insert(0, ...) and content prepending mutate the shared list in-place
• Reproduction path is provided with a concrete script, and the bug is also reproducible via unit test by verifying list identity before/after the Groq path executes.

No additional information is needed from the issue author.

---

Proceeding to Step 4: Reproduction

[Serhan-Asad]

Step 4: Reproduction

Status: Reproduced

Environment

• macOS Darwin 24.3.0
• Python 3.12.7
• pytest 8.3.5
• PDD repo: promptdriven/pdd @ commit d9ab3809

Steps Taken

1. Verified affected code — Confirmed the shared reference at llm_invoke.py:1966 and the in-place mutation at lines 2125-2129 match the issue description exactly.

2. Wrote 3 targeted reproduction tests that directly exercise the buggy code path without requiring API keys or network calls:

   • test_groq_structured_output_mutates_shared_messages — Proves litellm_kwargs["messages"] is the same object as formatted_messages (identity check via is), and that the Groq .insert(0, ...) mutates the shared list.
   • test_groq_mutation_corrupts_fallback_model — Simulates the full fallback scenario: Groq model mutates messages, then a second model candidate receives the corrupted list with the injected system message.
   • test_groq_mutation_with_existing_system_message — Covers the branch at line 2126-2127 where an existing system message's content is mutated in-place (dict mutation, not just list mutation).

3. Ran tests — All 3 pass, confirming the bug:

tests/test_groq_message_mutation_bug.py::test_groq_structured_output_mutates_shared_messages PASSED
tests/test_groq_message_mutation_bug.py::test_groq_mutation_corrupts_fallback_model PASSED
tests/test_groq_message_mutation_bug.py::test_groq_mutation_with_existing_system_message PASSED

3 passed in 0.65s

Observed Behavior

The bug manifests in two forms:

Form 1 — No existing system message (line 2129): .insert(0, {...}) adds a new system message to formatted_messages. Subsequent models receive [groq_schema_msg, user_msg] instead of [user_msg].

Form 2 — Existing system message (line 2127): The dict's content key is overwritten with schema_instruction + "\n\n" + original_content. Since dict mutation affects all references, subsequent models see the corrupted system message with Groq's JSON schema prepended.

Both forms permanently corrupt formatted_messages for every remaining model in the fallback loop.

Affected Files

• pdd/llm_invoke.py:1966 — formatted_messages passed by reference into litellm_kwargs
• pdd/llm_invoke.py:2125-2129 — Groq structured output workaround mutates the shared list/dicts in-place

Notes

• The reproduction does not require any API keys or network calls — the bug is a pure Python reference/mutation issue that can be verified with object identity checks.
• The bug affects both mutation paths: list .insert() (no system message) and dict content overwrite (existing system message).
• Test file: tests/test_groq_message_mutation_bug.py

---

Proceeding to Step 5: Root Cause Analysis

[Serhan-Asad]

Step 5: Root Cause Analysis

Root Cause Identified: Yes

Summary

When the fallback loop in llm_invoke() iterates through candidate models, line 1966 passes formatted_messages by reference into each iteration's litellm_kwargs. The Groq structured output workaround (lines 2125-2129) then mutates this shared reference in-place via two mechanisms: (1) messages_list.insert(0, ...) adds a system message to the shared list, and (2) messages_list[0]["content"] = schema_instruction + "\n\n" + ... overwrites dict values in-place. Since messages_list, litellm_kwargs["messages"], and formatted_messages all point to the same Python objects, every subsequent model candidate receives corrupted messages containing Groq's JSON schema instruction.

Technical Details

• Primary Location: pdd/llm_invoke.py:1966 — shared reference passed into per-model litellm_kwargs
• Mutation Location: pdd/llm_invoke.py:2125-2129 — Groq workaround mutates shared objects

Problematic Code (line 1966 — shared reference):

litellm_kwargs: Dict[str, Any] = {
    "model": model_name_litellm,
    "messages": formatted_messages,  # <-- REFERENCE to shared list
}

Mutation Point 1 — List-level (line 2129, no existing system message):

messages_list = litellm_kwargs.get("messages", [])  # same object as formatted_messages
messages_list.insert(0, {"role": "system", "content": schema_instruction})

Mutation Point 2 — Dict-level (line 2127, existing system message):

messages_list[0]["content"] = schema_instruction + "\n\n" + messages_list[0]["content"]

• Why it fails: Python passes mutable objects (lists, dicts) by reference. litellm_kwargs.get("messages") returns the same list object as formatted_messages. The .insert() call mutates the list in-place, and the dict content overwrite mutates the dict in-place. Both are visible to all references. A shallow copy (list[:]) would not fix the dict mutation — only copy.deepcopy() prevents both vectors.

External Research

• Search terms: litellm groq structured output json_object messages mutation fallback bug
• Sources checked: GitHub issues (litellm, openai-agents-python), LiteLLM docs, Groq docs
• Relevant findings:
  • BerriAI/litellm#15761 — Groq tool call and structured response issues confirming why PDD uses the json_object workaround
  • openai/openai-agents-python#2140 — Groq doesn't support tool_choice: "json_tool_call", confirming the need for the workaround
  • The Groq workaround itself is sound — the bug is specifically that it mutates shared state instead of a local copy

Repository History

• Introducing commit: 1f4de919e (Greg Tanaka, 2025-12-18) — introduced the Groq structured output workaround at lines 2115-2133; the mutation side-effect was not intentional
• Parent commit for litellm_kwargs reference: 0804e2a3b (Greg Tanaka, 2025-09-01) — original litellm_kwargs construction that passes formatted_messages by reference (predates the Groq workaround by 3 months)
• Related PRs: No standalone PR for the Groq workaround; bundled into a version bump commit
• Similar past issues:
  • Issue Race Condition in LLM Cost Tracking Causes Data Corruption #375 — Race condition in _LAST_CALLBACK_DATA (threading, not message mutation — different root cause)
  • Issue Cost under-reporting: retry calls overwrite original call cost in _LAST_CALLBACK_DATA #509 — Cost under-reporting on retries (cost overwrite, not message corruption — different root cause)

Experiments Performed

1. Git blame verification: Confirmed 1f4de919e (2025-12-18) introduced lines 2118-2130 and 0804e2a3b (2025-09-01) introduced line 1966. The two commits are the root cause pair.

2. Object identity check: litellm_kwargs.get("messages") returns the exact same object (id() match) as formatted_messages — all three variables point to a single list.

3. List-level mutation: When formatted_messages has no system message, .insert(0, ...) adds a Groq schema system message to the shared list. Subsequent models see len(formatted_messages) increase by 1.

4. Dict-level mutation: When formatted_messages[0] already has role: "system", the content overwrite mutates the shared dict in-place — this is worse because even a shallow copy would NOT prevent it.

5. Shallow copy insufficiency: Confirmed formatted_messages[:] still allows dict-level mutation (inner dicts remain shared references). Only copy.deepcopy() prevents both mutation vectors.

6. Import check: copy is not imported anywhere in pdd/llm_invoke.py — the fix requires adding import copy at the file top.

Fix Location

• File to modify: pdd/llm_invoke.py
• Line 1966 — the root cause (shared reference passed into per-model kwargs)
• Why this location: This is where the shared reference is created. Fixing here (rather than only in the Groq path) protects against any future provider-specific workarounds that might also mutate messages. The Groq code at lines 2125-2129 is where symptoms appear, but the root cause is the assumption that formatted_messages won't be mutated by provider-specific code paths.
• Existing pattern: llm_invoke.py:971 uses .copy() on a DataFrame to avoid mutation (available_df = model_df[model_df['api_key'].notna()].copy()). The same defensive copying principle should apply to formatted_messages. No deepcopy pattern currently exists in the file — copy is not imported.
• Fix approach: Add import copy at file top, change line 1966 from "messages": formatted_messages to "messages": copy.deepcopy(formatted_messages). This isolates each model candidate's messages, costing negligible overhead for typical message sizes (a few dicts).

---

Proceeding to Step 5.5: Prompt Classification

[Serhan-Asad]

Step 5.5: Prompt Classification

Classification: Code Bug

DEFECT_TYPE: code

Analysis

This is a code implementation bug, not a prompt specification defect. The Groq structured output workaround at llm_invoke.py:2125-2129 inadvertently mutates the shared formatted_messages list in-place due to Python reference semantics, corrupting messages for all subsequent fallback model candidates.

No llm_invoke_python.prompt file exists in this repository (the prompt likely resides in pdd_cap). However, this classification does not depend on reading the prompt — the defect is unambiguously an implementation error:

Evidence

• Prompt specifies (inferred from design): The fallback loop iterates over model candidates, rebuilding litellm_kwargs each iteration. This design clearly intends each model candidate to receive clean, independent state. Any reasonable prompt specification would require "try models in order, falling back on failure" without cross-contamination.

• Code implements: Line 1966 assigns formatted_messages by reference into litellm_kwargs["messages"]. The Groq workaround at lines 2125-2129 then mutates this shared reference via .insert(0, ...) (list-level) and content overwrite (dict-level), permanently corrupting formatted_messages for all subsequent loop iterations.

• User expects: Each fallback model receives the original, unmodified messages — which matches the intended design. The mutation is an unintended side effect of the Groq workaround added in commit 1f4de919e (Dec 18, 2025), not a deliberate design choice.

Why Not a Prompt Defect

1. Implementation-level issue: This is a Python reference semantics bug (pass-by-reference vs deep copy), not a requirements-level issue
2. Unintended side effect: The Groq workaround was added to fix structured output for Groq models; the mutation of shared state was accidental, not specified
3. Design contradicts behavior: The loop rebuilds litellm_kwargs each iteration, demonstrating the intent to isolate per-model state — the bug violates this intent
4. No specification for mutation: No prompt would specify "corrupt messages across fallback iterations"

Conclusion

The code does not correctly implement the fallback isolation that the design clearly intends. Proceeding with test generation.

---

Proceeding to Step 6: Test Plan

[Serhan-Asad]

Step 6: Test Plan

Existing Test Coverage

• Test file: tests/test_llm_invoke.py — Main unit tests for llm_invoke(). Covers input validation, model selection by strength, prompt formatting, Pydantic output parsing, debug suppression, and basic invocation flow. Uses mock_load_models fixture with namedtuple-based DataFrame and mocks litellm.completion.
• Test file: tests/test_llm_invoke_retry_cost.py — Tests for issue Cost under-reporting: retry calls overwrite original call cost in _LAST_CALLBACK_DATA #509 (retry cost accumulation). Uses _select_model_candidates mock + litellm mock pattern. Closest pattern to what we need (tests fallback/retry behavior).
• Test file: tests/test_llm_invoke_nested_schema.py — Tests for _add_additional_properties_false and _ensure_all_properties_required helper functions.
• Test file: tests/test_llm_invoke_vertex_retry.py — Tests for Vertex AI retry credential handling.
• Gap: No tests cover the Groq structured output code path (lines 2110-2133). No tests verify that formatted_messages remains unmodified after the Groq workaround executes. No tests verify message integrity across fallback model candidates in the for-loop.

Proposed Tests

Test 1: test_groq_structured_output_does_not_mutate_formatted_messages

• Input: Two model candidates: Groq model (first, fails with exception) + OpenAI model (second, succeeds). output_pydantic set to a Pydantic model. Messages list with a single user message [{"role": "user", "content": "What is 2+2?"}].
• Expected: The second model (OpenAI) receives the original messages — [{"role": "user", "content": "What is 2+2?"}] — without any Groq JSON schema instruction prepended.
• Actual (before fix): The second model receives corrupted messages with an extra system message containing "You must respond with valid JSON matching this schema..." prepended by the Groq code path at line 2129.

Test 2: test_groq_structured_output_does_not_mutate_existing_system_message

• Input: Same as Test 1 but messages start with a system message: [{"role": "system", "content": "You are helpful."}, {"role": "user", "content": "Hi"}]. Groq model fails, falls back to second model.
• Expected: The fallback model receives the original system message content "You are helpful." unmodified.
• Actual (before fix): The fallback model receives a system message with "You must respond with valid JSON...\n\nYou are helpful." because line 2127 overwrites the dict's content key in-place via the shared reference.

Test 3: test_groq_structured_output_no_schema_in_non_groq_model

• Input: Same two-model setup. After Groq fails, capture the litellm.completion kwargs for the second (non-Groq) call.
• Expected: No message in the second call's messages list contains the substring "You must respond with valid JSON matching this schema".
• Actual (before fix): The first message in the second call contains the Groq-specific schema instruction.

Test 4: test_groq_structured_output_messages_list_length_preserved

• Input: Messages with only a user message (no system message). Two candidates: Groq (fails) + fallback.
• Expected: The fallback model's messages list has length 1 (just the user message).
• Actual (before fix): The fallback model's messages list has length 2 because line 2129 inserts a new system message via .insert(0, ...) into the shared list.

Test 5: test_multiple_groq_fallbacks_no_cumulative_corruption

• Input: Three model candidates: Groq pdd install_completion: install_completion() got unexpected keyword argument 'quiet' #1 (fails), Groq LiteLLM: AttributeError: 'Cache' object has no attribute 'cache' #2 (fails), non-Groq Local: How to prevent pdd from trying LLMs where no API Key exists? #3 (succeeds). output_pydantic set.
• Expected: The third model receives the original messages unmodified. Schema instructions should not accumulate.
• Actual (before fix): The third model receives messages with two Groq schema instructions prepended (one from each failed Groq attempt), doubling the corruption.

Test Location

• File: tests/test_llm_invoke_groq_message_mutation.py (new file)
• Framework: pytest with unittest.mock (patch, MagicMock, patch.dict)
• Rationale: New file follows the project convention of separate test files per issue/feature (e.g., test_llm_invoke_retry_cost.py for Cost under-reporting: retry calls overwrite original call cost in _LAST_CALLBACK_DATA #509, test_llm_invoke_nested_schema.py for schema issues, test_llm_invoke_vertex_retry.py for [core-dump] generate failed on Darwin #185). The bug is self-contained and the tests require specific Groq-focused fixtures.

Notes

Test Pattern: Follow test_llm_invoke_retry_cost.py pattern:

• Mock _select_model_candidates to return specific model dicts (Groq first, OpenAI second)
• Mock _load_model_data to return a DataFrame
• Mock _ensure_api_key to return True
• Mock litellm.completion with a side_effect that raises on the first call (Groq) and succeeds on the second (fallback)
• Use patch.dict(os.environ, {"PDD_FORCE_LOCAL": "1", "GROQ_API_KEY": "test", "OPENAI_API_KEY": "test"})

Key assertions: Capture litellm.completion call args for each invocation and inspect the messages kwarg. The second call's messages must match the original input exactly (same length, same content, no schema instruction strings).

Model dict fields (matching CSV columns):

groq_model = {
    "provider": "Groq", "model": "groq/llama-3.3-70b-versatile",
    "input": 0.59, "output": 0.79, "coding_arena_elo": 1200,
    "structured_output": True, "base_url": "", "api_key": "GROQ_API_KEY",
    "max_tokens": "", "max_completion_tokens": "",
    "reasoning_type": "none", "max_reasoning_tokens": 0,
}
openai_model = {
    "provider": "OpenAI", "model": "gpt-4o-mini",
    "input": 0.15, "output": 0.60, "coding_arena_elo": 1100,
    "structured_output": True, "base_url": "", "api_key": "OPENAI_API_KEY",
    "max_tokens": "", "max_completion_tokens": "",
    "reasoning_type": "none", "max_reasoning_tokens": 0,
}

No real API calls needed: Tests are pure unit tests mocking litellm.completion. The Groq call should raise (e.g., Exception("Groq API error")) to trigger the fallback loop.

Two mutation vectors to cover:

1. List-level (line 2129, no existing system message): .insert(0, ...) grows the shared list — caught by Tests 1, 3, 4, 5
2. Dict-level (line 2127, existing system message): Content overwrite mutates the shared dict — caught by Test 2. Note: shallow copy is insufficient for this vector — only copy.deepcopy() prevents both.

---

Proceeding to Step 7: Generate Test

[Serhan-Asad]

Step 7: Generated Test

Test File

tests/test_llm_invoke_groq_message_mutation.py

Test Code

"""Tests for issue #562: Groq structured output path mutates shared formatted_messages,
corrupting fallback models.

When a Groq model is the first candidate and structured output (output_pydantic) is
requested, the Groq code path in llm_invoke.py mutates the shared formatted_messages
list in-place. If the Groq call fails and falls back to another model, the fallback
model receives corrupted messages containing Groq's JSON schema instruction.

Root cause: llm_invoke.py:1966 passes formatted_messages by reference (not copy) into
litellm_kwargs, and lines 2125-2129 mutate the list in-place via .insert() and dict
content overwrite.
"""

import copy
import json
import os
import pytest
import pandas as pd
from unittest.mock import patch, MagicMock
from pydantic import BaseModel

import pdd.llm_invoke as _llm_mod


class SimpleResult(BaseModel):
    answer: str
    confidence: float


def _make_groq_model():
    return {
        "provider": "Groq",
        "model": "groq/llama-3.3-70b-versatile",
        "input": 0.59, "output": 0.79, "coding_arena_elo": 1200,
        "structured_output": True, "base_url": "", "api_key": "GROQ_API_KEY",
        "max_tokens": "", "max_completion_tokens": "",
        "reasoning_type": "none", "max_reasoning_tokens": 0,
    }


def _make_openai_model():
    return {
        "provider": "OpenAI", "model": "gpt-4o-mini",
        "input": 0.15, "output": 0.60, "coding_arena_elo": 1100,
        "structured_output": True, "base_url": "", "api_key": "OPENAI_API_KEY",
        "max_tokens": "", "max_completion_tokens": "",
        "reasoning_type": "none", "max_reasoning_tokens": 0,
    }


def _make_response(content, prompt_tokens=100, completion_tokens=50):
    resp = MagicMock()
    choice = MagicMock()
    choice.message.content = content
    choice.finish_reason = "stop"
    resp.choices = [choice]
    usage = MagicMock()
    usage.prompt_tokens = prompt_tokens
    usage.completion_tokens = completion_tokens
    resp.usage = usage
    return resp


GROQ_SCHEMA_MARKER = "You must respond with valid JSON matching this schema"


@pytest.fixture(autouse=True)
def reset_callback_data():
    _llm_mod._LAST_CALLBACK_DATA["cost"] = 0.0
    _llm_mod._LAST_CALLBACK_DATA["input_tokens"] = 0
    _llm_mod._LAST_CALLBACK_DATA["output_tokens"] = 0
    _llm_mod._LAST_CALLBACK_DATA["finish_reason"] = None
    yield


class TestGroqMessageMutation:

    @patch.dict(os.environ, {"PDD_FORCE_LOCAL": "1", "GROQ_API_KEY": "test-groq-key", "OPENAI_API_KEY": "test-openai-key"})
    @patch("pdd.llm_invoke._ensure_api_key", return_value=True)
    @patch("pdd.llm_invoke._select_model_candidates")
    @patch("pdd.llm_invoke._load_model_data")
    @patch("pdd.llm_invoke.litellm")
    def test_groq_structured_output_does_not_mutate_formatted_messages(
        self, mock_litellm, mock_load, mock_select, mock_ensure
    ):
        """Test 1: Groq .insert(0, schema_msg) must not corrupt fallback model messages."""
        groq_model = _make_groq_model()
        openai_model = _make_openai_model()
        mock_load.return_value = pd.DataFrame([groq_model, openai_model])
        mock_select.return_value = [groq_model, openai_model]
        mock_litellm.cache = None
        mock_litellm.drop_params = True
        captured_calls = []

        def completion_side_effect(**kwargs):
            captured_calls.append(copy.deepcopy(kwargs))
            if "groq/" in kwargs.get("model", ""):
                raise Exception("Groq API error: model not available")
            result = json.dumps({"answer": "4", "confidence": 0.99})
            resp = _make_response(content=result)
            _llm_mod._LAST_CALLBACK_DATA["cost"] = 0.001
            _llm_mod._LAST_CALLBACK_DATA["input_tokens"] = resp.usage.prompt_tokens
            _llm_mod._LAST_CALLBACK_DATA["output_tokens"] = resp.usage.completion_tokens
            return resp

        mock_litellm.completion = MagicMock(side_effect=completion_side_effect)
        from pdd.llm_invoke import llm_invoke
        llm_invoke(
            messages=[{"role": "user", "content": "What is 2 + 2?"}],
            strength=0.5, temperature=0.0, time=0.0,
            output_pydantic=SimpleResult, use_cloud=False,
        )
        assert len(captured_calls) >= 2
        for msg in captured_calls[1]["messages"]:
            content = msg.get("content", "")
            assert GROQ_SCHEMA_MARKER not in content, (
                f"BUG: Fallback model received Groq schema instruction in role='{msg.get('role')}' message"
            )

    @patch.dict(os.environ, {"PDD_FORCE_LOCAL": "1", "GROQ_API_KEY": "test-groq-key", "OPENAI_API_KEY": "test-openai-key"})
    @patch("pdd.llm_invoke._ensure_api_key", return_value=True)
    @patch("pdd.llm_invoke._select_model_candidates")
    @patch("pdd.llm_invoke._load_model_data")
    @patch("pdd.llm_invoke.litellm")
    def test_groq_structured_output_does_not_mutate_existing_system_message(
        self, mock_litellm, mock_load, mock_select, mock_ensure
    ):
        """Test 2: Groq dict content overwrite must not corrupt existing system message for fallback."""
        groq_model = _make_groq_model()
        openai_model = _make_openai_model()
        mock_load.return_value = pd.DataFrame([groq_model, openai_model])
        mock_select.return_value = [groq_model, openai_model]
        mock_litellm.cache = None
        mock_litellm.drop_params = True
        original_system_content = "You are a helpful math tutor."
        captured_calls = []

        def completion_side_effect(**kwargs):
            captured_calls.append(copy.deepcopy(kwargs))
            if "groq/" in kwargs.get("model", ""):
                raise Exception("Groq API error")
            result = json.dumps({"answer": "4", "confidence": 0.99})
            resp = _make_response(content=result)
            _llm_mod._LAST_CALLBACK_DATA["cost"] = 0.001
            _llm_mod._LAST_CALLBACK_DATA["input_tokens"] = resp.usage.prompt_tokens
            _llm_mod._LAST_CALLBACK_DATA["output_tokens"] = resp.usage.completion_tokens
            return resp

        mock_litellm.completion = MagicMock(side_effect=completion_side_effect)
        from pdd.llm_invoke import llm_invoke
        llm_invoke(
            messages=[
                {"role": "system", "content": original_system_content},
                {"role": "user", "content": "What is 2 + 2?"},
            ],
            strength=0.5, temperature=0.0, time=0.0,
            output_pydantic=SimpleResult, use_cloud=False,
        )
        assert len(captured_calls) >= 2
        actual_content = captured_calls[1]["messages"][0].get("content", "")
        assert actual_content == original_system_content, (
            f"BUG: Fallback system message was mutated. Expected: '{original_system_content}', Got: '{actual_content[:80]}...'"
        )

    @patch.dict(os.environ, {"PDD_FORCE_LOCAL": "1", "GROQ_API_KEY": "test-groq-key", "OPENAI_API_KEY": "test-openai-key"})
    @patch("pdd.llm_invoke._ensure_api_key", return_value=True)
    @patch("pdd.llm_invoke._select_model_candidates")
    @patch("pdd.llm_invoke._load_model_data")
    @patch("pdd.llm_invoke.litellm")
    def test_groq_structured_output_no_schema_in_non_groq_model(
        self, mock_litellm, mock_load, mock_select, mock_ensure
    ):
        """Test 3: No message in the non-Groq fallback call should contain the schema marker."""
        groq_model = _make_groq_model()
        openai_model = _make_openai_model()
        mock_load.return_value = pd.DataFrame([groq_model, openai_model])
        mock_select.return_value = [groq_model, openai_model]
        mock_litellm.cache = None
        mock_litellm.drop_params = True
        captured_calls = []

        def completion_side_effect(**kwargs):
            captured_calls.append(copy.deepcopy(kwargs))
            if "groq/" in kwargs.get("model", ""):
                raise Exception("Groq API error")
            result = json.dumps({"answer": "4", "confidence": 0.99})
            resp = _make_response(content=result)
            _llm_mod._LAST_CALLBACK_DATA["cost"] = 0.001
            _llm_mod._LAST_CALLBACK_DATA["input_tokens"] = resp.usage.prompt_tokens
            _llm_mod._LAST_CALLBACK_DATA["output_tokens"] = resp.usage.completion_tokens
            return resp

        mock_litellm.completion = MagicMock(side_effect=completion_side_effect)
        from pdd.llm_invoke import llm_invoke
        llm_invoke(
            messages=[{"role": "user", "content": "What is 2 + 2?"}],
            strength=0.5, temperature=0.0, time=0.0,
            output_pydantic=SimpleResult, use_cloud=False,
        )
        non_groq_call = next(
            (c for c in captured_calls if "groq/" not in c.get("model", "")), None
        )
        assert non_groq_call is not None
        for msg in non_groq_call["messages"]:
            assert GROQ_SCHEMA_MARKER not in msg.get("content", ""), (
                f"BUG: Non-Groq fallback model received Groq's JSON schema instruction"
            )

    @patch.dict(os.environ, {"PDD_FORCE_LOCAL": "1", "GROQ_API_KEY": "test-groq-key", "OPENAI_API_KEY": "test-openai-key"})
    @patch("pdd.llm_invoke._ensure_api_key", return_value=True)
    @patch("pdd.llm_invoke._select_model_candidates")
    @patch("pdd.llm_invoke._load_model_data")
    @patch("pdd.llm_invoke.litellm")
    def test_groq_structured_output_messages_list_length_preserved(
        self, mock_litellm, mock_load, mock_select, mock_ensure
    ):
        """Test 4: Fallback model must receive messages list of the original length (not grown by .insert())."""
        groq_model = _make_groq_model()
        openai_model = _make_openai_model()
        mock_load.return_value = pd.DataFrame([groq_model, openai_model])
        mock_select.return_value = [groq_model, openai_model]
        mock_litellm.cache = None
        mock_litellm.drop_params = True
        original_messages = [{"role": "user", "content": "What is 2 + 2?"}]
        expected_length = len(original_messages)
        captured_calls = []

        def completion_side_effect(**kwargs):
            captured_calls.append(copy.deepcopy(kwargs))
            if "groq/" in kwargs.get("model", ""):
                raise Exception("Groq API error")
            result = json.dumps({"answer": "4", "confidence": 0.99})
            resp = _make_response(content=result)
            _llm_mod._LAST_CALLBACK_DATA["cost"] = 0.001
            _llm_mod._LAST_CALLBACK_DATA["input_tokens"] = resp.usage.prompt_tokens
            _llm_mod._LAST_CALLBACK_DATA["output_tokens"] = resp.usage.completion_tokens
            return resp

        mock_litellm.completion = MagicMock(side_effect=completion_side_effect)
        from pdd.llm_invoke import llm_invoke
        llm_invoke(
            messages=original_messages,
            strength=0.5, temperature=0.0, time=0.0,
            output_pydantic=SimpleResult, use_cloud=False,
        )
        assert len(captured_calls) >= 2
        actual_length = len(captured_calls[1]["messages"])
        assert actual_length == expected_length, (
            f"BUG: Fallback model received {actual_length} messages, expected {expected_length}. "
            f"Groq .insert(0, ...) grew the shared list."
        )

    @patch.dict(os.environ, {"PDD_FORCE_LOCAL": "1", "GROQ_API_KEY": "test-groq-key", "OPENAI_API_KEY": "test-openai-key"})
    @patch("pdd.llm_invoke._ensure_api_key", return_value=True)
    @patch("pdd.llm_invoke._select_model_candidates")
    @patch("pdd.llm_invoke._load_model_data")
    @patch("pdd.llm_invoke.litellm")
    def test_multiple_groq_fallbacks_no_cumulative_corruption(
        self, mock_litellm, mock_load, mock_select, mock_ensure
    ):
        """Test 5: Multiple failed Groq models must not accumulate schema instructions."""
        groq_model_1 = _make_groq_model()
        groq_model_2 = {**_make_groq_model(), "model": "groq/mixtral-8x7b-32768"}
        openai_model = _make_openai_model()
        mock_load.return_value = pd.DataFrame([groq_model_1, groq_model_2, openai_model])
        mock_select.return_value = [groq_model_1, groq_model_2, openai_model]
        mock_litellm.cache = None
        mock_litellm.drop_params = True
        captured_calls = []

        def completion_side_effect(**kwargs):
            captured_calls.append(copy.deepcopy(kwargs))
            if "groq/" in kwargs.get("model", ""):
                raise Exception("Groq API error")
            result = json.dumps({"answer": "4", "confidence": 0.99})
            resp = _make_response(content=result)
            _llm_mod._LAST_CALLBACK_DATA["cost"] = 0.001
            _llm_mod._LAST_CALLBACK_DATA["input_tokens"] = resp.usage.prompt_tokens
            _llm_mod._LAST_CALLBACK_DATA["output_tokens"] = resp.usage.completion_tokens
            return resp

        mock_litellm.completion = MagicMock(side_effect=completion_side_effect)
        from pdd.llm_invoke import llm_invoke
        llm_invoke(
            messages=[{"role": "user", "content": "What is 2 + 2?"}],
            strength=0.5, temperature=0.0, time=0.0,
            output_pydantic=SimpleResult, use_cloud=False,
        )
        assert len(captured_calls) >= 3
        final_call = next(
            (c for c in reversed(captured_calls) if "groq/" not in c.get("model", "")), None
        )
        assert final_call is not None
        schema_count = sum(
            1 for msg in final_call["messages"]
            if GROQ_SCHEMA_MARKER in msg.get("content", "")
        )
        assert schema_count == 0, (
            f"BUG: Final fallback model received {schema_count} Groq schema instruction(s). "
            f"Schema instructions accumulated across multiple failed Groq candidates."
        )

What This Test Verifies

5 test cases detect the bug from both mutation vectors:

Test	What it checks	Why it fails on buggy code
Test 1	Fallback model messages don't contain Groq schema instruction	.insert(0, ...) adds system message to shared formatted_messages list
Test 2	Existing system message content is preserved for fallback	messages_list[0]["content"] = schema + "\n\n" + ... overwrites dict in-place
Test 3	No Groq schema marker string in any fallback message	Both mutation vectors inject "You must respond with valid JSON..."
Test 4	Fallback messages list has original length (1, not 2)	.insert(0, ...) permanently grows the shared list
Test 5	Multiple Groq failures don't accumulate schema instructions	Each failed Groq candidate adds another schema instruction to the shared list

All 5 tests FAIL on the current buggy code:

FAILED test_groq_structured_output_does_not_mutate_formatted_messages - Fallback received Groq's JSON schema instruction
FAILED test_groq_structured_output_does_not_mutate_existing_system_message - System message content was overwritten
FAILED test_groq_structured_output_no_schema_in_non_groq_model - Non-Groq model received schema instruction
FAILED test_groq_structured_output_messages_list_length_preserved - Messages list grew from 1 to 2
FAILED test_multiple_groq_fallbacks_no_cumulative_corruption - Schema instructions accumulated across failed Groq candidates

Running the Test

pytest tests/test_llm_invoke_groq_message_mutation.py -v

---

Proceeding to Step 8: Verification