check if its within context window before running pdd

[Serhan-Asad]

Feature description

llm_invoke.py sends prompts to LLMs without checking if the assembled prompt fits within the model's context window. When the prompt exceeds the limit, the LLM returns a raw API error with no actionable guidance — no token count, no model limit, no suggestion to reduce prompt size.

PDD has a token counting utility (pdd/server/token_counter.py) with count_tokens() and get_context_limit(), but it's only used in the server API layer — not wired into the LLM invocation pipeline.

Current behavior

1. Prompt is formatted via _format_messages() → formatted_messages (line 1852 or 1856)
2. Passed into litellm_kwargs["messages"] (line 1966) inside the per-model candidate loop
3. Sent to litellm.completion() (line 2419) with no token validation
4. If prompt exceeds context window → raw API error, no helpful message

There is zero handling for context window overflow errors anywhere in llm_invoke.py.

Expected behavior

Before calling litellm.completion(), validate the assembled prompt against the model's context limit:

• Over limit → raise a clear error with: token count, model limit, usage percentage, and which prompt/file caused it
• >90% of limit + verbose → log a warning with token metrics
• Verbose mode → always log token count so users can understand prompt size

Affected code

File: pdd/llm_invoke.py

• formatted_messages assigned at line 1852 (from messages param) or 1856 (from _format_messages())
• Used at line 1966: litellm_kwargs["messages"] = formatted_messages
• LLM called at line 2419: litellm.completion(**litellm_kwargs, timeout=LLM_CALL_TIMEOUT)
• Validation should go inside the per-model loop (after line 1966) since the model name is needed for get_context_limit()

Existing utility: pdd/server/token_counter.py

• count_tokens(text) → tiktoken cl100k_base token count
• get_context_limit(model) → context limit by model family prefix
• MODEL_CONTEXT_LIMITS → GPT-4 (128K), GPT-5 (200K), Claude (200K), Gemini (1M), default (128K)

Implementation notes

1. Claude 1M context mismatch — llm_invoke.py line 2013-2014 already adds anthropic-beta: context-1m-2025-08-07 for Claude models, extending effective context to 1M. But token_counter.py lists Claude at 200K. The validation must use the actual effective limit (1M for Claude when the beta header is present).

2. Bedrock model names — token_counter.py does prefix matching (e.g., "claude-3" in model_lower), but Bedrock models use names like anthropic.claude-opus-4-6-v1. The matching logic needs to handle these.

3. No context column in llm_model.csv — The model CSV (pdd/data/llm_model.csv) has cost/elo/provider but no context window column. Consider adding one as the source of truth, or keep the hardcoded dict in token_counter.py.

4. Insertion point — The check goes inside the for model_info in candidate_models: loop (line 1941), after litellm_kwargs is built (line 1966) and model-specific headers are set (line 2014), but before litellm.completion() (line 2419).

[Serhan-Asad]

target 2/23/026

[Serhan-Asad]

Step 1: Duplicate Check

Status: No duplicates found

Search Performed

• Searched for: "context window", "token count", "token_counter", "llm_invoke token", "prompt size validation", "context limit overflow", "litellm completion error", "prompt exceeds"
• Issues reviewed: 12 candidates evaluated in detail

Related Issues (Not Duplicates)

• include in tests into the context window when generating #97 (CLOSED) — "include in tests into the context window when generating" — About including test files in context during generation, not about validating prompt size against model limits.
• Configurable context assembly for updates #107 (OPEN) — "Configurable context assembly for updates" — About configuring which files are included in context (neighbor files, README), not about token limit validation.
• Feature: Selective includes and LLM-powered extraction for token-efficient prompts #190 (OPEN) — "Selective includes and LLM-powered extraction for token-efficient prompts" — About reducing prompt size through selective extraction. Complementary but different: Feature: Selective includes and LLM-powered extraction for token-efficient prompts #190 reduces token usage, check if its within context window before running pdd #559 validates against limits before sending.
• Integrate context map sampling into code generation paths #318 (OPEN) — "Integrate context map sampling into code generation paths" — About instrumentation/observability of token usage for analysis and visualization, not about blocking calls that exceed limits.
• Model Parameter Validation & Temperature Error Flooding #192 (OPEN) — "Model Parameter Validation & Temperature Error Flooding" — About pre-call parameter validation (temperature), not context window validation. Related in spirit (pre-call validation) but different scope.

Findings

This is a new change request. No existing issue (open or closed) requests the same feature: pre-call validation of assembled prompt token count against model context window limits in llm_invoke.py, with clear error messages including token count, model limit, and usage percentage. While several issues touch on token efficiency (#190), context instrumentation (#318), and pre-call validation (#192), none address the specific gap of blocking LLM calls that exceed the model's context window.

---

Proceeding to Step 2: Documentation Check

[Serhan-Asad]

[Serhan-Asad]

Step 2/13: Check if already implemented

Status: FAILED

Error Details

All agent providers failed: anthropic: Exit code 1: ; google: Exit code 1: YOLO mode is enabled. All tool calls will be automatically approved.
Both GOOGLE_API_KEY and GEMINI_API_KEY are set. Using GOOGLE_API_KEY.
YOLO mode is enabled. All tool calls will be aut; openai: 

---

Automated fallback comment — agent did not execute for this step.

[Serhan-Asad]

Step 3/13: Research to clarify specifications

Status: FAILED

Error Details

All agent providers failed: anthropic: Exit code 1: ; google: Exit code 1: YOLO mode is enabled. All tool calls will be automatically approved.
Both GOOGLE_API_KEY and GEMINI_API_KEY are set. Using GOOGLE_API_KEY.
YOLO mode is enabled. All tool calls will be aut; openai: 

---

Automated fallback comment — agent did not execute for this step.

[Serhan-Asad]

Step 4: Requirements Clarity Check

Status: Requirements Clear

Clarity Assessment

Aspect	Status
Problem statement	✅ Clear
Expected outcomes	✅ Defined
Scope	✅ Bounded
Edge cases	✅ Addressable

Understanding Summary

Add pre-call context window validation inside the per-model candidate loop in llm_invoke.py, using the existing token_counter.py utilities. Before each litellm.completion() call, count the prompt tokens and compare against the model's context limit — raising a clear error if over limit, logging a warning at >90% usage in verbose mode, and always logging token count when verbose is enabled. Update token_counter.py to handle Claude 1M effective limits (when the anthropic-beta header is present) and Bedrock-style model names.

Assumptions Made

• Candidate model fallback behavior: When a prompt exceeds one candidate model's context limit, the implementation will skip to the next candidate model (consistent with existing error-handling patterns in the loop), and only raise a hard error if all candidate models are exceeded.
• Token counting for message lists: Since count_tokens() accepts a str but formatted_messages is a List[Dict], the messages will be serialized to a string (e.g., concatenating all content fields) before counting. This provides a reasonable approximation.
• Claude context limit: Since the code unconditionally adds anthropic-beta: context-1m-2025-08-07 for all Claude models (line 2013), the effective context limit for Claude will always be 1,000,000 tokens in token_counter.py.
• Context limit source of truth: Will update the hardcoded MODEL_CONTEXT_LIMITS dict in token_counter.py (rather than adding a CSV column), keeping the change minimal and self-contained.
• "Which prompt/file caused it": The error message will include the total token count, model limit, and usage percentage. Identifying the specific file/prompt section that pushed it over the limit is not feasible without prompt-level accounting, so the error will focus on actionable metrics.

---

Proceeding to Step 5: Documentation Changes

[Serhan-Asad]

Step 5: Documentation Changes

Status: Documentation Updates Required

Summary

The context window validation feature adds new error messages, a warning tier, and verbose logging that users will encounter. Documentation needs updates in the README's Error Handling and Troubleshooting sections to describe these new behaviors. No new documentation files are needed — this is an internal validation enhancement, not a new command or user-facing concept.

Files to Update

README.md

Section: Error Handling (line 2818)
Change: Add context window overflow to the list of handled error scenarios
Draft content:

## Error Handling

PDD provides informative error messages when issues occur during command execution. Common error scenarios include:

- Invalid input files or formats
- Insufficient permissions to read/write files
- AI model-related errors (e.g., API failures)
- Syntax errors in generated code
- **Prompt exceeding model context window** (token count, model limit, and usage percentage are reported)

When an error occurs, PDD will display a message describing the issue and, when possible, suggest steps to resolve it.

---

Section: Troubleshooting (line 2848)
Change: Add new troubleshooting entry for context window overflow errors
Draft content (append as item 9):

9. **Prompt exceeds context window**:
   - PDD validates prompt size against the model's context limit before making LLM calls
   - The error message includes: token count, model limit, and usage percentage
   - To resolve: reduce prompt size by trimming `<include>` content, splitting into smaller modules, or switching to a model with a larger context window
   - Use `--verbose` to see token counts for all LLM calls, even when within limits

---

Section: Global Options — --verbose (line 602)
Change: Expand the --verbose description to mention token count logging
Draft content:

- `--verbose`: Increase output verbosity for more detailed information, including token counts and context window usage for LLM calls.

Files to Create

None required.

No Changes Needed

• docs/prompting_guide.md — Covers prompt-writing methodology, not runtime validation. The feature is an internal safety check that doesn't change how users write prompts.
• docs/faq.md — Philosophical/conceptual Q&A; context window validation is an operational detail, not a paradigm question.
• docs/TUTORIALS.md — Workflow tutorials; this feature requires no tutorial (it's automatic validation).
• docs/ONBOARDING.md — Setup guide; no onboarding steps needed for this feature.
• docs/prompt-driven-development-doctrine.md — Already discusses "The Context Window Advantage" conceptually; this runtime feature doesn't change the doctrine.
• CONTRIBUTING.md — Contributor guide; no new contribution patterns needed.
• context/server/token_counter_example.py — Already demonstrates count_tokens(), get_context_limit(), and get_token_metrics(). No updates needed since the example already covers the public API that will be reused.
• CHANGELOG.md — Will need an entry when the feature ships, but that's added at release time, not pre-drafted.

Notes

• The documentation changes are minimal because this feature is primarily an internal validation guard — users only interact with it when they hit the limit (error message) or when they opt into verbose mode (token count logging)
• The >90% warning tier is only visible in --verbose mode, consistent with existing verbose behavior patterns in the codebase

---

Proceeding to Step 6: Identify Dev Units

[Serhan-Asad]

Step 6: Dev Units Identified

Status: Found 2 dev units (both Direct Edit Candidates — no PDD prompts exist)

Change Summary

Add context window validation to llm_invoke() before calling litellm.completion(), using the existing token_counter.py utilities. Update token_counter.py to handle Claude 1M context and Bedrock model name formats.

Project Structure (from .pddrc)

• Source directory: pdd/ (pdd_cli context), pdd/server/ (server context)
• Test directory: tests/ (pdd_cli context), tests/server/ (server context)
• Example directory: context/ (pdd_cli context), context/server/ (server context)
• Prompt directory: pdd/prompts/ (symlinked from prompts/ at root) — flat, no subdirectories
• Language: Python

Dev Units to MODIFY

Neither module has a corresponding .prompt file — both are Direct Edit Candidates.

No PDD-managed dev units to MODIFY. See "Direct Edit Candidates" below.

Dev Units to CREATE

None. All required functionality can be achieved by modifying existing files.

Reuse Analysis

• token_counter.py already provides count_tokens(), get_context_limit(), get_token_metrics(), MODEL_CONTEXT_LIMITS — the full token counting and context limit infrastructure exists. No new module needed.
• llm_invoke.py already has the per-model candidate loop with verbose logging infrastructure. Validation plugs directly into the existing loop structure.

Dependencies (may need interface updates)

• pdd/server/routes/prompts.py — Imports get_token_metrics (line 28) and MODEL_CONTEXT_LIMITS (line 476). If get_context_limit() signature changes (e.g., adding an effective_1m parameter for Claude), this file's duplicate get_context_limit() function (line 482) would also need updating. No prompt file exists for this module.
• llm_invoke callers (34+ files) — The llm_invoke() function signature is NOT changing, only internal behavior. Callers are unaffected.

Integration Points Discovered

No new modules are being created, so no registration/aggregation updates needed.

Integration Point	Reason for Update	Prompt File
pdd/server/routes/prompts.py	Contains duplicate get_context_limit() at line 482 and imports MODEL_CONTEXT_LIMITS; if token_counter.py model limits change, this needs syncing	None (manual update needed)

Search performed:

• grep "from.*token_counter import\|import.*token_counter" across entire codebase → found 3 importers: routes/prompts.py, test_token_counter.py, token_counter_example.py
• grep "from.*llm_invoke import\|import.*llm_invoke" across entire codebase → found 34+ importers
• find pdd/prompts -type d → confirmed prompts directory is flat (no subdirectories)
• glob **/llm_invoke_python.prompt and **/token_counter_python.prompt → confirmed no prompt files exist for either module

Frontend Dev Units (Cross-Layer)

No frontend files reference token counting, context windows, or the affected backend functions.

Backend endpoints being created/modified:

• llm_invoke() in pdd/llm_invoke.py (internal validation only, no API change)
• get_context_limit() in pdd/server/token_counter.py (may gain parameter)

Frontend files referencing these endpoints:

Frontend File	Prompt File	Action
(none found)	—	—

Search performed:

• grep "context.window\|context_window\|token.count\|token_count\|context.limit\|context_limit" in pdd/frontend/**/*.{ts,tsx} → no matches

Direct Edit Candidates (No Prompt)

Both affected modules lack corresponding .prompt files and must be edited directly.

File	Edit Type	What Needs to Change
pdd/llm_invoke.py	Add validation logic	Insert context window check inside for model_info in candidate_models: loop (after line ~2014 where anthropic-beta header is set, before litellm.completion() at line ~2419). Serialize formatted_messages → count tokens → compare against get_context_limit(model) → raise clear error if over limit, warn if >90% + verbose, log token count if verbose.
pdd/server/token_counter.py	Extend existing functions	1) Update MODEL_CONTEXT_LIMITS or get_context_limit() to return 1M for Claude models when the beta header is active (currently hardcoded at 200K). 2) Fix prefix matching in get_context_limit() to handle Bedrock model names like anthropic.claude-opus-4-6-v1.
tests/test_llm_invoke.py	Add tests	Add test cases for: context window overflow error, >90% warning in verbose mode, verbose token count logging, Claude 1M effective limit.
tests/server/test_token_counter.py	Add tests	Add test cases for: Bedrock model name resolution, Claude 1M context limit parameter, updated model limits.
pdd/server/routes/prompts.py	Sync changes	If get_context_limit() interface changes or MODEL_CONTEXT_LIMITS values change, sync the duplicate get_context_limit() function at line 482.

Note: Direct edits are appropriate here because these are core infrastructure modules without PDD prompt files. All changes are scoped additions (new validation logic, extended model matching) that don't alter existing interfaces.

Complete Dev Unit Inventory

Module	Prompt	Code	Example	Test
llm_invoke	❌ None	pdd/llm_invoke.py (3,101 lines)	context/llm_invoke_example.py	tests/test_llm_invoke.py
token_counter	❌ None	pdd/server/token_counter.py (222 lines)	context/server/token_counter_example.py	tests/server/test_token_counter.py

Files Explored

• pdd/prompts/ — all 120+ prompt files (flat directory, no subdirectories)
• pdd/llm_invoke.py — lines 1728-1870 (function signature, formatted_messages), 1940-2040 (per-model loop, headers), 2400-2450 (litellm.completion call)
• pdd/server/token_counter.py — full file (222 lines)
• tests/test_llm_invoke.py — full file (80+ lines of imports/fixtures)
• tests/server/test_token_counter.py — full file (215 lines)
• context/llm_invoke_example.py — full file (141 lines)
• context/server/token_counter_example.py — full file (86 lines)
• pdd/server/routes/prompts.py — token_counter integration points
• pdd/__init__.py — package exports
• pdd/server/__init__.py — server package exports
• .pddrc — project configuration (8 contexts)

---

Proceeding to Step 7: Architecture Review

[Serhan-Asad]

Step 7: Architecture Review

Status: Architecture Clear

Reuse Validation

• No new modules proposed — Step 6 correctly identified only MODIFY operations on 2 existing files (pdd/llm_invoke.py, pdd/server/token_counter.py). No new files or modules need to be created.
• Existing token_counter.py utilities are reused — count_tokens() and get_context_limit() are imported into llm_invoke.py rather than reimplementing token counting logic.
• Duplicate cleanup identified — pdd/server/routes/prompts.py contains a local duplicate of get_context_limit() (line ~482). This should be refactored to import from token_counter.py to prevent the two implementations from diverging after we update the limits.

Analysis

The implementation approach is straightforward based on the dev units identified in Step 6. All decisions follow directly from existing code patterns:

1. Error handling follows established pattern — The for model_info in candidate_models: loop already handles errors by breaking to the next model (auth errors, schema validation, rate limits). Context window overflow will follow this same pattern: skip to next candidate model, and only raise a clear error with token metrics if ALL candidates are exhausted.

2. Claude 1M is the effective limit — Since llm_invoke.py unconditionally adds the anthropic-beta: context-1m-2025-08-07 header for all Claude models (lines 2013-2014), get_context_limit() should return 1M for Claude. No conditional logic needed.

3. Message serialization is straightforward — formatted_messages (a List[Dict]) will be serialized to a string via json.dumps() for token counting. This is a conservative overestimate (JSON overhead) but safe for a pre-flight validation check.

4. Context limit source of truth stays in token_counter.py — The hardcoded MODEL_CONTEXT_LIMITS dict is the right place. Adding a CSV column would be over-engineering for this feature. The dict just needs updating for Claude (200K → 1M) and Bedrock model name patterns.

Implementation Approach

• pdd/server/token_counter.py — Update MODEL_CONTEXT_LIMITS to reflect Claude's 1M effective limit. Add Bedrock model name patterns (e.g., "anthropic.claude" prefix) to the matching logic.
• pdd/llm_invoke.py — Import count_tokens and get_context_limit from token_counter.py. Insert validation after litellm_kwargs is fully built (after line ~2016, after the anthropic-beta header), but before litellm.completion() (line ~2419). Three tiers:
  • Over limit → log error with token count, model limit, usage %, break to next candidate model
  • >90% of limit + verbose → log warning with token metrics
  • Verbose → always log token count
  • If ALL candidate models fail due to context overflow → raise a clear ContextWindowExceededError (or similar) with actionable guidance
• pdd/server/routes/prompts.py — Replace local get_context_limit() duplicate with import from token_counter.py

Assumptions

1. Context window overflow follows the existing error-handling pattern: try next candidate model before failing
2. The final error (when all candidates exhausted) includes token count, model limit, usage %, and the prompt/file that caused it (from existing context in the function)
3. json.dumps() serialization of messages is acceptable for token estimation (slight overcount is safe)
4. Claude's 1M limit is used unconditionally since the beta header is always added
5. The 90% warning threshold is hardcoded (not configurable) — matching the issue's specification

---

Proceeding to Step 8: Analyze Prompt Changes

[Serhan-Asad]

Step 8: Prompt Change Analysis

Status: No Changes Required

Summary

Both affected modules (pdd/llm_invoke.py and pdd/server/token_counter.py) are Direct Edit Candidates with no corresponding .prompt files in the prompts directory. The code changes required by issue #559 cannot be driven through PDD prompt regeneration and must be implemented as direct code edits.

Analysis

Investigation Performed

1. Searched for prompt files: Globbed for **/llm_invoke*.prompt and **/token_counter*.prompt — zero results. Neither module has a PDD-managed prompt.

2. Verified prompts directory: The pdd/prompts/ directory contains 120+ .prompt files covering agentic workflows (agentic_*), extraction utilities (extract_*), code generation (agentic_fix_python.prompt, agentic_bug_python.prompt), and LLM orchestration prompts — but none for llm_invoke or token_counter.

3. Checked for indirect references: auto_include_LLM.prompt references llm_invoke_python.prompt as a <pdd-dependency>, but this prompt file does not exist — it appears to be a forward reference or planned artifact that was never created.

4. Context example files exist: context/llm_invoke_example.py (141 lines) and context/server/token_counter_example.py (86 lines) exist as usage demonstrations, but these are documentation artifacts, not PDD-managed generation sources.

5. No project-level architecture.json: Only example architecture.json files exist under examples/. No architecture entry defines these modules.

Why No Prompt Changes Are Needed

Module	Has .prompt?	Has Example?	PDD-Managed?	Change Strategy
pdd/llm_invoke.py	No	Yes	No	Direct code edit
pdd/server/token_counter.py	No	Yes	No	Direct code edit
pdd/server/routes/prompts.py	No	No	No	Direct code edit (cleanup duplicate get_context_limit)

These modules are infrastructure/library code that other PDD-managed modules depend on. They sit below the PDD prompt layer — llm_invoke.py is the engine that executes prompt generations, and token_counter.py is a server utility. Creating new prompts to manage them is out of scope for this issue (it would be a separate initiative to bring core infrastructure under PDD management).

What Changes ARE Needed (Direct Edits — Step 9 Scope)

For pdd/server/token_counter.py:

• Update MODEL_CONTEXT_LIMITS to reflect Claude's effective 1M context limit (since llm_invoke.py always adds the anthropic-beta: context-1m-2025-08-07 header)
• Fix get_context_limit() to handle Bedrock-style model names (e.g., anthropic.claude-opus-4-6-v1)

For pdd/llm_invoke.py:

• Import count_tokens and get_context_limit from pdd.server.token_counter
• Add context window validation inside the for model_info in candidate_models: loop (after litellm_kwargs is built at ~line 1966, before litellm.completion() at ~line 2419)
• Over-limit → raise clear error with token count, model limit, usage %, and prompt/file info
• >90% of limit + verbose → log warning with token metrics
• Verbose mode → always log token count

For pdd/server/routes/prompts.py:

• Replace the duplicate local get_context_limit() function (lines 482-488) with an import from token_counter.py

Context Example File Updates

The context/server/token_counter_example.py should be updated to demonstrate the new Bedrock model name handling and updated Claude context limits, since other prompts may include it as a dependency reference.

Interface Changes

• token_counter.get_context_limit(): Behavior change — Claude models will return 1,000,000 instead of 200,000. This is a correctness fix (matches the actual effective limit when the beta header is present), not a breaking change.
• token_counter.MODEL_CONTEXT_LIMITS: Dict values for Claude entries change from 200,000 to 1,000,000.
• llm_invoke() function signature: No change. The validation is internal — callers are unaffected.
• New exception type: llm_invoke() may raise a new descriptive error when context limit is exceeded. Callers that catch broad exceptions will handle this automatically.

Risk Assessment

• Low: No prompt files are being modified or created. The code changes are additive (new validation guard) and the interface change to get_context_limit() is a correctness fix. The 34+ callers of llm_invoke() are unaffected since the function signature doesn't change. Existing tests for token_counter.py will need to be updated to reflect the new Claude context limits.
• Mitigation: The context window check is a pre-flight guard that only triggers when prompts are too large — it adds safety without changing the happy path.

---

Proceeding to Step 9: Implement Changes (Direct Code Edits)