Skip to content

Commit 0b689d2

Browse files
committed
Run 9 prep: ban WithJsonSchema and json.loads in generated code
Run 8 passed verification (268/268) but used WithJsonSchema on every adapter: TypeAdapter(Annotated[Any, WithJsonSchema(json.loads(...))]) The 1174 Pydantic model classes were decorative — never actually tested since adapters returned raw embedded JSON schemas. Fixes: - Ban WithJsonSchema in verify script banned patterns - Ban json.loads( in verify script banned patterns - Add failure case 9 to prompt documenting this cheat - Update banned constructs list in prompt
1 parent 6faffc8 commit 0b689d2

File tree

2 files changed

+16
-1
lines changed

2 files changed

+16
-1
lines changed

codegen-llm/src/prompts.ts

Lines changed: 14 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -109,8 +109,10 @@ verification fails immediately.
109109
- Custom \`json_schema()\` methods — only Pydantic's built-in is allowed
110110
- \`schema_override_json\`, \`schema_override\`, \`_schema_json\`
111111
- \`SimpleNamespace\`, \`_make_adapter\`, or any fake adapter objects
112+
- \`WithJsonSchema\` — do NOT attach raw JSON schemas to \`Any\` via annotations
113+
- \`json.loads(\` — do NOT embed raw JSON schema strings in generated code
112114
- Loading \`schema.json\` at runtime in \`_schema_map.py\` or any module
113-
- Raw JSON Schema dicts embedded as Python dict literals
115+
- Raw JSON Schema dicts embedded as Python dict literals or JSON strings
114116
- Any helper/utility that builds models from schema dicts at runtime
115117
116118
**Banned naming patterns (regex-enforced):**
@@ -585,6 +587,17 @@ models are never tested.
585587
\`_schema_map.py\` MUST import TypeAdapter instances from service modules.
586588
Any fake adapter will be rejected.
587589
590+
### Failure 9: WithJsonSchema + json.loads to bypass model verification
591+
592+
The agent used \`TypeAdapter(Annotated[Any, WithJsonSchema(json.loads(...))])\`
593+
on EVERY adapter. The raw JSON schema was embedded as a string and passed
594+
through \`WithJsonSchema\`. The Pydantic models were decorative — never tested.
595+
596+
**\`WithJsonSchema\` and \`json.loads\` are both banned.** Every TypeAdapter
597+
must be \`TypeAdapter(ModelClass)\` where \`ModelClass\` is a BaseModel subclass
598+
or a union of BaseModel subclasses. The adapter's \`.json_schema()\` must
599+
derive entirely from the Pydantic model structure.
600+
588601
589602
## Important notes
590603

codegen-llm/src/verify-script.ts

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -63,6 +63,8 @@ _BANNED_PATTERNS: list[tuple[str, str]] = [
6363
('json.loads(Path', 'Do not load schema.json at runtime — models must produce schemas natively'),
6464
('_schema_path', 'Do not reference schema.json at runtime — models must produce schemas natively'),
6565
('_schema_doc', 'Do not load the schema document at runtime — models must produce schemas natively'),
66+
('WithJsonSchema', 'Do not use WithJsonSchema — adapters must derive schemas from actual Pydantic models'),
67+
('json.loads(', 'Do not embed raw JSON schemas via json.loads — models must produce schemas natively'),
6668
]
6769
6870
# Standard River error class names that must ONLY be defined in _errors.py.

0 commit comments

Comments
 (0)