STACK Question Generator

Open-source tool (GPL-3.0) that turns structured question + answer input into valid, Moodle-importable STACK XML — without hand-writing Maxima, PRT logic, or XML. The tool authors for STACK; it never reimplements STACK's grading engine.

Status: Phase 6 (all phases through Phase 6 done — see PRD.md §13)

Seven pattern modules implement PRD §7.1/§7.2 coverage for: numerical_tolerance (unitless/units NumRelative, NumAbsolute, NumSigFigs), units (the dedicated UnitsRelative/UnitsAbsolute/UnitsStrictRelative/ UnitsStrictAbsolute coverage), algebraic_equivalence (AlgEquiv), matrix (matrix/varmatrix), multipart (several independently-graded sub-answers in one question), mcq (dropdown/radio/checkbox), and equiv (reasoning-by-equivalence, line-by-line working). This covers a subset of the full target surface (see PRD §7); additional answer kinds and tests are added as new pattern modules. See CONTRIBUTING.md for the recipe to add another one.

The inference layer sits in front of all of them: given only a natural-language question and answer, an LLM classifies the input against the decision rules in prompts/inference.md — answer kind, answer test, tolerance, MCQ options, and anticipated wrong-answer PRT branches — and the resulting plan is routed to whichever pattern module claims it. Explicit fields in the input JSON always override inferred ones (PRD F-2).

Also shipped: the Docker Moodle+STACK validation loop with LLM self-correction (Phase 1), the React/Tailwind web frontend (Phase 2), a local zero-cost Ollama backend alongside the cloud Claude one (Phase 4), CSV/JSON batch generation and n8n workflow templates (Phase 5), and image input, a documented security model, and this contributor guide (Phase 6).

Quick start

pip install .            # installs the `stackgen` CLI (no runtime deps)
stackgen generate examples/02_free_fall_velocity.json -o out/

Or straight from a checkout without installing: python -m cli.main generate ...

Input JSON for the numerical/tolerance pattern:

{
  "pattern": "numerical_tolerance",
  "name": "Free fall impact velocity",
  "question_text": "<p>...\\(g = {@g@}\\,\\mathrm{m/s^2}\\)...</p>",
  "variables": ["g: 9.81;", "t: 3;"],
  "answer": "29.43",
  "unit": "m/s",
  "deviation": 0.01
}

• "deviation" is the accepted relative deviation around the answer (default 0.05). The generated question variables expose the band as min_deviation / max_deviation (stackunits((1±deviation)*taRn, unit)) for use in feedback.
• With a "unit", the input type is units and grading uses UnitsRelative. An answer of 0 becomes stackunits(0, unit) — an exact match.
• For compound units that stackunits mishandles (e.g. Nm), set "unit_exception": true: the input stays numerical and grading uses NumRelative, with the unit only in the question text. Unitless questions work the same way.
• Each generated question ships with question tests covering every PRT branch.

Natural-language input (inference)

{
  "question": "A car travelling at 21 m/s brakes uniformly to rest in 5 s. Taking the direction of travel as positive, find the car's acceleration while braking, in m/s^2. Give your answer with units.",
  "answer": "-4.2 m/s^2"
}

pip install .[llm]                 # anthropic SDK + python-dotenv
stackgen generate examples/04_natural_language_braking.json -o out/ --llm claude

BYOK: put ANTHROPIC_API_KEY=sk-ant-... in your environment or in a .env file in the working directory (a real environment variable wins). The key is never stored. Or skip the key entirely and run zero-cost against a local model: ollama pull qwen2.5:7b && ollama serve, then stackgen generate examples/04_natural_language_braking.json -o out/ --llm ollama — see "LLM backend for generation" below for the quality tradeoff.

Any JSON without a pattern/question_text field is treated as natural language and needs an LLM backend. Add explicit fields (deviation, answer_test, unit, ...) to override whatever was inferred. The decision rules live in prompts/inference.md; the plan itself is validated strictly before any XML is generated.

Batch generation (PRD F-7)

stackgen batch examples/batch_questions.csv -o out/
stackgen batch examples/batch_questions.json -o out/ --llm claude --validate

batch runs a whole CSV or JSON file of rows through the same pipeline as generate, one XML per row, into an output folder. Structured and natural-language rows (see above) can be mixed freely in the same file — --llm is only required if at least one row needs it. One bad row does not abort the batch: it's recorded as a per-row error and every other row still runs.

Row shape. Each row is exactly what a single generate input JSON would be: pattern/question_text/answer for structured rows, or question/answer for natural-language rows, plus any of the same explicit overrides (deviation, unit, sig_figs, answer_test, ...). An optional id column/key labels the row in the report and output filename; rows without one are labelled by their 1-based position.

CSV column rules (see examples/batch_questions.csv):

• A blank cell means the field is absent — same as omitting the key in JSON.
• Cells that parse as JSON (numbers, true/false, null, [...], {...}) are decoded to that type, so typed fields come out right: sig_figs → int, unit_exception → bool, variables / distractors / options → list (write these as a JSON array in one cell, e.g. ["g: 9.81;", "t: 3;"]).
• Cells that aren't valid JSON (plain text, bare Maxima expressions like 2*pi*sqrt(L/g) or {1,2,3}) are kept as literal text.
• Exception: the answer column is always kept as literal text — unless it starts with [, in which case it's decoded as a JSON array. This preserves exact answer text (trailing zeros matter for NumSigFigs, e.g. 12.40) while still letting matrix/equiv rows give their list-of-rows / list-of-working-lines answer as [[1,0],[0,1]] or ["x^2-5*x+6=0", "x=2 or x=3"] in a single cell.

JSON batch input is a plain array of row objects — no cell-encoding tricks needed, since JSON already has real types.

Output. Each row writes <output>/NNN-<slug>.xml (NNN is the row's 1-based position, <slug> is the row's id if given, else the generated question's name) plus a summary <output>/batch_report.csv with one line per row: row, id, status, output_file, validated, error. status is ok, validation_failed (XML was still written — see below), or error (nothing written for that row). Exit code is 0 only when every row succeeded.

--validate behaves like generate --validate (self-correcting against the Docker Moodle+STACK instance, PRD F-8/F-9) but checks the stack is already running up front rather than waiting — a batch is the wrong place to discover Docker isn't started after generating the first 50 questions.

Validation against a real STACK engine

cd docker && docker compose up -d --build   # Moodle 4.5 + STACK 4.12 + Maxima
stackgen generate question.json -o out/ --validate

--validate imports the generated XML into the Dockerised Moodle+STACK instance and runs the question's own STACK question tests there. When an LLM backend is in play, failures are fed back to the model with the exact STACK/Maxima error and retried (--max-attempts, default 3). stackgen eval --validate additionally runs every passing eval case's generated XML through the same check. See docker/README.md.

The inference eval set

stackgen eval                      # evals/inference_cases.json, claude-cli
stackgen eval my_cases.json --llm claude --jobs 4 -v

Runs each case's question+answer through inference and checks the inferred {type, test, tolerance} against the expected values. Exits 0 when ≥90% pass. Edit prompts/inference.md and re-run to iterate on the rules.

Eval defaults to the claude-cli backend, which shells out to a locally installed claude CLI and its Claude Code login. That backend is an eval-only developer convenience: stackgen generate refuses it, because generation is strictly bring-your-own-key (F-10).

LLM backend for generation

Two backends, chosen with --llm (config/env only — no separate feature flag):

• --llm claude — cloud, BYOK. Anthropic SDK; pip install .[llm] and set ANTHROPIC_API_KEY (environment variable or .env file). Higher quality, ≈$0.01–0.03/question.
• --llm ollama — local, zero-cost (PRD F-10/§9). Talks to a locally running ollama serve; no pip extra needed (stdlib urllib only) and no API key — just install Ollama and ollama pull <model> first. Config is env-only, same pattern as ANTHROPIC_API_KEY: OLLAMA_HOST (default http://localhost:11434) and OLLAMA_TIMEOUT_SECONDS (default 900 — local inference on CPU-only hardware can be slow; see measurements below).

Both backends implement the same LLMBackend.complete(prompt) -> str interface (backend/llm/base.py); nothing outside backend/llm/ imports a specific provider.

Natural-language generation makes two model calls. For claude they default to different models (≈$0.015/question at list prices): the inference call (classification judgment) runs on claude-sonnet-5, the feedback call (one paragraph of prose) on claude-haiku-4-5. For ollama both calls default to the same model (qwen2.5:7b) — there's no cost asymmetry locally, only a latency one, so split them yourself with --model / --feedback-model if your hardware can run two sizes at once. Outside inference, the LLM only writes free-text feedback; question structure and grading logic are always generated deterministically. System prompts live in prompts/.

Local-model quality tradeoff — current status: structured input works, natural-language input does not. Live-tested against qwen2.5:7b (2026-07-11/12, CPU-only — 16GB RAM, no dedicated GPU; Ollama drops integrated-GPU offload unless OLLAMA_IGPU_ENABLE=1 is set, and enabling it made no measurable difference on Intel Iris Xe):

• Natural-language input: 0/36 (0%). The full stackgen eval run (evals/inference_cases.json, --llm ollama --model qwen2.5:7b, ~2.5 hours wall-clock) failed every single case — not a marginal or rounded-down number. 30/36 failed with plan has invalid answer_kind None (the reply parsed as some JSON object, but not one shaped like the strict plan schema — most likely the model wraps the plan in surrounding prose/nesting that this repo's bracket-matching JSON extraction doesn't unwrap correctly); the other 6 failed with malformed JSON, reproduced live on examples/04_natural_language_braking.json: both the first attempt and infer_plan's built-in one-shot retry leaked an unescaped backslash (\(/\)) from the LaTeX math delimiters in the echoed question text into a JSON string. In both failure modes stackgen generate handled it correctly — a clean error: inference failed: ..., no file written, no crash — but the underlying result stands: --llm ollama with qwen2.5:7b cannot currently do this repo's NL inference step, against inference.md as written.
• Structured/JSON input: 13/13 (100%). Every reference example in examples/ (numerical, units, multipart, mcq, equiv) passed --llm ollama --validate against a live Docker Moodle+STACK instance, on the first attempt. Structured input (a pattern/question_text already given) skips NL inference entirely — question structure and grading logic are always generated deterministically regardless of LLM backend — so this path doesn't depend on model quality and works the same on ollama as on claude. This also confirms --llm ollama's own wiring (backend construction, the --validate self-correction loop, error handling) is sound end to end — though the repair loop specifically was never triggered by these passing examples, so its live output quality with a local model is still unverified.
• Speed (for the calls that do run): ~2.85–3.4 tokens/sec generation, ~60–90 tokens/sec prompt processing. A single call against this repo's inference.md system prompt (~2000 tokens) took 96–159s end to end.

In short: local mode is production-ready for structured input today, and not yet usable for natural-language input. Closing that gap would need either a larger/different model (untested candidates: qwen2.5:14b, models with stronger structured-output training) or dedicated prompt-engineering work on inference.md (e.g. requesting a fenced code block instead of a bare JSON object, which might extract more reliably) — neither has been attempted; both are out of scope here. --max-attempts still defaults to 5 for ollama vs. 3 for claude (--validate's repair loop, F-9), on the expectation that whatever model eventually clears the NL-inference bar will still need more self-correction than the cloud default.

Before trusting local mode for NL input with any model, run stackgen eval --llm ollama --model <model> yourself and look at the real pass rate (budget for it: at OLLAMA_NUM_PARALLEL:1, Ollama serves one request at a time regardless of --jobs, so 36 cases × ~130–250s/case is ~2–3 hours). Report back what works if you find a model that actually clears this bar.

Web frontend (PRD F-11)

A thin React + Tailwind client over a FastAPI wrapper (backend/api.py) — all grading logic still lives in backend/; the frontend only renders what the API returns.

Run locally (two terminals)

pip install ".[llm,api,validate]"
uvicorn backend.api:app --reload          # http://localhost:8000

cd frontend
npm install
npm run dev                                # http://localhost:5173

The validate extra (the docker Python SDK) is only needed for the Validate button / --validate; skip it if you don't have the Moodle+STACK stack running.

Put ANTHROPIC_API_KEY=sk-ant-... in a .env file at the repo root first (copy .env.example) — generation always needs the Claude backend (F-10).

One-command startup (Docker Compose)

cd docker
docker compose up -d --build

This starts the frontend (:5173), the backend (:8000), and the Moodle+STACK validation stack (:8080) together, so the "Validate" button in the UI works out of the box. The backend container validates by POSTing the generated XML to a small HTTP endpoint inside the moodle container (moodle:8090, see backend/validator.py and docker/scripts/validate_server.php); it mounts no Docker socket. The repo-root .env must exist — copy .env.example and add your key before starting.

⚠ This stack is a local dev/validation environment only. The backend API has no authentication, and the validation endpoint imports and runs whatever XML it is handed as a Moodle admin (the admin password is in docker/moodle/moodle-init.sh). Never deploy this compose file to a shared or production host, and never expose it to an untrusted network.

The backend container no longer mounts /var/run/docker.sock. It used to, for docker exec into the moodle container, which granted it root-equivalent control of the host Docker daemon; validating over HTTP removes that.

As the mitigation, every published port is loopback only — the backend (127.0.0.1:8000:8000), Moodle (127.0.0.1:8080:80), its validation endpoint (127.0.0.1:8090:8090) and the frontend (127.0.0.1:5173:5173) — so they are reachable from this machine but not from the LAN. The in-container uvicorn --host 0.0.0.0 is deliberate and must stay: the frontend container reaches the API as backend:8000 over the compose network, which is unaffected by the host-side loopback bind.

Security model

Read this before wiring the tool into anything.

Generated XML is author-grade content and is deliberately not sanitized. A STACK question legitimately contains active markup: [[jsxgraph]] blocks carry real author-written JavaScript, and question text carries HTML, LaTeX (\(...\), \[...\]) and CASText ({@...@}, [[...]]) whose delimiters an HTML sanitizer would corrupt. This matches how Moodle/STACK themselves work — question authors are a privileged role, trusted to embed script. Sanitizing the deliverable would silently break valid questions while still having to re-admit JSXGraph JS, so it would buy a false sense of safety.

The consequences of that choice:

• This tool is not a sanitization boundary. Treat its output exactly as you would a hand-authored STACK question: import it into Moodle only as a trusted author. Do not expose the API as an untrusted-input service, and do not feed it questions from the public.
• The preview is sandboxed. PreviewPanel renders question text and PRT feedback through frontend/src/components/SafeHtml.tsx, an iframe with sandbox="allow-same-origin" and no allow-scripts, so the author's own browser never executes that markup while reviewing it. allow-scripts must never be added: combined with allow-same-origin it lets framed content remove its own sandbox.
• User-supplied images are validated — and that is not an inconsistency. image_data_uri has exactly one legitimate shape, because a client builds it mechanically (FileReader.readAsDataURL on a picked file). So backend/api.py can require a base64 data: URI with an allowlisted raster MIME type (png, jpeg, gif, webp), and escape it with quoteattr on interpolation, without ever taking a view on which content is "safe" — image/svg+xml is rejected because SVG can carry <script>. The free-text fields (question_text, name, the feedback fields) have no comparable shape: arbitrary author HTML is their legitimate content, so they are checked for nothing. Validating a URI's structure and sanitizing HTML are different operations, and only the first can be done without breaking valid questions.
• The API has no authentication and must stay on loopback. Anyone who can reach :8000 can drive the paid LLM and the validator, which imports and runs the XML inside the Moodle container. The compose stack binds it to 127.0.0.1 (see the warning above); the documented local command uvicorn backend.api:app --reload is already loopback-only, since that is uvicorn's default host.
• Request bodies are capped at 8 MiB (MAX_BODY_BYTES), above the 5 MiB per-image cap (MAX_IMAGE_BYTES), so an unbounded xml or image field can't exhaust memory.

Automation (n8n, optional — PRD F-13)

For institutions running the API as a service, n8n/workflows/ has an importable n8n workflow template: webhook → POST /api/generate → POST /api/validate → write XML to storage, with a disabled, clearly-labelled Moodle web-service auto-import stub (import varies by institution — see PRD.md §15 risk table). See n8n/README.md for import steps, configuration, and limitations.

Layout

backend/
  models.py              # StackQuestion dataclasses mirroring qtype_stack XML
  xml_builder.py         # serialises to Moodle-importable XML
  generator.py           # inference pre-step + validate/repair loop
  validator.py           # wrapper around the docker validation instance
  patterns/              # one module per question pattern
    base_pattern.py      #   shared interface: generate(input) -> StackQuestion
    numerical_tolerance.py #   NumRelative/NumAbsolute/NumSigFigs, unitless or units
    algebraic_equivalence.py #   AlgEquiv
    units.py              #   UnitsRelative/UnitsAbsolute/UnitsStrictRelative/UnitsStrictAbsolute
    matrix.py              #   matrix/varmatrix
    multipart.py           #   several independently-graded sub-answers, one question
    mcq.py                 #   dropdown/radio/checkbox
    equiv.py                #   reasoning-by-equivalence, line-by-line working
  llm/                   # pluggable LLM backends (provider imports stay here)
    claude_backend.py    #   Anthropic SDK (ANTHROPIC_API_KEY) — cloud, BYOK
    claude_cli_backend.py#   local `claude -p` — eval-only convenience
    ollama_backend.py    #   local `ollama serve` — zero-cost (F-10)
cli/                     # `stackgen` command (generate, batch, eval)
prompts/                 # all STACK schema/LLM knowledge as editable text
  stack_schema.md        #   XML schema notes + flagged assumptions
  inference.md           #   NL -> grading-plan decision rules
evals/                   # inference eval cases for `stackgen eval`
examples/                # input JSON; `out/` is the scratch dir generators write to
tests/                   # pytest suite (tolerance inference, XML structure)
docker/                  # Moodle+STACK validation environment (--validate)
                         #   + backend/frontend Dockerfiles for one-command startup
frontend/                # React + Tailwind single-question editor (PRD F-11)
n8n/                     # optional workflow templates (PRD F-13)
  workflows/              #   importable webhook -> generate -> validate -> store

Schema fidelity

Generated XML mirrors a real qtype_stack export (stackversion 2026010500) element-for-element; see prompts/stack_schema.md for the documented schema and the assumptions that still need verification against a live Moodle.

Development

pip install pytest
python -m pytest tests

Adding a new STACK pattern (a new input type, answer test, or structural feature)? See CONTRIBUTING.md for the step-by-step recipe, including which shared files genuinely need editing and which don't.

License

GPL-3.0. See LICENSE.

Copyright (C) 2026 Aarya Gowdar