Question Generation

Question Generation

The application can automatically generate test cases from uploaded documents using AI or a rule-based fallback.

---

Two Modes

Mode	Description	Requirement
AI-Powered	Uses an Azure OpenAI (or OpenAI-compatible) LLM to understand document context and produce high-quality, diverse questions	Azure OpenAI endpoint and API key
Rule-Based	Extracts key sentences from document chunks and wraps them in question starters	None — always available as fallback

---

AI-Powered Generation

How It Works

1. Document chunks are collected and concatenated (truncated to ~12 000 characters to stay within model context limits)
2. A structured prompt asks the model to generate N questions covering:
   • Direct factual questions (What/When/Where)
   • Conceptual understanding (How/Why)
   • Comparison and contrast
   • Application and troubleshooting
   • Edge cases
3. Each question includes:
   • Question text
   • Expected answer — a concise, accurate answer from the documents
   • Expected intent — the user's goal
   • Expected entities — key terms or concepts
4. Questions are returned as a typed list and imported as test cases

Configuration

Enable AI generation in the Setup Wizard (Step 1) or in the global/per-agent settings:

{
  "QuestionGeneration": {
    "Endpoint": "https://your-resource.openai.azure.com/",
    "ApiKey": "<env-var>",
    "Model": "gpt-4o-mini"
  }
}

Environment variables:

QUESTIONGENERATION__ENDPOINT=https://your-resource.openai.azure.com/
QUESTIONGENERATION__APIKEY=<api-key>
QUESTIONGENERATION__MODEL=gpt-4o-mini

Cost Estimate

Model	Cost per generation batch (5–10 questions)
gpt-4o-mini	~$0.001–$0.002
gpt-4o	~$0.02–$0.04

---

Rule-Based Fallback

When no AI endpoint is configured, the rule-based generator:

1. Randomly samples document chunks
2. Extracts sentences meeting minimum length thresholds
3. Wraps each sentence in one of several question starters:
   • "What can you tell me about…"
   • "Can you explain…"
   • "Tell me about…"
   • "How does…"
   • etc.
4. Sets the source sentence as the expected answer reference

This produces functional (but less contextually rich) test cases without requiring any API access.

---

Generating Questions

Via the Setup Wizard

1. Upload documents in Step 2
2. In Step 3, set the number of questions (5–50)
3. Click Generate Questions
4. Review, select, or deselect individual questions
5. Add custom questions manually with the Add Custom button
6. Proceed to Step 4 to create the test suite

Via the Documents Page

1. Navigate to Documents
2. Click Generate Questions on a document
3. Select questions and add to an existing test suite

Via the CLI

Use the generate command to create test cases from a local file entirely from the terminal:

# Preview generated questions without saving
testrunner generate --document ./docs/product-manual.txt --count 10

# Generate and save directly into a suite
testrunner generate --document ./docs/product-manual.txt --suite "Regression Tests" --count 20

See CLI Reference for the full option reference.

---

Reviewing Generated Questions

For each generated question you can:

• ✅ Select/deselect it for inclusion
• ✏️ Edit the question text or expected answer before saving
• ➕ Add a custom question at any time
• 🔄 Regenerate a completely new set

---

Customising the AI Prompt

The question-generation system prompt can now be customised directly in the UI — no code changes required.

Global Prompt (Settings page)

1. Navigate to Settings → AI Question Gen
2. Scroll to the System Prompt field
3. Click Load default to edit to load the built-in prompt into the textarea
4. Edit as needed and click Save
5. Leave the field empty to revert to the built-in default at any time (click Revert to built-in)

The global prompt applies to all question-generation calls unless a per-agent override is set.

Per-Agent Prompt (Agents page)

1. Navigate to Agents → create or edit an agent
2. In the Question Generation section, uncheck Use global question generation settings
3. Use the System Prompt Override field — click Load default to edit to start from the built-in template
4. Save the agent

Prompt resolution order:

Agent-level system prompt
    │  (if set)
    └── Global system prompt (Settings → AI Question Gen)
            │  (if set)
            └── Built-in hardcoded default

Prompt Customisation Tips

To adjust question type distribution, add to the prompt:

Generate questions with this distribution:
- 40% factual recall
- 30% conceptual understanding
- 20% application/troubleshooting
- 10% edge cases

To include domain-specific context:

Domain: Customer support for a SaaS product
Focus on subscription management, billing, and onboarding workflows.

The {domainContext} placeholder in the built-in prompt is automatically replaced at generation time with the agent or suite name — you do not need to add it manually.

---

Troubleshooting

Issue	Likely Cause	Fix
"Failed to parse response"	Wrong model deployment name	Check model name matches the deployment in Azure
401 Unauthorized	Invalid API key	Regenerate key in Azure portal
429 Rate limit	Too many requests	Wait a few seconds and retry
No questions generated	Document too short	Upload a longer document (≥ 500 words recommended)
Poor question quality	Generic prompt	Add domain context in Setup Wizard agent name

---

See Also

• Document Processing — upload and chunk documents
• Setup Wizard — integrated generation during wizard
• Test Suites and Cases — using generated questions as test cases