-
Notifications
You must be signed in to change notification settings - Fork 0
Setup Wizard
The Setup Wizard provides a guided, agent-first onboarding workflow for first-time users.
- Automatically on first run (no agents configured in the database)
- Manually via Setup Wizard in the sidebar navigation
Configure a Copilot Studio agent with all its Direct Line credentials and evaluation settings.
| Field | Description | Where to Find |
|---|---|---|
| Agent Name | Descriptive name, e.g. "Production Bot" | Your choice |
| Environment |
dev / staging / production
|
Your choice |
| Direct Line Secret | Web Channel security secret | Copilot Studio → Settings → Security → Web channel security |
| Bot ID | Agent identifier | Copilot Studio agent URI |
| Judge Endpoint | Azure AI Foundry or OpenAI-compatible URL | Azure Portal → AI Foundry resource → Keys and Endpoint |
| Judge API Key | API key for the judge model | Azure Portal → AI Foundry resource → Keys and Endpoint |
| Judge Model | Deployment name | Azure AI Foundry Studio → Deployments |
| Pass Threshold | Minimum score (0–1) for a test to pass | Default: 0.70
|
Check Use AI to generate questions and provide:
- Azure OpenAI endpoint URL
- Azure OpenAI API key
- Model deployment name (e.g.,
gpt-4o-mini)
If left unconfigured, rule-based question generation is used as a fallback.
Agent configuration is stored in the database. Additional agents can be added at any time from the Agents page.
Upload knowledge base documents to drive AI test generation.
| Format | Notes |
|---|---|
.pdf |
Text layer extracted automatically via PdfPig |
.txt |
Plain text |
.md |
Markdown |
File size limit: 10 MB per file. Upload multiple files if needed.
The application:
- Extracts text
- Normalizes and chunks the content (sliding window with overlap)
- Stores chunks in SQLite for question generation
You can skip this step if you plan to define test cases manually.
Specify how many questions to generate (5–50) and click Generate Questions.
AI generation (if configured in Step 1):
- Azure OpenAI reads document chunks and produces contextually rich questions
- Each question includes suggested expected answer, intent, and entities
- Questions span factual recall, conceptual understanding, application, and edge cases
Rule-based fallback:
- Key sentences are extracted from chunks
- Questions are wrapped using a library of starters ("What can you tell me about…", "How does…", etc.)
Actions available:
- Select/deselect individual questions
- Select all
- Regenerate (new set)
- Add a custom question manually
Provide:
- Suite Name (required) — e.g., "Product Knowledge Q1"
- Description (optional)
The suite is automatically associated with the agent created in Step 1.
Click Run Tests. The application:
- Connects to the agent via Direct Line (WebSocket or polling)
- Sends each test question
- Collects responses
- Evaluates each response using the AI Judge
- Displays results immediately
Result metrics:
- Passed / Failed counts and pass rate
- Average response latency
- Per-test: question, verdict, judge rationale, overall score, latency, token count
| Next Step | Where |
|---|---|
| Add more agents (dev, staging, prod) | Agents → New Agent |
| Associate a suite with additional agents | Test Suites → Edit → Agents tab |
| Run the same suite against multiple agents | Test Suites → Run |
| View run history and compare agents | Dashboard / Runs |
| Configure team authentication | Authentication |
- Use descriptive agent names that include the environment: "Prod — Customer FAQ"
- Upload documents that reflect your agent's actual knowledge domain
- Generate more questions than needed, then hand-pick the best ones
- Add edge-case questions manually to complement AI-generated ones
- After the wizard, run the same suite against a staging agent to compare before promotion