CLI Reference

CLI Reference

The CLI provides all core test execution capabilities for integration into CI/CD pipelines.

---

Installation

No separate installation required. The CLI is built as part of the solution:

dotnet build

Run from the project:

cd CopilotStudioTestRunner.CLI
dotnet run -- <command> [options]

Or from the published binary:

CopilotStudioTestRunner.CLI.exe <command> [options]

---

Commands

list — List Test Suites

dotnet run -- list

Outputs a table of all available test suites with their names and test case counts.

---

run — Execute a Test Suite

dotnet run -- run --suite "<suite-name>" [options]

Options:

Option	Required	Default	Description
--suite <name|id>	Yes	—	Name or GUID of the test suite to run
--output <path>	No	./results	Directory to write JSON result files
--dry-run	No	false	Preview what would execute without calling Direct Line or AI Judge
--config <path>	No	../appsettings.json	Path to appsettings.json

Examples:

# Run against all associated agents
dotnet run -- run --suite "Regression Tests" --output ./results

# Dry run — shows test cases without executing
dotnet run -- run --suite "Regression Tests" --dry-run

# CI/CD with explicit config and output path
dotnet run -- run --suite "Smoke Tests" --output ./ci-results --config ./appsettings.json

---

agents — List Configured Agents

dotnet run -- agents [options]

Outputs a table of all agents with name, environment, active status, and ID.

Options:

Option	Required	Default	Description
--config <path>	No	../appsettings.json	Path to appsettings.json

Example output:

Configured Agents:
--------------------------------------------------------------------------------
  Production Bot                 [production] active   (ID: 3fa85f64-...)
  Staging Bot                    [staging   ] active   (ID: 7c9e6679-...)
  Dev Bot                        [dev       ] inactive (ID: 1a2b3c4d-...)

  Total: 3 agent(s)

---

report — Export a Completed Run

dotnet run -- report --run <run-id> [options]

Exports the results of a completed run as JSON (default) or CSV without re-running anything.

Options:

Option	Required	Default	Description
--run <id>	Yes	—	GUID of the completed run to export
--format <json|csv>	No	json	Output format
--output <path>	No	./results	Directory to write the report file
--config <path>	No	../appsettings.json	Path to appsettings.json

Examples:

# Export as JSON (default)
dotnet run -- report --run 3fa85f64-5717-4562-b3fc-2c963f66afa6

# Export as CSV for spreadsheet analysis
dotnet run -- report --run 3fa85f64-5717-4562-b3fc-2c963f66afa6 --format csv --output ./exports

The CSV columns are: TestCase, AIVerdict, HumanVerdict, Score, LatencyMs, JudgeRationale.

---

generate — Generate Test Cases from a Document

dotnet run -- generate --document <path> [options]

Uses the configured judge LLM to generate test cases from a local document file. When --suite is provided the test cases are saved directly to that suite; otherwise they are printed to the console for review.

Options:

Option	Required	Default	Description
--document <path>	Yes	—	Path to a .txt, .md, or .pdf file
--suite <name|id>	No	—	Name or GUID of the suite to save into; omit to print only
--count <n>	No	5	Number of questions to generate
--config <path>	No	../appsettings.json	Path to appsettings.json

Requires: Judge:Endpoint and Judge:ApiKey configured (same credentials as the Web UI judge).

Examples:

# Preview 10 questions without saving
dotnet run -- generate --document ./docs/product-manual.txt --count 10

# Generate and save directly into a suite
dotnet run -- generate --document ./docs/product-manual.txt --suite "Regression Tests" --count 20

# Use a specific config file
dotnet run -- generate --document ./docs/faq.md --suite "FAQ Suite" --config ./appsettings.json

Generated test cases are flagged IsGenerated = true in the database so you can distinguish them from manually authored cases.

---

Exit Codes

Code	Meaning
0	All tests passed
1	One or more tests failed, or an error occurred

Use exit codes in CI/CD pipelines:

# GitHub Actions example
- name: Run regression tests
  run: dotnet run --project CopilotStudioTestRunner.CLI -- run --suite "Regression" --output ./results
  working-directory: .

- name: Check results
  if: failure()
  run: echo "Tests failed — check ./results for details"

---

Output Files

Results are written to the --output directory as JSON files:

results/
  run-<suite>-<agent>-<timestamp>.json

JSON Structure

{
  "runId": "...",
  "suiteId": "...",
  "suiteName": "Regression Tests",
  "agentId": "...",
  "agentName": "Production Bot",
  "startedAt": "2026-02-19T10:00:00Z",
  "completedAt": "2026-02-19T10:05:00Z",
  "status": "completed",
  "totalTestCases": 20,
  "passedCount": 18,
  "failedCount": 2,
  "passRate": 0.90,
  "averageLatencyMs": 1250,
  "results": [
    {
      "testCaseName": "...",
      "verdict": "pass",
      "overallScore": 0.87,
      "taskSuccessScore": 0.9,
      "intentMatchScore": 1.0,
      "factualityScore": 0.85,
      "helpfulnessScore": 0.8,
      "safetyScore": 1.0,
      "latencyMs": 1100,
      "rationale": "..."
    }
  ]
}

---

Configuration for the CLI

The CLI reads configuration from the same appsettings.json as the Web UI. For CI/CD environments, provide credentials via environment variables:

# PowerShell
$env:DIRECTLINE__SECRET          = "<secret>"
$env:DIRECTLINE__BOTID           = "<bot-id>"
$env:JUDGE__ENDPOINT             = "https://your-resource.openai.azure.com/"
$env:JUDGE__APIKEY               = "<api-key>"
$env:JUDGE__MODEL                = "gpt-4o-mini"
$env:STORAGE__SQLITEPATH         = "./data/app.db"

dotnet run --project CopilotStudioTestRunner.CLI -- run --suite "Regression Tests"

If Authentication:Enabled is true, the CLI can acquire a token using Device Code flow (interactive) or Client Credentials flow (non-interactive CI/CD). See Authentication → CLI Authentication for details.

---

CI/CD Integration Examples

GitHub Actions

name: Copilot Studio Tests

on: [push, pull_request]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-dotnet@v4
        with:
          dotnet-version: 9.0.x
      
      - name: Build
        run: dotnet build --configuration Release
      
      - name: Run Smoke Tests
        env:
          DIRECTLINE__SECRET: ${{ secrets.DL_SECRET }}
          DIRECTLINE__BOTID: ${{ secrets.BOT_ID }}
          JUDGE__ENDPOINT: ${{ secrets.AI_ENDPOINT }}
          JUDGE__APIKEY: ${{ secrets.AI_KEY }}
        run: |
          dotnet run --project CopilotStudioTestRunner.CLI --configuration Release -- \
            run --suite "Smoke Tests" --output ./results
      
      - name: Upload Results
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: test-results
          path: ./results/

Azure DevOps

- task: DotNetCoreCLI@2
  displayName: Run Copilot Tests
  inputs:
    command: run
    projects: CopilotStudioTestRunner.CLI/CopilotStudioTestRunner.CLI.csproj
    arguments: '-- run --suite "Regression Tests" --output $(Build.ArtifactStagingDirectory)/results'
  env:
    DIRECTLINE__SECRET: $(DL_SECRET)
    JUDGE__ENDPOINT: $(AI_ENDPOINT)
    JUDGE__APIKEY: $(AI_KEY)