Skip to content
/ Jarvis Public

AI-powered Microsoft Teams meeting assistant with voice interaction, conversation memory, and intelligent responses. Built with Azure OpenAI, Speech Services, and Bot Framework. Includes complete Bicep IaC for automated Azure deployment.

github.com/gregunger-microsoft/Jarvis/blob/main/README.md

License

MIT license
1 star 0 forks Branches Tags Activity
Star
Notifications You must be signed in to change notification settings

gregunger-microsoft/Jarvis

BranchesTags

Repository files navigation

Jarvis – AI Meeting Assistant

AI-powered Microsoft Teams meeting assistant with voice interaction, conversation memory, and intelligent responses.

What Is Jarvis?

Jarvis is an AI meeting assistant that joins Teams meetings with audio and chat capabilities, providing:

  • 🎤 Voice Interaction - Responds to "Hey Jarvis" wake phrase
  • 💬 Chat Integration - Answers @mentions in meeting chat
  • 🧠 Conversation Memory - Remembers context within meetings
  • 🤖 AI-Powered - Uses Azure OpenAI GPT-4.1 mini
  • 🔊 Speech Processing - Azure Speech Services for STT/TTS

Built on:

  • Microsoft Bot Framework (messaging)
  • Microsoft Graph Calling API (audio/video)
  • Azure Speech Services (wake word, STT, TTS)
  • Azure OpenAI (intelligent responses)
  • ASP.NET Core 8.0

Quick Start

For Users

Want to use Jarvis? Start here:

  1. User Guide - Complete guide to using Jarvis
  2. Adding Jarvis to Meetings - How to add Jarvis to Teams meetings
  3. Quick Start - Quick overview and basic usage

For Developers/Admins

Want to deploy or modify Jarvis? Start here:

🏗️ Azure Infrastructure (Bicep IaC)

  1. Azure Deployment Strategy - Complete infrastructure architecture
  2. Azure Deployment Guide - Step-by-step Bicep deployment
  3. Bicep Templates - Infrastructure as Code (17 Azure resources)

🚀 Application Deployment

  1. Runtime Deployment - Deploy app to VM
  2. VM Host Setup - Azure VM configuration
  3. Environment Variables - Configuration reference
  4. Operations Guide - Running and maintaining Jarvis

Documentation Inventory

📚 User Documentation

Document Description
User Guide Complete guide to using Jarvis - voice commands, chat, memory
Adding to Meetings Simple steps to add Jarvis to your Teams meetings
Quick Start Quick overview and getting started guide

🏗️ Architecture & Design

Document Description
Architecture System architecture, components, data flow
Diagrams Mermaid sequence and component diagrams
Overview Feature matrix, problems solved, technical details
Executive Summary Business value proposition and ROI

🏗️ Infrastructure & Deployment

Document Description
Azure Deployment Strategy Complete Azure infrastructure architecture and planning
Azure Deployment Guide Step-by-step Bicep IaC deployment (recommended)
Bicep Validation Report Infrastructure validation results
Bicep Templates 11 Bicep modules for 17 Azure resources
Runtime Deployment Deploy application to Azure VM
VM Host Setup Azure VM configuration and requirements

⚙️ Configuration

Document Description
Environment Variables Complete configuration reference
Configuration Guide Application settings and options
Teams App Teams app manifest and installation

🔧 Operations & Maintenance

Document Description
Operations Guide Running, monitoring, and maintaining Jarvis
Logging How to access and query logs (dev and production)
Security Security considerations and best practices
Azure Resources Azure resources and dependencies

🚀 Features & Functionality

Document Description
Conversation Memory How conversation memory works
Leave Meeting Leave meeting functionality and commands

📋 Reference

Document Description
Index Master documentation index
Meeting Agenda Project planning and milestones

⚠️ Security Notice

Never commit secrets to git! This repository includes template files with placeholders.

First-Time Setup

Copy template files to create your local configuration:

Copy-Item src/MeetingBot/appsettings.json.template src/MeetingBot/appsettings.json
Copy-Item src/MeetingBot/appsettings.Development.json.template src/MeetingBot/appsettings.Development.json

Then replace all YOUR-*-HERE placeholders with actual values from your Azure resources.

See: SETUP.md for complete setup instructions.

Important: The .gitignore file prevents committing these files. Use environment variables or Azure Key Vault for production deployments.

Key Features

Voice Interaction

  • Wake Phrase Detection: Responds to "Hey Jarvis"
  • Natural Language: Ask questions in natural language
  • Voice Responses: Speaks answers via Azure TTS
  • Fuzzy Matching: Recognizes common misheard variations

Conversation Memory

  • Context Awareness: Remembers last 10 exchanges
  • Follow-up Questions: No need to repeat context
  • Personalization: Remembers preferences within meeting
  • Time-Based: 30-minute memory window

Chat Integration

  • @Mentions: Responds to @Jarvis in chat
  • Text Commands: All voice commands work in chat
  • Dual Response: Voice commands also post to chat

Meeting Control

  • Leave Commands: "Hey Jarvis, leave the meeting"
  • Confirmation Flow: Confirms before leaving
  • Immediate Leave: "leave now" bypasses confirmation
  • Cancel Support: Can cancel leave requests

Infrastructure

Azure Resources (Bicep Templates)

Jarvis uses Infrastructure as Code (Bicep) to deploy 17 Azure resources:

Category Resources
Compute Virtual Machine (Windows Server 2022), Managed Identity
AI Services Azure OpenAI (GPT-4o-mini), Speech Services (STT/TTS)
Bot Platform Bot Framework Service (Teams channel)
Monitoring Application Insights, Log Analytics, 10 Alert Rules, 3 Availability Tests
Security Key Vault, Network Security Group
Storage Storage Account (diagnostics, logs, backups)
Networking Virtual Network, Public IP with DNS

Cost Estimate: ~$975/month (prod) | ~$200/month (dev)

Deployment Time: 30-45 minutes (fully automated)

Bicep Modules

bicep/
├── main.bicep                    # Subscription-level orchestrator
├── modules/
│   ├── network.bicep             # VNet, NSG, Public IP
│   ├── compute.bicep             # VM with Azure Monitor agent
│   ├── cognitive.bicep           # OpenAI + Speech Services
│   ├── bot-service.bicep         # Bot Framework Service
│   ├── monitoring.bicep          # App Insights + Log Analytics
│   ├── alerts.bicep              # Action Groups + Alert Rules
│   ├── availability-tests.bicep  # Synthetic monitoring
│   ├── keyvault.bicep            # Secrets storage
│   ├── storage.bicep             # Blob storage
│   └── identity.bicep            # RBAC assignments
└── parameters/
    ├── dev.parameters.json       # Dev environment
    ├── staging.parameters.json   # Staging environment
    └── prod.parameters.json      # Production environment

See: Azure Deployment Strategy for complete architecture

Configuration

Required Environment Variables

Variable Purpose
BOT_APP_ID Azure AD App (Client) ID
BOT_APP_PASSWORD Client secret (never commit!)
BOT_BASEURL Public HTTPS URL (e.g., https://jarvis.aicollaborator.net)
AZURE_SPEECH_REGION Azure Speech Service region (e.g., eastus)
AZURE_SPEECH_KEY Azure Speech Service API key
AZURE_OPENAI_ENDPOINT Azure OpenAI endpoint URL
AZURE_OPENAI_DEPLOYMENT OpenAI model deployment name
AZURE_OPENAI_KEY Azure OpenAI API key
CERTIFICATE_THUMBPRINT Certificate thumbprint for media/calling

Optional Configuration

Variable Default Purpose
JARVIS_CONVERSATION_MEMORY_TURNS 10 Number of exchanges to remember
AZURE_OPENAI_MAX_TOKENS 350 Max tokens for responses
SPEECH_INITIAL_SILENCE_MS 2000 STT initial silence timeout
SPEECH_END_SILENCE_MS 500 STT end silence timeout

See: Environment Variables for complete list

Local Development

Prerequisites

  • .NET 8.0 SDK
  • Azure subscription (Speech + OpenAI resources)
  • Bot registration in Azure
  • Teams app manifest installed

Build and Run

# Restore dependencies
dotnet restore JarvisMeetingAssistant.sln

# Build solution
dotnet build JarvisMeetingAssistant.sln -c Release

# Run locally (messaging only, no calling)
$env:ASPNETCORE_URLS = 'http://localhost:8080'
dotnet run --project src/MeetingBot/MeetingBot.csproj -c Release

Testing

# Run unit tests
dotnet test tests/MeetingBot.Tests/MeetingBot.Tests.csproj

# Check build (no errors, no warnings)
dotnet build --no-incremental

Deployment

🏗️ Infrastructure Deployment (Bicep IaC)

Recommended approach - Deploy all 17 Azure resources using Infrastructure as Code:

# 1. Configure parameters
notepad bicep/parameters/prod.parameters.json
# Set: adminSourceIP, adminPassword, customDomain

# 2. Deploy Azure infrastructure (~30-45 minutes)
cd scripts/infra
.\Deploy-Infrastructure.ps1 -Environment prod

# 3. Create Entra ID app registration
.\Create-EntraApp.ps1 -Environment prod -GrantAdminConsent

# 4. Validate deployment
.\Validate-Deployment.ps1 -Environment prod

# 5. Configure VM (RDP to VM)
.\Configure-VM.ps1

Complete Guide: Azure Deployment Guide

Resources Created:

  • Virtual Machine (Windows Server 2022)
  • Azure OpenAI + Speech Services
  • Application Insights + Log Analytics
  • Key Vault, Storage, Networking
  • Bot Service + Alerts + Monitoring

🚀 Application Deployment

After infrastructure is deployed, deploy the Jarvis application:

# 1. Build and publish
dotnet publish src/MeetingBot/MeetingBot.csproj -c Release -o .\publish\JarvisApp

# 2. Package
Compress-Archive -Path .\publish\JarvisApp\* -DestinationPath .\JarvisApp.zip -Force

# 3. Copy to VM and extract to C:\Jarvis\JarvisApp

# 4. Set environment variables (on VM)
.\scripts\Set-CanonicalEnv.ps1

# 5. Install as Windows Service
New-Service -Name JarvisMeetingBot -BinaryPathName "C:\Jarvis\MeetingBot.exe"
Start-Service JarvisMeetingBot

Detailed Instructions: Runtime Deployment Guide

Architecture

High-Level Components

┌──────────────────────────────────────────────────────────┐
│                    Microsoft Teams                       │
│  (User speaks "Hey Jarvis..." or types @Jarvis)         │
└───────────────────────┬──────────────────────────────────┘
                        │
                        ↓
┌──────────────────────────────────────────────────────────┐
│              Microsoft Graph Calling API                 │
│  (Audio/Video stream + Chat messages)                    │
└───────────────────────┬──────────────────────────────────┘
                        │
                        ↓
┌──────────────────────────────────────────────────────────┐
│                  Jarvis Bot Service                      │
│  ┌─────────────┐  ┌──────────────┐  ┌─────────────────┐ │
│  │  Media      │  │  Speech      │  │  Agent          │ │
│  │  Pipeline   │→ │  Recognition │→ │  Orchestrator   │ │
│  └─────────────┘  └──────────────┘  └─────────────────┘ │
└───────────────────────┬──────────────────────────────────┘
                        │
                        ↓
┌──────────────────────────────────────────────────────────┐
│                   Azure Services                         │
│  ┌──────────────┐  ┌────────────┐  ┌──────────────────┐ │
│  │  Speech STT  │  │  OpenAI    │  │  Speech TTS      │ │
│  │  (Recognize) │  │  (Respond) │  │  (Synthesize)    │ │
│  └──────────────┘  └────────────┘  └──────────────────┘ │
└──────────────────────────────────────────────────────────┘

See: Architecture Document for detailed diagrams

API Endpoints

Endpoint Method Purpose
/healthz GET Health check (liveness probe)
/api/messages POST Bot Framework messaging endpoint
/api/calling POST Graph Calling webhook
/api/calling/callback POST Graph Calling event notifications
/api/diagnostics/status GET Runtime status and metrics

Teams App Integration

Install Jarvis in Teams

# Package the Teams app
.\scripts\Package-TeamsApp.ps1

# Upload JarvisTeamsApp.zip to Teams:
# Teams → Apps → Manage your apps → Upload an app → Upload a custom app

Add to Meeting

  1. During Meeting: People → Add people → "Jarvis Assistant" → Add
  2. Before Meeting: Add "Jarvis Assistant" as meeting attendee
  3. Via Script: Run .\scripts\CreateAndJoinMeeting-Delegated.ps1

Detailed Instructions: Adding Jarvis to Meetings

Logging & Monitoring

Development

  • Console Output: Real-time logs in terminal
  • Debug Output: Visual Studio debug window

Production

  • Application Insights: Primary logging (query with KQL)
  • Console Fallback: If App Insights unavailable
  • Windows Event Viewer: System-level events

Access Logs: See Logging Guide

Common Operations

Update Configuration

# On Azure VM
[Environment]::SetEnvironmentVariable("AZURE_OPENAI_MAX_TOKENS", "500", "Machine")
Restart-Service JarvisMeetingBot

View Logs

// Application Insights (Azure Portal)
traces
| where timestamp > ago(1h)
| where message contains "Jarvis"
| order by timestamp desc

Update Bot Endpoint

az bot update `
  --name JarvisMeetingBot `
  --resource-group jarvis-rg `
  --endpoint "https://jarvis.aicollaborator.net/api/messages"

Troubleshooting

Jarvis Doesn't Respond to Voice

  1. Check Jarvis is in participants list
  2. Wait 10 seconds after adding
  3. Check microphone isn't muted
  4. Try: "Hey Jarvis, hello"
  5. Check logs for [STT Final] entries

Jarvis Doesn't Join Meeting

  1. Verify Azure VM is running
  2. Check certificate is valid
  3. Verify URL is accessible
  4. Review Azure VM logs
  5. Check firewall rules

Complete Troubleshooting: Operations Guide

Security Considerations

  • Never commit secrets to git (use environment variables)
  • Use Key Vault for production secrets
  • Certificate rotation: Plan for cert expiry
  • HTTPS only: All endpoints require valid SSL
  • Firewall rules: Restrict inbound to necessary ports
  • Compliance: All conversations are logged

See: Security Guide

Project Structure

Jarvis/
├── bicep/                   # Azure Infrastructure as Code (Bicep)
│   ├── main.bicep           # Main orchestrator
│   ├── modules/             # 10 resource modules
│   ├── parameters/          # Environment configs (dev/staging/prod)
│   ├── AZURE_DEPLOYMENT_GUIDE.md  # Complete deployment guide
│   └── VALIDATION_REPORT.md       # Validation results
├── docs/                    # Complete documentation
│   ├── AZURE_DEPLOYMENT_STRATEGY.md  # Infrastructure architecture
│   └── ...                  # 20+ documentation files
├── src/MeetingBot/          # Main application code
│   ├── Bot/                 # Bot logic and call handlers
│   ├── Services/            # Core services (orchestrator, pipeline, etc.)
│   ├── Controllers/         # API endpoints
│   ├── Media/               # Audio processing
│   └── Program.cs           # Application entry point
├── tests/                   # Unit and integration tests
├── scripts/                 # Deployment and utility scripts
│   └── infra/               # Infrastructure deployment scripts
│       ├── Deploy-Infrastructure.ps1
│       ├── Create-EntraApp.ps1
│       ├── Configure-VM.ps1
│       └── Validate-Deployment.ps1
├── manifest/                # Teams app manifest and icons
└── publish/                 # Build output directory

Contributing

Development Workflow

  1. Create feature branch
  2. Make changes (follow existing patterns)
  3. Test locally (unit tests + manual testing)
  4. Build clean (no errors, no warnings)
  5. Update docs if adding features
  6. Submit for review

Code Standards

  • ✅ No build errors or warnings
  • ✅ Follow existing code style
  • ✅ Add unit tests for new features
  • ✅ Update documentation
  • ✅ Never commit secrets

Support & Resources

Documentation

External Resources

License

Internal project - follow organizational guidelines for licensing and usage.

Recent Updates

November 14, 2025:

Infrastructure & Deployment

  • ✅ Complete Bicep Infrastructure as Code - 11 modules, 17 Azure resources
  • ✅ Automated Deployment Scripts - One-command infrastructure deployment
  • ✅ Azure Deployment Guide - Comprehensive step-by-step guide
  • ✅ Infrastructure Validation - All templates validated and tested
  • ✅ Deployment Strategy Document - Complete architecture planning

Features & Improvements

  • ✅ Enhanced wake phrase detection with fuzzy matching
  • ✅ Added conversation memory (remembers last 10 exchanges)
  • ✅ Improved leave meeting functionality
  • ✅ Fixed media platform logging errors
  • ✅ Updated all documentation
  • ✅ Canonical environment variable enforcement

See: JARVIS_MEETING_AGENDA.md for project history

Security Reminders

  • Do not store credentials in versioned files—prefer User Secrets, env vars, or Azure Key Vault
  • docs/AZURE_RESOURCES_CREATED.md should never contain live secrets; treat it as a template
  • Rotate app secrets regularly; consider Managed Identity for production Graph calls where possible

Environment Variable Scope Policy: All required runtime environment variables MUST be set at the Windows Machine scope to ensure persistence across service restarts and scheduled tasks. The script scripts/Set-CanonicalEnv.ps1 enforces Machine scope exclusively.

Platform: Microsoft Teams + .NET 8
Infrastructure: Bicep IaC (11 modules, 17 Azure resources)
Deployment: Fully automated via PowerShell + Bicep
Documentation: 25+ guides covering user, dev, ops, and infrastructure

About

AI-powered Microsoft Teams meeting assistant with voice interaction, conversation memory, and intelligent responses. Built with Azure OpenAI, Speech Services, and Bot Framework. Includes complete Bicep IaC for automated Azure deployment.

github.com/gregunger-microsoft/Jarvis/blob/main/README.md

Topics

csharp bot-framework speech-recognition dotnet-core infrastructure-as-code microsoft-teams voice-assistant azure-deployment conversational-ai bicep teams-bot ai-assistant meeting-assistant azure-openai azure-speech teams-integration conversational-ai-chatbot

Resources

Readme

License

MIT license

Security policy

Security policy
Activity

Stars

1 star

Watchers

1 watching

Forks

0 forks
Report repository

Releases

No releases published

Packages

No packages published

Languages