External Threat Detection and Protection for Copilot Studio 🛡️

Introduction ℹ️

This repository demonstrates how to implement the new "external threat detection and protection for Copilot Studio custom agents" feature.

It lets you add a custom security layer to your AI agents by calling an external webhook that evaluates each interaction and decides whether to allow or block it.

This repo uses a simple Python Flask server to implement the webhook contract.

Contact 📬

If you're having trouble setting up this repo or want to chat about the concept, feel free to reach out on LinkedIn.

Setting Up The Webhook 🔌

This feature is based on a webhook (an HTTPS endpoint) that Copilot Studio calls. Your webhook receives request details and returns a decision to block or allow the tool execution.

In this repo, the webhook endpoints are served by this app (default port 5000):

• POST /validate – health/verification endpoint
• POST /analyze-tool-execution – evaluates a tool execution and returns a decision
• GET /health – simple health check

Deploy this server somewhere reachable and configure the URL in Copilot Studio as your "External security provider" webhook endpoint.

Azure Configuration ⚙️

To use this feature, follow the official guide:

• External security provider setup

When configuring the application in Microsoft Entra (Azure AD):

• Ensure your app registration is set correctly for the expected audience (ENDPOINT_URL)
• Make sure the application exposes an API ("Expose an API") for your endpoint URL audience value

Validation ✅

The webhook interface is described here.

In this repo, the main decision point is:

• POST /analyze-tool-execution
  • The server extracts the user message and internal thought from the request body
  • It validates the request (by default, via validator.py using the OpenAI Responses API with gpt-3.5-turbo)
  • It returns a JSON response with:
    • blockAction: boolean – whether to block the tool execution

Example decision response:

{
  "blockAction": true,
}

Response time requirements ⏱️

The agent expects a response from the threat detection system within less than 1,000 ms. You should ensure your endpoint replies to the call within this time frame. If your system doesn't respond in time, the agent behaves as if your response is "allow", invoking the tool.

Note: In my opinion, the 1000ms limit is quite strict and MUST be reconsidered.

This is why, I used the gpt-3.5-turbo model for validation: it provides a good balance of speed and accuracy, helping ensure the endpoint can reliably respond within the required time window.

Running The App ▶️

python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

# Set required environment variables
export ENDPOINT_URL="https://your-public-endpoint"   # expected audience for JWTs
export TENANT_ID="<your-entra-tenant-guid>"

# Use OpenAI Responses API for validation
export OPENAI_API_KEY="sk-..."

python app.py