Defining Semantic Filtering Policy to discover tool definitions.

[NaveenSandaruwan]

1. The Problem Statement

Currently, LLM providers (like OpenAI, Anthropic, or Gemini) support native tool-calling where tools are passed in a specific tools array. However, developers often bypass this by:

• Defining tools directly in the System Prompt.

• Using custom JSON structures inside the User Query.

This makes it difficult for API platform creators to enforce policies or "filter" which tools an agent is allowed to access. We need a standardized way to define where these tools "live" in a request so we can apply semantic filtering.

2. Proposed Scenarios & Solutions

I am considering two primary methods for API creators to define tool locations. I’d love your feedback on which is more extensible.

Scenario A: JSON Path Targeting (For Structured Tools)
If the tools are sent via the standard API body or a custom JSON object, we could allow creators to define a JSON Path.

The Setup: An API creator wants to ensure the financial_plugin isn't used by unauthorized users.

The Definition: $.tools[*].function.name

Example Request:

{
  "model": "gpt-4",
  "messages": [...],
  "tools": [
    { "type": "function", "function": { "name": "get_weather" } },
    { "type": "function", "function": { "name": "transfer_funds" } }
  ]
}

Scenario B: Regex-Based Prompt Analysis (For System Prompts)
When tools are described in plain text (e.g., "You have access to the following tools: [ToolA, ToolB]"), we need to extract them from the string.

• The Setup: A developer puts tool definitions in the System Message.

• The Definition: (?i)tools?:\s*\[(.*?)\] or Available tools:\s*([a-zA-Z0-9_, ]+)

• Example Request:

Role: System
Content: "You are a helpful assistant. Available tools: search_web, update_database."

Scenario C: Semantic Prompt Analysis (Embedding-Based)
When system prompts are lengthy or complex, traditional Regex often fails because tool descriptions are woven into conversational instructions. In this scenario, we use vector similarity to "detect" tool-like capabilities within a prompt.

• The Setup: A platform admin wants to prevent "Database Write" actions, but the developer might describe the tool as "You can update records in the cloud" instead of using a specific keyword.

• The Method:

1. Chunking: Divide the incoming System Prompt into smaller semantic chunks (e.g., 200–500 characters).
2. Vector Comparison: Generate embeddings for these chunks and compare them against a "Blocked Capabilities" vector database (e.g., embeddings of phrases like "write to database," "delete files," or "execute shell commands").
3. Threshold Extraction: If a chunk exceeds a similarity threshold (e.g., 0.85 cosine similarity), the gateway flags it as a tool-calling attempt.

• Example Request:

    Role: System
    
    Content: "You are a senior admin bot. If the user asks for data persistence, you have the ability to modify the underlying SQL          entries to reflect their changes."
  

Comparison

Method	Best For	Pros	Cons
JSON Path	Native Tool-Calling	High Precision, Fast	Only works for structured data
Regex	Standardized Prompts	Zero Latency, Simple	Brittle; easily bypassed by rephrasing
Semantic (Embed)	Complex/Hidden Tools	Catches "Hidden" Intent	High Latency, requires Vector DB

3. Questions for the Community

• Complexity: Is Regex too brittle for prompt analysis?

• User experience: Will it be difficult to define tools paths or regex patterns?

• Standardization: Is there an existing schema (like OpenAPI or JSON Schema) that we should force developers to use for tool definitions to make filtering easier?

[NaveenSandaruwan]

I resolved above problem by introducing new parameters. First decide whether tools list and user query is resides in text content or as a JSON list by userQueryIsJson and toolsIsJson then ask where is those content in the request bidy as queryJSONPath and toolsJSONPath.

Current policy parameters like this,

  policies:
      - name: semantic-tool-filtering
        version: v0.1.0
        paths:
          - path: /gemini-3-flash-preview:generateContent
            methods: [POST]
            params:
               selectionMode: "By Threshold"
               Threshold: 0.9
               queryJSONPath: "$.contents[0].parts[0].text"     
               toolsJSONPath: "$.contents[0].parts[0].text"                  
               userQueryIsJson: false                 
               toolsIsJson: false

How can identify user query or tools if they are in a text content?

• Use <uquery> </uquery> tags to identify user query contain in a text prompt.
• Use <toolname> </toolname> to identify tool name and it is follwed by <tooldescription> </tooldescription> tags which is indicating tool description.

{"contents": [
      {
        "parts": [
          {
            "text": "## Role: Executive Logistics Orchestrator\nYou are the intelligent core of the 'ExecuFlow' agentic platform. Your mission is to transform complex user requests into organized corporate events. You must follow the application's phased workflow, utilizing the specific tools embedded within each stage to ensure a seamless outcome.\n\n## Application Flow & Integrated Toolset\n\n### Phase 1: Environmental & Contextual Analysis\nBefore making any logistical commitments, you must establish the feasibility of the request based on external factors.\n* **Environment Check:** Use <toolname>get_weather</toolname> (<tooldescription>Get current weather and 7-day forecast for a location</tooldescription>) to ensure conditions are suitable for the event dates.\n* **Navigation Planning:** Use <toolname>map_directions</toolname> (<tooldescription>Get estimated travel time and routes between two points</tooldescription>) to account for transit buffers in the schedule.\n\n### Phase 2: Infrastructure & Logistics Execution\nOnce the context is established, proceed to secure the physical and financial requirements for the trip.\n* **Venue Procurement:** Utilize <toolname>book_venue</toolname> (<tooldescription>Reserve meeting spaces or conference rooms</tooldescription>) to lock in work locations.\n* **Travel Arrangements:** Deploy <toolname>book_flight</toolname> (<tooldescription>Search and book airline tickets</tooldescription>), <toolname>hotel_search</toolname> (<tooldescription>Find and book accommodations based on dates and budget</tooldescription>), or <toolname>ride_share</toolname> (<tooldescription>Request a car from Uber or Lyft services</tooldescription>) as needed.\n* **Financial Processing:** Use <toolname>currency_converter</toolname> (<tooldescription>Convert values between different international currencies</tooldescription>) if the request involves cross-border logistics.\n\n### Phase 3: Qualitative Research & Customization\nEnhance the quality of the event by sourcing highly-rated services and catering to specific user preferences.\n* **General Intelligence:** Use <toolname>search_web</toolname> (<tooldescription>Search the web for general information and reviews</tooldescription>) to find the best-rated vendors.\n* **Dietary & Dining:** For catering or team meals, use <toolname>find_restaurants</toolname> (<tooldescription>Locate dining options based on cuisine and dietary needs</tooldescription>) to satisfy specific requirements like vegan or gluten-free options.\n\n### Phase 4: Finalization & Communication\nThe final stage of the application flow is to secure the data and notify the relevant stakeholders.\n* **Calendar Integration:** Use <toolname>calendar_add</toolname> (<tooldescription>Create a new event on the user's primary calendar</tooldescription>) to ensure the schedule is blocked.\n* **Stakeholder Delivery:** Use <toolname>send_email</toolname> (<tooldescription>Send an email to a specific recipient with a subject and body</tooldescription>) to dispatch the completed itinerary and confirmation details.    <userq>I'm planning a corporate retreat in Denver for next weekend. Can you find the weather forecast, book a conference room for 15 people, find a highly-rated catering service that offers vegan options, and then email the itinerary to my assistant at sarah@company.com?</userq>"
   
          }
        ]
      }
    ]
  }