.Net: Filters use cases to be supported before making the feature non-experimental

[matthewbolanos]

Tasks

• Create showcase application which demonstrates all functionality below

To make filters non-experimental, the following user stories should be met:

• Telemetry – any of the telemetry that is made available via Semantic Kernel logging should be possible to recreate using filters. This would allow a developer to author telemetry with additional coordination IDs and signals that are not available in the out-of-the-box telemetry. To validate this, we should ensure that filters are available in the same spots as existing telemetry and the filters have the same information.
• Approving function calls – a developer should be able to request approval from a user before performing an action requested by an AI. If the user approves, the function should be invoked. If the user disapproves, the developer should be able to customize the cancelation message sent to the AI so that it understands that the action was rejected. For example, it should be possible to return back a message like "The function call to wire $1000 was rejected by the current user".
• Semantic Caching – with filters, a developer should be able to get a rendered prompt and check if there is a cached result that can be given to a user instead of spending tokens with an LLM. To achieve this, a developer should be able to add a filter after a prompt is rendered to check if there is a cached result. If there is, the dev should be able to cancel the operation and send back the cached result instead. It should also be possible to cache results from an LLM request using the function invoked filter.
• FrugalGPT – a dev should be able to implement FrugalGPT with filters. With filters, a developer should be able to call a local AI model to determine how complex a request is. Depending on the complexity of the request, the dev should be able to change the execution settings of the prompt so that it can use different models. This might be possible with the AI service selector. If it is, we should rationalize which path should be used when.
• Catching/throwing errors – a function may result in an error. Instead of sending the default error to the LLM, the developer should be able to do one of three things... 1) customize the error message sent to the LLM, 2) let the error propagate up so that it can be caught elsewhere, 3) do nothing and let the existing error message go to the LLM.

Additionally, thought should be considered to make applying filters more easily. These are likely extensions, so they should not be blocking for making filters non-experimental:

• Targeting filter – It should be "easy" to apply a filter to a single function or an entire plugin without having to write conditions within the filter. For example, semantic caching may only be necessary (or valid) on a couple semantic functions. The developer should be able to take an off-the-shelf filter (e.g., written by the Redis Cache team) and selectively apply it to only some prompts.
• Targeting filters by property – Some filters should only be enabled if a function has a particular property (e.g., if they are a semantic function). One property we may also consider is if a function is "consequential". With this property, a developer could choose which functions should require approvals.
• Targeting by invocation – Some filters should only be enabled if the function is invoked via a tool call. For example, it's likely not necessary to request user approval if a function is called within a template or explicitly by the developer. Instead, it's only necessary to run the filter if its invoked by an AI (where less trust is available).

[lavinir]

For Approving function calls:

Not sure I completely understand the use case for User vs Developer. In terms of what happens after a 'cancellation' there should be higher flexibility. There are two places for filters Pre function invocation and post invocation.
If the user decides to cancel, it should also bypass (or at least have the option) to bypass any additional call to the LLM and return the relevant data back to the user:

1. Updated chat history including any successful function invocations that were not cancelled.
2. Function response (if this is a post invocation filter)
3. Perhaps some additional context object supplied by the function filter.

@matthewbolanos , does that make sense ?

[matthewbolanos]

I created a second issue to track the need for filters at the automatic function call level here: #5470