[API Proposal]: Add a provider-neutral abstraction for decision-oriented AI models #7764

Background and motivation

Microsoft.Extensions.AI provides provider-neutral abstractions for several distinct AI model capabilities, including chat completion, embeddings, image generation, speech-to-text, text-to-speech, and realtime interaction.

A new category of AI models is emerging whose primary operation is different from text generation:

Evaluate application state against one or more bounded questions and return structured, probabilistic decisions that software can consume directly.

The motivating implementation is TypeSafe AI's Jev, described as a "System One Model".

Jev is not being proposed as the abstraction itself. It is an example of a model exposing a capability that currently does not have a corresponding provider-neutral contract in Microsoft.Extensions.AI.

The important distinction is between generative inference:

messages / prompt
        ↓
generative model
        ↓
generated sequence
        ↓
optional structured parsing

and decision-oriented inference:

application state
+ bounded typed questions
        ↓
decision model
        ↓
typed probabilistic decisions

The output space of each question is defined before inference. Applications consume probabilities, distributions, selections, and scores directly rather than parsing generated prose.

A concrete example is:

Customer state
   │
   ├── "Is the customer requesting a refund?"
   │       → P(true) = 0.97
   │
   ├── "What is the request type?"
   │       → refund
   │       → refund: 0.82
   │       → rebooking: 0.12
   │       → information: 0.06
   │
   └── "How frustrated is the customer?"
           → score: 1.3
           → level probabilities:
                calm:       0.0
                concerned:  0.7
                angry:      0.3

These values can participate directly in normal application code:

if (refundRequested.TrueProbability > 0.9)
{
    ...
}

if (intent.Confidence < 0.5)
{
    ...
}

if (frustration.Score > 1.5)
{
    ...
}

This proposal asks whether Microsoft.Extensions.AI.Abstractions should define a small provider-neutral capability for this form of inference.

The exact naming is open to API review. For the proposal below, I use IDecisionClient.

Why this is not just structured output from IChatClient

An IChatClient can certainly be prompted to return JSON or use a structured response format.

That provides syntactic structure, but does not define the semantics of a decision-model capability.

A decision-oriented contract can standardize concepts that IChatClient intentionally does not:

shared application state
heterogeneous typed questions
closed output domains
batch evaluation
binary probability
categorical probability distribution
ordinal probability distribution
continuous score derived from ordered levels
decision confidence/uncertainty

For example, an application should be able to ask for a categorical choice and rely on the response containing a probability distribution over exactly the supplied candidates.

With IChatClient, that behavior is an application-specific prompt/schema convention.

With a dedicated decision-model abstraction, it becomes part of the model capability contract and can be implemented by different providers.

This would be analogous to the reason embeddings have IEmbeddingGenerator<TInput,TEmbedding> rather than being represented as "ask an IChatClient to output an array of floats".

Proposed semantic model

The initial abstraction should support three general-purpose decision primitives.

The names below are intentionally provider-neutral.

These map naturally to Jev's current Noul, Choice, and Score capabilities, but none of those provider-specific names need to appear in the common API.

Binary decision

A binary decision answers a yes/no proposition.

The result is:

P(true) ∈ [0, 1]

A value near 1 means strong support for true, a value near 0 means strong support for false, and a value near 0.5 represents uncertainty between the two outcomes.

There should not be a separate mandatory Confidence property for this primitive.

The probability already captures the two-outcome distribution:

P(false) = 1 - P(true)

This also matches Jev's Noul semantics, where no separate confidence value is returned.

Optional true/false criteria should be supported so callers can clarify the semantic boundary:

true:
    "The customer explicitly or implicitly asks for money back."

false:
    "The customer asks only for information or replacement."

Choice decision

A choice selects exactly one member of a caller-supplied unordered set.

For example:

refund
rebooking
information

The result should contain:

SelectedChoice
Probabilities[choice]
Confidence

Probabilities represents the complete categorical distribution and should contain an entry for every supplied choice.

The probabilities should each be in [0, 1] and sum to approximately 1.

SelectedChoice must be one of the supplied choices.

Confidence should be optional in the common abstraction. Some providers, including Jev, expose a useful scalar derived from the shape of the distribution, but the full probability distribution is the more fundamental interoperable value.

The contract should not claim that confidence is calibrated or comparable across providers unless an implementation explicitly documents that guarantee.

Score decision

A score represents a position on an ordered caller-defined set of levels.

For example:

0: Cosmetic; no functional impact
1: Feature degraded, but a workaround exists
2: Blocking issue; no workaround exists

Unlike a Choice, these values have an order.

The result should contain:

Score
Probabilities[level]
Confidence

The probability distribution represents the model's probability for every supplied level.

For an N-level scale, levels have ordinal positions:

0 ... N - 1

The continuous score can then have a well-defined provider-neutral meaning:

Score = Σ(levelIndex × P(levelIndex))

For example:

P(level 0) = 0.0
P(level 1) = 0.7
P(level 2) = 0.3

Score =
    0 × 0.0 +
    1 × 0.7 +
    2 × 0.3
    = 1.3

This is useful because two results with the same rounded level may still carry materially different distributions.

As with Choice, Confidence should be optional and should not replace the full distribution.

Batching heterogeneous questions is a core requirement

A central property of this capability is that multiple questions may be evaluated against the same state in one model request.

For example:

State
 ├── Binary: "Is a refund requested?"
 ├── Choice: "What is the request category?"
 ├── Score:  "How frustrated is the customer?"
 ├── Binary: "Does this require human escalation?"
 └── Choice: "Which department should handle it?"

The request must therefore support a heterogeneous collection of question types.

This should not require callers to make one request per decision.

A provider may optimize these questions using parallel inference, shared state encoding, batching, or some other mechanism.

The abstraction should expose batching as a first-class capability without prescribing how the provider implements it internally.

An important semantic property is that every question is evaluated against the same supplied state.

Providers may have limits on:

number of questions
number of choices
number of score levels
request size
supported primitive types

Those limits should remain provider/model-specific rather than being encoded from Jev into the common abstraction.

API Proposal

The following is intended as a concrete starting point for API review, not a claim that every name is final.

namespace Microsoft.Extensions.AI;

public interface IDecisionClient : IDisposable
{
    Task<DecisionResponse> GetResponseAsync(
        DecisionRequest request,
        DecisionOptions? options = null,
        CancellationToken cancellationToken = default);

    object? GetService(
        Type serviceType,
        object? serviceKey = null);
}

Request

namespace Microsoft.Extensions.AI;

public sealed class DecisionRequest
{
    public DecisionRequest(
        JsonElement state,
        IEnumerable<DecisionQuestion> questions);

    public JsonElement State { get; set; }

    public IList<DecisionQuestion> Questions { get; }
}

JsonElement is proposed for the shared state because the state may naturally be:

a JSON object
a JSON array
a string
a scalar
a structured application snapshot

and because it avoids requiring providers to serialize arbitrary runtime object instances.

It also provides a predictable boundary for trimming and Native AOT scenarios.

A convenience API could later allow strongly typed state:

public static class DecisionClientExtensions
{
    public static Task<DecisionResponse> GetResponseAsync<TState>(
        this IDecisionClient client,
        TState state,
        JsonTypeInfo<TState> stateTypeInfo,
        IEnumerable<DecisionQuestion> questions,
        DecisionOptions? options = null,
        CancellationToken cancellationToken = default);
}

This helper could serialize the state with the supplied JsonTypeInfo<TState> without making the provider-facing abstraction generic.

Question model

namespace Microsoft.Extensions.AI;

public abstract class DecisionQuestion
{
    protected DecisionQuestion(
        string id,
        string instructions);

    public string Id { get; }

    public string Instructions { get; set; }

    public AdditionalPropertiesDictionary? AdditionalProperties { get; set; }
}

Question IDs must be unique within a request.

They are application correlation identifiers and should not be assumed to carry model semantics.

Binary question

public sealed class BinaryDecisionQuestion : DecisionQuestion
{
    public BinaryDecisionQuestion(
        string id,
        string instructions);

    public BinaryDecisionCriteria? Criteria { get; set; }
}

public sealed class BinaryDecisionCriteria
{
    public string? TrueDescription { get; set; }

    public string? FalseDescription { get; set; }
}

Choice question

public sealed class ChoiceDecisionQuestion : DecisionQuestion
{
    public ChoiceDecisionQuestion(
        string id,
        string instructions,
        IEnumerable<DecisionChoice> choices);

    public IList<DecisionChoice> Choices { get; }
}

public sealed class DecisionChoice
{
    public DecisionChoice(
        string name,
        string? description = null);

    public string Name { get; }

    public string? Description { get; set; }
}

Choice names must be unique within the question.

Their descriptions provide semantic criteria but do not impose ordering.

Score question

public sealed class ScoreDecisionQuestion : DecisionQuestion
{
    public ScoreDecisionQuestion(
        string id,
        string instructions,
        IEnumerable<DecisionScoreLevel> levels);

    public IList<DecisionScoreLevel> Levels { get; }
}

public sealed class DecisionScoreLevel
{
    public DecisionScoreLevel(string description);

    public string Description { get; set; }
}

A Score must contain at least two levels.

Their list order defines their ordinal position:

Levels[0] → position 0
Levels[1] → position 1
...
Levels[n] → position n

The abstraction should not adopt Jev's current maximum number of levels as a general API constraint.

Response model

namespace Microsoft.Extensions.AI;

public sealed class DecisionResponse
{
    public IDictionary<string, DecisionAnswer> Answers { get; }

    public string? ResponseId { get; set; }

    public string? ModelId { get; set; }

    public UsageDetails? Usage { get; set; }

    public object? RawRepresentation { get; set; }

    public AdditionalPropertiesDictionary? AdditionalProperties { get; set; }
}

The key in Answers corresponds to the question ID.

The answer types are heterogeneous:

public abstract class DecisionAnswer
{
    public object? RawRepresentation { get; set; }

    public AdditionalPropertiesDictionary? AdditionalProperties { get; set; }
}

Binary answer

public sealed class BinaryDecisionAnswer : DecisionAnswer
{
    public double TrueProbability { get; set; }
}

Invariant:

0 <= TrueProbability <= 1

Choice answer

public sealed class ChoiceDecisionAnswer : DecisionAnswer
{
    public required string SelectedChoice { get; set; }

    public IDictionary<string, double> Probabilities { get; }

    public double? Confidence { get; set; }
}

The common semantic expectation is:

Probabilities contains every requested choice

0 <= probability <= 1

Σ probabilities ≈ 1

SelectedChoice belongs to the requested choice set

Score answer

public sealed class ScoreDecisionAnswer : DecisionAnswer
{
    public double Score { get; set; }

    public IDictionary<int, double> Probabilities { get; }

    public double? Confidence { get; set; }
}

For levels indexed 0 ... N - 1:

Probabilities contains every requested level

0 <= probability <= 1

Σ probabilities ≈ 1

0 <= Score <= N - 1

and the portable Score semantic is:

Score = Σ(levelIndex × probability)

A provider adapter may compute Score from its probability distribution if the underlying provider does not return the expected value directly.

Options and metadata

The capability should follow existing MEAI conventions where useful.

A minimal options type could be:

public class DecisionOptions
{
    public string? ModelId { get; set; }

    public AdditionalPropertiesDictionary? AdditionalProperties { get; set; }

    public Func<IDecisionClient, object?>? RawRepresentationFactory { get; set; }

    public virtual DecisionOptions Clone();
}

And metadata could be available through GetService:

public class DecisionClientMetadata
{
    public DecisionClientMetadata(
        string? providerName = null,
        Uri? providerUri = null,
        string? defaultModelId = null);

    public string? ProviderName { get; }

    public Uri? ProviderUri { get; }

    public string? DefaultModelId { get; }
}

This mirrors existing MEAI capability patterns without requiring a larger builder/middleware proposal yet.

Example usage

Mixed decision batch

JsonElement state = JsonSerializer.SerializeToElement(new
{
    TicketMessage =
        "My flight was cancelled. I'm really frustrated. Can I get a refund?",

    RefundPolicy =
        "Cancelled flights are eligible for a full refund."
});

DecisionRequest request = new(
    state,
    [
        new BinaryDecisionQuestion(
            "refund_requested",
            "Does the customer request a refund?"),

        new ChoiceDecisionQuestion(
            "request_type",
            "What is the customer's main request?",
            [
                new("refund", "The customer wants money returned."),
                new("rebooking", "The customer wants another flight."),
                new("information", "The customer only wants information.")
            ]),

        new ScoreDecisionQuestion(
            "frustration",
            "How frustrated does the customer appear?",
            [
                new("Calm and neutral."),
                new("Concerned but civil."),
                new("Very angry or using strong language.")
            ])
    ]);

DecisionResponse response =
    await decisionClient.GetResponseAsync(request);

Binary probability

var refund =
    (BinaryDecisionAnswer)response.Answers["refund_requested"];

if (refund.TrueProbability >= 0.9)
{
    // The application decides what a sufficiently high probability means.
}

Choice distribution

var requestType =
    (ChoiceDecisionAnswer)response.Answers["request_type"];

Console.WriteLine(requestType.SelectedChoice);

foreach ((string choice, double probability) in requestType.Probabilities)
{
    Console.WriteLine($"{choice}: {probability}");
}

if (requestType.Confidence is < 0.5)
{
    // Ask for clarification or use another decision path.
}

Continuous Score

var frustration =
    (ScoreDecisionAnswer)response.Answers["frustration"];

if (frustration.Score >= 1.5)
{
    // Application-specific escalation.
}

The important point is that the provider returns decision values, while application code remains responsible for thresholds and business actions.

Why probability is part of the contract

I think probability should be treated as part of the semantics of these primitives rather than hidden in AdditionalProperties.

For example, this API:

bool IsRefundRequested

would lose important information compared with:

double TrueProbability

Similarly:

string Category

loses information compared with:

refund      0.52
billing     0.46
other       0.02

That distribution can materially change how an application behaves.

The same applies to Score. A score of 1.0 might mean:

P(1) = 1.0

or:

P(0) = 0.5
P(2) = 0.5

Those have the same expected score but very different uncertainty.

For that reason, the full distribution should be the portable result for Choice and Score.

Confidence, in contrast, can remain optional because it is a summary statistic derived from the distribution and its exact computation may vary between providers.

Probability and calibration semantics

The abstraction should distinguish probability from calibration guarantees.

A probability value means:

the model/provider's estimated probability for an outcome.

The common API should not imply:

a reported probability of 0.8 is empirically correct 80% of the time.

Calibration is a property of a particular model/provider and possibly of a particular workload.

Providers that make calibration guarantees can expose that through metadata, documentation, or future capability metadata.

Likewise, Confidence should not be assumed comparable between providers unless explicitly documented.

State and AOT considerations

I do not think the provider-facing API should accept:

object state

because that leaves serialization policy implicit and makes Native AOT/trimming harder.

JsonElement provides a simple stable interchange representation.

A strongly typed convenience overload accepting:

TState
JsonTypeInfo<TState>

can preserve ergonomic and source-generated serialization without requiring the core client abstraction to become generic.

This also allows the same request to naturally carry either structured state or a simple JSON string.

Structured instructions and criteria

Some providers may support richer instructions or criteria than plain strings, such as JSON objects or arrays carrying examples, positive/negative guidance, or provider-specific hints.

I would not standardize those richer shapes in the first version unless there is evidence they are common across providers.

The portable v1 contract can use textual instructions/descriptions.

Provider-specific richer representations can remain reachable through:

AdditionalProperties
RawRepresentationFactory
provider-specific client APIs

This keeps the common abstraction meaningful rather than turning it into a generic JSON transport.

Failure and capability semantics

There are a few points where maintainer feedback would be especially useful.

The initial API could use normal request-level exceptions when an implementation cannot execute the request.

However, heterogeneous batches raise useful future questions:

What if a provider supports Binary and Choice but not Score?

Should supported primitive kinds be discoverable from metadata?

Should one unsupported question fail the whole request?

Should partial answers be representable?

Should provider cardinality/model limits be discoverable?

I would prefer not to lock these into the initial proposal without evidence from multiple providers.

The core API should nevertheless be designed so such capability metadata can be added later without redesigning the basic request/answer model.

Relationship to IEvaluator

Microsoft.Extensions.AI.Evaluation.IEvaluator represents an evaluation operation/framework.

A decision client would represent an inference provider capability.

These can compose:

IEvaluator
    ↓
IDecisionClient
    ↓
Decision Model

For example, an evaluator might use a Score or Binary decision internally.

That does not make the two abstractions equivalent.

The same distinction exists between an evaluation framework and the IChatClient it may currently use.

Relationship to reranking

A reranker answers a much narrower question:

given query Q and documents D,
rank D by relevance to Q

A decision model covers general application judgments such as:

Is this proposition true?

Which of these actions fits?

Where does this state fall on an ordered scale?

A reranker could potentially be implemented using a decision model, but a general decision client should not depend on retrieval-specific concepts such as documents, search queries, or ranking pipelines.

This is therefore orthogonal to the retrieval/reranking discussion in dotnet/extensions#7507.

Relationship to chat routing

This proposal is also separate from RoutingChatClient and the response-quality/cascading discussion in dotnet/extensions#7712.

Those abstractions answer:

Which IChatClient should handle this chat request?

A decision client answers:

What bounded probabilistic judgment does this model make about this state?

A router could choose to consume an IDecisionClient as part of its policy, but routing is a consumer of decision inference rather than the decision-model abstraction itself.

Why not one completely generic structured-inference API?

A possible alternative is something such as:

IInferenceClient<TRequest, TResponse>

or:

GenerateStructuredAsync(JsonSchema schema)

That would be flexible, but it would standardize very little.

Two implementations could satisfy such an interface while having no interoperable semantics.

The proposed Binary / Choice / Score primitives deliberately standardize a small set of useful decision semantics:

binary probability
categorical distribution
ordinal distribution + continuous expected score

That gives middleware and consuming libraries something meaningful to build on.

Why not one interface per primitive?

Another option would be:

IBinaryDecisionClient
IChoiceDecisionClient
IScoreDecisionClient

That would make individual primitives strongly typed, but it makes heterogeneous batching around one shared state difficult and can force providers to make multiple calls even when their native API handles all question types together.

A single IDecisionClient with typed question and answer subclasses better represents providers that natively evaluate mixed decisions in one request.

Why not a generic IDecisionClient<TState, TResult>?

This could improve compile-time typing for one decision shape, but it does not naturally represent this important request:

one TState

→ BinaryDecisionAnswer
→ ChoiceDecisionAnswer
→ ScoreDecisionAnswer

A generic result type therefore works against heterogeneous batching.

The non-generic provider contract plus typed question/result primitives appears to provide a better interoperability boundary.

Typed helpers can be layered above it.

Existing related proposals

Before filing this proposal I reviewed several adjacent API discussions.

dotnet/extensions#7507 discusses retrieval pipeline abstractions and IReranker. That is retrieval-specific and narrower than general decision inference.

dotnet/extensions#7712 discusses CascadingChatClient and intentionally keeps response-quality policy outside the routing client. A decision model could become one possible implementation of such a policy, but it is not itself a routing client.

dotnet/extensions#7587 is useful precedent for deciding whether a capability belongs in Microsoft.Extensions.AI versus a separate peer package. I think that same question is worth explicit maintainer feedback here.

Package placement

My initial hypothesis is:

Microsoft.Extensions.AI.Abstractions

because this is a model-provider capability analogous to:

IChatClient
IEmbeddingGenerator
IImageGenerator
ISpeechToTextClient
ITextToSpeechClient

rather than an application workflow domain.

However, I would like maintainer feedback on this point.

The important objective is not the package name; it is establishing the right provider-neutral boundary if this category is considered sufficiently general.

Evidence and ecosystem maturity

I want to be explicit about an important limitation of this proposal.

Jev is currently the clearest concrete motivating implementation I have found for this exact combination of:

shared state
heterogeneous decision questions
closed typed outputs
probability distributions
batch evaluation

There are many adjacent technologies—classifiers, reward models, judge models, semantic routers, guardrail models, and rerankers—but they should not be claimed to implement the same capability without verifying their semantics.

So this issue is not claiming that a mature multi-provider ecosystem already exists.

Instead, the question is:

Is the semantic model useful and sufficiently general that MEAI should provide an experimental provider-neutral contract before applications begin exposing vendor-specific decision-model APIs?

If the maintainers believe one provider is not enough evidence yet, an experimental API or further prototype/provider survey may be the appropriate next step.

Design goals

The proposed abstraction should be:

Provider-neutral
Small
Asynchronous
Cancellation-aware
Batch-aware
Heterogeneous-question aware
AOT-friendly
Probability-preserving
Composable with existing MEAI APIs
Extensible through GetService / AdditionalProperties / raw representations
Independent of any Agent framework

It should not attempt to standardize:

agent orchestration
tool selection
model routing
business thresholds
authorization
approval
retry policy
caching
logging
DI registration
provider-specific limits
provider-specific rich criteria

Those can be built above the core model capability.

Potential consumers

A provider-neutral decision capability could later be used by libraries and applications for scenarios such as:

intent classification
tool shortlisting
skill selection
agent routing
model routing
workflow branching
RAG relevance filtering
guardrail signals
completion judgments
response-quality scoring
triage
risk signals
semantic policy inputs

These are examples of consumers, not responsibilities of IDecisionClient.

In particular, model output should not be treated as authorization:

Decision model
    → probabilistic signal
    → deterministic application policy

not:

Decision model says "safe"
    → grant permission

Follow-up integration

If MEAI adopts a provider-neutral decision-model abstraction, a separate proposal can explore integration with Microsoft Agent Framework.

Potential consumers there include:

LoopEvaluator
tool selection
Agent Skills selection
handoff routing
model routing
agent evaluation
context filtering

That integration should depend on the MEAI abstraction rather than defining a competing Agent Framework-specific decision-model interface.

This issue is intentionally scoped only to the underlying model capability.

Open questions for API review

The areas where I would especially value maintainer guidance are:

1. Is IDecisionClient the right conceptual name, or is another term more appropriate?
2. Does this capability belong in Microsoft.Extensions.AI.Abstractions, or should it be a peer abstraction package?
3. Is JsonElement the appropriate provider-facing representation for shared state?
4. Should the first contract require full probability distributions for Choice and Score, as proposed here?
5. Should Confidence exist in the common contract at all, or should consumers always derive their own certainty measure from probabilities?
6. Is the ordinal expected-value definition proposed for Score sufficiently general?
7. Should supported primitive kinds and provider cardinality limits be discoverable in the first version or deferred?
8. Should heterogeneous batch partial failures be representable, or should v1 use request-level failure?
9. Should typed TState + JsonTypeInfo<TState> helpers ship together with the abstraction?
10. Is the current provider evidence sufficient for an experimental abstraction, or should the first step be an experimental prototype/provider survey?

The main goal of this proposal is to determine whether decision-oriented inference deserves a first-class provider-neutral model capability in the Microsoft.Extensions.AI ecosystem, in the same way that applications today can depend on IChatClient or IEmbeddingGenerator without depending directly on a specific model provider.