Access LogProbs in ValidationOutcome

[abhishek9sharma]

Description
Support for accessing logprobs in ValidationOutcome.
I am currently using something similar to this https://github.com/guardrails-ai/guardrails-api/blob/main/guardrails_api/utils/openai.py#L4 in order to stream back the validation outcome to a downstream client in open ai format.

However I am having trouble with accessing the logprob information in validation outcome. Is there a way to do it.

Why is this needed
Some downstream clients need logprobs for each token for some logic at their end. Also in general as guardrails-ai has to be used as a proxy/wrapper across open ai endpoints, clients downstream may expect everything generated by open-ai compliant APIs to be passed down to the after validation.

End result
All the info guard-rails fetches from a backend LLM API can be passed down with ValidationOutcome

[CalebCourier]

@abhishek9sharma

All the info guard-rails fetches from a backend LLM API can be passed down with ValidationOutcome

Agreed, I think as we work towards a 1.x release our interface should move towards an augmented OpenAI API Specification or double down on being a pluggable, distinct process that does not effect/hide any output from the LLM rather working as side effects or hooks.

In the meantime, the way to achieve this is to use Guard.parse or Guard.validateas a post processor along your pipeline. I.e. if you need input and output validation it would look something like:generate a prompt -> inputGuard.validate(prompt) -> If prompt ok -> Call LLM -> outputGuard.validate(LLM.output)`. This way the LLM call is without your own process and you have access to it in it's entirety.

CC: @zsimjee for comments on 1.x spec

[abhishek9sharma]

Thanks @CalebCourier I will try something like this for now

import guardrails as gd
import litellm
from guardrails import AsyncGuard, Guard, OnFailAction
from guardrails.hub import ProfanityFree
L = [ProfanityFree(on_fail=OnFailAction.EXCEPTION)]
async def guarded_stream():
    guard = AsyncGuard().use_many(*L)
    try:
        llm_response_streamer = await litellm.acompletion(
            api_base="https://api.deepseek.com",
            api_key="sk-1234",
            model="deepseek/deepseek-chat",
            messages=[{"role": "user", "content": "HELLO "}],
            max_tokens=100,
            stream=True,
        )
        async for resp in llm_response_streamer:
            #add any custom logic here
            resp_text = resp.choices[0].delta.content
            valoutcome = await guard.validate(resp_text)
            if valoutcome.validation_passed:
                print(resp_text)
    except Exception as e:
        print(e)
# Run the async function to generate text
import asyncio
asyncio.run(guarded_stream())