[bug] Langchain OpenInference Trace output.value not a json dump of a BaseMessage

[njbrake]

Hi! Please let me know if I'm missing something silly with how I should be reloading the lanchain message from the openinference trace, it's possible that I am misunderstanding the intended use of the trace.

Describe the bug
If I export a Langchain trace and look at the output value from the final "openinference.span.kind": "AGENT", I can see that the output.value is a serialized list of LangChain Messages. However, The message doesn't seem to be serialized into something that I can re-load into a langchain BaseMessage:

The output in the openinference trace looks like:

      "output.value": "{\"messages\": [\"content='My output content here.' additional_kwargs={'refusal': None} response_metadata={'token_usage': {'completion_tokens': 1040, 'prompt_tokens': 2190, 'total_tokens': 3230, 'completion_tokens_details': {'accepted_prediction_tokens': 0, 'audio_tokens': 0, 'reasoning_tokens': 896, 'rejected_prediction_tokens': 0}, 'prompt_tokens_details': {'audio_tokens': 0, 'cached_tokens': 1920}}, 'model_name': 'o3-mini-2025-01-31', 'system_fingerprint': 'fp_42bfad963b', 'id': 'chatcmpl-BDE6vvr0h4Z0xOF5bMaEiji8yiREa', 'finish_reason': 'stop', 'logprobs': None} id='run-7ab1854f-7bd7-40a2-9cc0-3c79d01d6d57-0' usage_metadata={'input_tokens': 2190, 'output_tokens': 1040, 'total_tokens': 3230, 'input_token_details': {'audio': 0, 'cache_read': 1920}, 'output_token_details': {'audio': 0, 'reasoning': 896}}\"]}",

With that object, I can successfully do a output_value=json.loads(span['attributes']['output.value'] and then access the first message via output_value['messages'][0]. But after that, I can neither do a json.loads of the message (because it's serialized with '=' separating key/value pairs) nor a BaseMessage(**output_value['messages'][0]), because langchain expects a dict, not a string.

To Reproduce
Create a trace with a langchain agent, export it locally, and look at the last span that has "openinference.span.kind": "AGENT".

Expected behavior
I would expect that if openinference is serializing a langchain message, it would provide some mechanism to load and recover the langchain message object.

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

• macOS

Additional context
Add any other context about the problem here.

[dosubot]

The issue you're encountering seems to be related to the serialization format used by OpenInference for LangChain messages. The serialized messages use '=' to separate key/value pairs, which is not directly compatible with the expected dictionary format for LangChain's BaseMessage.

Currently, there isn't a specific utility function in OpenInference to convert these serialized messages back into a dictionary format suitable for BaseMessage instantiation. You might need to manually parse the serialized string to convert it into a dictionary format that LangChain can accept.

Here's a potential approach to manually parse the serialized message:

1. Split the serialized string by spaces to separate each key/value pair.
2. Further split each pair by '=' to get the key and value.
3. Construct a dictionary from these key/value pairs.

This manual parsing can help you convert the serialized message into a format that can be used to instantiate a BaseMessage.

If this is a common requirement, it might be worth considering implementing a utility function within your codebase to handle this conversion, or suggesting an enhancement to the OpenInference project to include such a utility.

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other}

[njbrake]

@dosu is correct, there is a workaround, here's the code I wrote to hack it. However, the point remains that I don't have to do this for other openinference instrumentors (e.g. smolagents). I'm not sure if needing to write a normalization layer on top of openinference is an expected hurdle that an engineer is supposed to do themselves, or if this is something that openinference is looking to standardize so that we don't need to do the normalization ourselves. Thanks!

    """Extract the hypothesis agent final answer from the trace"""
    # smolagents puts the final result in the output.value attribute
    for span in reversed(trace):
        if span['attributes']['openinference.span.kind'] == 'AGENT':
            content = span['attributes']['output.value']
            from langchain_core.messages import BaseMessage
            message = json.loads(content)['messages'][0]
            message = parse_generic_key_value_string(message)
            base_message = BaseMessage(**message, type="AGENT")
            return base_message

def parse_generic_key_value_string(text):
    """
    Parse a string that has items of a dict with key-value pairs separated by '='.
    Only splits on '=' signs, handling quoted strings properly.
    """
    pattern = r"(\w+)=('.*?'|\".*?\"|[^'\"=]*?)(?=\s+\w+=|\s*$)"
    
    result = {}
    
    matches = re.findall(pattern, text)
    for key, value in matches:
        # Clean up the key
        key = key.strip()
        
        # Clean up the value - remove surrounding quotes if present
        value = value.strip()
        if (value.startswith("'") and value.endswith("'")) or \
           (value.startswith('"') and value.endswith('"')):
            value = value[1:-1]
            
        # Store in result dictionary
        result[key] = value
    
    return result

[dosubot]

The need to manually parse serialized LangChain messages in OpenInference, as you've done, highlights a potential inconsistency in how different instrumentors handle serialization. While OpenInference does not currently provide a standardized method for deserializing these messages, it might be beneficial for the project to consider standardizing serialization formats across all instrumentors to avoid the need for custom parsing solutions like yours.

Your workaround effectively addresses the issue by converting the serialized string into a dictionary format that LangChain can accept. However, it would be ideal if OpenInference could provide built-in support for this, similar to how other instrumentors like smolagents handle serialization.

If this is a common challenge, it might be worth discussing with the OpenInference maintainers to see if there are plans to standardize serialization formats or provide utility functions for deserialization. This could help reduce the need for engineers to implement their own normalization layers.

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other}

[mikeldking]

hey @njbrake thanks for giving the instrumentor a spin! for langchain specifically we leverage what the langchain tracer deems as the input and output and serialize it - the structure of that payload I believe is schema'd by langchain itself and they have a way of deserializing it but I'm not 100% sure.

For the parts that are "semantic" we do parse it into these parts https://arize-ai.github.io/openinference/spec/semantic_conventions.html - the idea here is that these parts of the payload are stable and the same across all instrumentation so you can parse and replay LLM spans consistently.

Hope that makes sense. LMK if there's ways in which we can help. Definitely think it's an important workflow.

[njbrake]

Thanks for the response! That helps. I do see that my trace has the "llm.output_messages.0.message.content which is great! That's the part that I want to access. The problem is that in the langchain world that we have built at the moment (maybe we need to change something here, idk), that value isn't set in some of the final spans, and specifically, it's not set after the "should_continue" span, which is the part of the agent logic that is trying figure out if it should return the final answer. I'm trying to extract the final answer from the trace.

So I don't know that I can rely on that llm.output_messages.0.message.content key, because I don't have that key show up anytime after the agent has decided to finish. I'm using the agent: from langgraph.prebuilt import create_react_agent

As you mentioned, I'm not sure if there's anything that openinference can do to fix the problem if this is the way that langchain gives it to you, but it makes it hard to see the final answer, and I also see this issue if I load it up in Phoenix as opposed to inspecting the trace manually.