Skip to content

LLM Langchain wrap call in chain to display it in sentry AI tab #13905

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 6 commits into
base: master
Choose a base branch
from
Open

LLM Langchain wrap call in chain to display it in sentry AI tab #13905

Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
53fad65
Update index.mdx
Luke31 Jun 3, 2025
5b8bf3d
Update index.mdx
Luke31 Jun 3, 2025
745e223
Update index.mdx
Luke31 Jun 3, 2025
734bbc8
Update index.mdx
Luke31 Jun 3, 2025
b25ee21
Update index.mdx
Luke31 Jun 3, 2025
2ffcbc6
add import for StrOutputParser
Luke31 Jun 3, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 5 additions & 1 deletion docs/platforms/python/integrations/langchain/index.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,8 @@ An additional dependency, `tiktoken`, is required to be installed if you want to

In addition to capturing errors, you can monitor interactions between multiple services or applications by [enabling tracing](/concepts/key-terms/tracing/). You can also collect and analyze performance profiles from real users with [profiling](/product/explore/profiling/).

Tracing is required to see AI pipelines in sentry's [AI tab](https://sentry.io/insights/ai/llm-monitoring/).
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for adding this, definitely an oversight that we forgot to add this before!


Select which Sentry features you'd like to install in addition to Error Monitoring to get the corresponding installation and configuration instructions below.

<OnboardingOptionButtons
Expand Down Expand Up @@ -76,13 +78,15 @@ Verify that the integration works by inducing an error:

```python
from langchain_openai import ChatOpenAI
from langchain_core.output_parsers import StrOutputParser
import sentry_sdk

sentry_sdk.init(...) # same as above

llm = ChatOpenAI(model="gpt-3.5-turbo-0125", temperature=0, api_key="bad API key")
with sentry_sdk.start_transaction(op="ai-inference", name="The result of the AI inference"):
response = llm.invoke([("system", "What is the capital of paris?")])
chain = (llm | StrOutputParser()).with_config({"run_name": "test-run"})
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm admittedly not super familiar with Langchain, could you explain what this change does differently than what we had before? I get that the run_name allows it to show up in Sentry, but what is the StrOutputParser() for?

Copy link
Author

@Luke31 Luke31 Jun 11, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@szokeasaurusrex Thank you for the review!
To be honest the StrOutputParser() is more or less syntactic sugar for response.content. Here it is mainly used in combination with the pipe | to create a chain.

The important difference here is that the syntax (X | Y) creates a LangChain chain/Runnable using LangChain Expression Language LCEL. This is the new syntax of the deprecated LLMChain class.

image

As this example has no prompt and is just invoked directly with a question, no prompt is passed and the chain is just created by adding an output-parser.

I have tried following code:

llm = llm.with_config({"run_name": "llm_raw"})
resp_llm_raw = llm.invoke([("system", "What is the capital of France?")])
print(f"llm response raw without StrOutputParser:\n{resp_llm_raw}\n\n")
print(f"llm response raw content:\n{resp_llm_raw.content}\n\n")

chain_parsed = (llm_session_summary | StrOutputParser()).with_config({"run_name": "chain_parsed"})
resp_chain_parsed = chain_parsed.invoke([("system", "What is the capital of France?")])
print(f"chain response with StrOutputParser:\n{resp_chain_parsed}\n\n")

resulting in this output:

llm response raw without StrOutputParser:
content='The capital of France is Paris.' additional_kwargs={'refusal': None} response_metadata={'token_usage': {'completion_tokens': 7, 'prompt_tokens': 14, 'total_tokens': 21, 'completion_tokens_details': {'accepted_prediction_tokens': 0, 'audio_tokens': 0, 'reasoning_tokens': 0, 'rejected_prediction_tokens': 0}, 'prompt_tokens_details': {'audio_tokens': 0, 'cached_tokens': 0}}, 'model_name': 'gpt-4o-mini-2024-07-18', 'system_fingerprint': 'XXXXX', 'id': 'XXXXX', 'finish_reason': 'stop', 'logprobs': None} id='run-343435-4d8c-4e85-9ca5-3414e396e6f4-0' usage_metadata={'input_tokens': 14, 'output_tokens': 7, 'total_tokens': 21, 'input_token_details': {'audio': 0, 'cache_read': 0}, 'output_token_details': {'audio': 0, 'reasoning': 0}}

llm response raw content:
The capital of France is Paris.

chain response with StrOutputParser:
The capital of France is Paris.

Only chain_parsed is visible in AI-tab, llm_raw is not visible:
image

I dug a bit deeper in the sentry-python sdk and found

So a plain llm-call will not appear as an AI pipeline.

TL;DR:

  • Only Langchain root-chains appear in the sentry AI tab.
  • From my point of view that makes sense as an AI-pipeline is the same concept as a chain. It just wasn't clear from the documentation.

response = chain.invoke([("system", "What is the capital of France?")])
print(response)
```

Expand Down
Loading