Supporting AI tools

[antonpirker]

Hey everybody!

Not long ago we added an OpenAI integration to the Sentry SDK. Right now we have PRs for two more AI related integrations open, Anthropic and Langchain.

If you work in the AI sector or are an AI enthusiast, I would like to ask you a couple of questions:

• What data from your AI project would you like to see in Sentry?
• What data would help to improve your AI pipelines, the training of your models, help you improve your AI project?
• What other AI related features would you love to see in Sentry?
• Are there any other tools you would like to see automatic instrumentation for?

Let's get a discussion going, so together we can build kick ass AI support into Sentry!

[smeubank]

After talking with some folks internally and from the community (downloads and stars etc), I believe the most useful platforms for folks would be as follows, grouped by product/service, and then popularity within. ✅ Means we already have a PR in the works for them

LLM Providers

• openai ✅
• huggingface-hub https://github.com/huggingface/huggingface_hub
• anthropic ✅

Framework

• langchain ✅
• llama-index https://github.com/run-llama/llama_index
• haystack https://github.com/deepset-ai/haystack

Vector DBs

• chromadb https://github.com/chroma-core/chroma
• cohere https://github.com/cohere-ai
• qdrant-client https://github.com/qdrant/qdrant-client
• weaviate-client https://github.com/weaviate/weaviate-python-client

[czyber]

Hey hey! Sure I'll give a vector db a spin. If you have any preferences on which to integrate first, I'm open to do any of them. Otherwise I'll have time on the weekend to have a deeper look 👍

[antonpirker]

@colin-sentry you probably know best what vector db @czyber could look into. You do cohere. Chroma is the one that is included in Langchain, so maybe this is a good candidate? Or should we wait until you do a cohere integration and just copy it and modify it to fit another db?

[colin-sentry]

I was just going to do the cohere LLM, not their vector DB yet.

We do have the embeddings measured in the OpenAI integration, which is most of the interesting bits for most vector DBs

[smeubank]

in the future we may have better/more native OTel support in the Python SDK

but until then :) @czyber if you want it might be interesting to work on chromadb, and refer the implementation here for inspiration

https://github.com/traceloop/openllmetry/tree/main/packages/opentelemetry-instrumentation-chromadb

[czyber]

Sure thing!

[smeubank]

Interesting use cases:

the following I am regurgitating from @czyber 😆 for full transparency 🚀

An interesting metric to observe for LLMs in general could be the output tokens from the LLM categorized by prompts.

Lets take the following example:
I have the use case that I want the LLM to create studying material for students (e.g. task descriptions for math problems),

The prompt is a template, and has some dynamic inputs (such as 'output_language', 'grade', 'topic'). It would be interesting to have
some metrics (maybe even alerts) if the LLM responses deviate from the expectation value of the output_token count.

[smeubank]

Hugging Face does provide some support in the way that you can indicate an OTLP endpoint (Text Generation Interface: Distributed Tracing). Which appear to be what openLLMetry takes advantage of here

[antonpirker]

Just a quick update:

• Our LLM monitoring is now out of Beta: https://docs.sentry.io/product/insights/llm-monitoring/
• The Python libraries we support right now can be found here: https://docs.sentry.io/platforms/python/integrations/#ai

[iloveitaly]

@antonpirker the documentation isn't clear: If I have the py openai library installed, will monitoring automatically be setup or do I need to @ai_track("My AI pipeline") each function which makes an openai call?

[nsaef]

The same goes for LangChain: the way the documentation is written, I would assume it to track my LLM calls automatically. However, the "LLM Monitoring" section only showed up once I explicitely initialized Sentry with the LangChain integration, and it's not creating any traces without using the decorators.

Relevant excerpt from the docs:

The Sentry LLM Monitoring feature relies on the fact that you have an orchestrator (like LangChain) creating pipelines of one or more LLMs (such as gpt-4). [...]

If you're using a provider like OpenAI without an orchestrator like LangChain, you'll need to manually create pipelines with the @ai_track annotation.

[smeubank]

hi @iloveitaly and @nsaef

thanks for this feedback. We discussed this in planning today, I don't have an ETA, but we did create an issue. Generally i think we should better communicate what auto-instrumentation does from the SDK, and what we expect form users. Feel free to comment any corrections or feedback also on that issue.

getsentry/sentry-docs#13167

[invke]

Hi @antonpirker, we are typically using https://github.com/griptape-ai/griptape as our AI framework. Is there plans to support integration there? The current project I'm working on is using OpenAI models (for now) via Griptape, which in turn uses the OpenAI SDK, so I had thought perhaps it would work. However, it doesn't seem to report token usage, has non realistic 0.02ms durations and of course then no cost estimates. I figure perhaps it's just an incompatibility with how Griptape operates around the SDKs.

Did think about them both supporting Open Telemetry but figured I'd check here first as I wasn't immediately sure how that'd work and was time-bound.

[smeubank]

hi @invke

It isn't planned. We are working on OpenTelemetry support, but OpenAI is not necessarily a core feature support of this work, https://github.com/getsentry/projects/issues/68. I don't know that OTel has native support for AI libraries today. There is OpenLLMetry which is not part of OTel, and is 3rd party.

i also created an issue to track this topic, and let the team consider it, and any more context you might want to add to the issue.

#4330