chore(llmobs): dac strip io from gemini #13874

jsimpher · 2025-07-03T15:34:45Z

Remove potentially sensitive i/o data from apm spans. This way, prompt and completion data will only appear on the llm obs spans, which are/will be subject to data access controls.

Mostly, this just removes io tag sets. A few things (mostly metrics) have llmobs tags dependent on span tags, so there is a bit more refactoring there.

Checklist

PR author has checked that all the criteria below are met
The PR description includes an overview of the change
The PR description articulates the motivation for the change
The change includes tests OR the PR description describes a testing strategy
The PR description notes risks associated with the change, if any
Newly-added code is easy to change
The change follows the library release note guidelines
The change includes or references documentation updates if necessary
Backport labels are set (if applicable)

Reviewer Checklist

Reviewer has checked that all the criteria below are met
Title is accurate
All changes are related to the pull request's stated goal
Avoids breaking API changes
Testing strategy adequately addresses listed risks
Newly-added code is easy to change
Release note makes sense to a user of the library
If necessary, author has acknowledged and discussed the performance implications of this PR as reported in the benchmarks PR comment
Backport labels are set in a manner that is consistent with the release branch maintenance policy

github-actions · 2025-07-03T15:38:34Z

CODEOWNERS have been resolved as:

releasenotes/notes/remove-io-data-from-span-gemini-integration-743cc458841345b4.yaml  @DataDog/apm-python
ddtrace/contrib/internal/google_generativeai/_utils.py                  @DataDog/ml-observability
ddtrace/contrib/internal/google_generativeai/patch.py                   @DataDog/ml-observability
ddtrace/llmobs/_integrations/gemini.py                                  @DataDog/ml-observability
ddtrace/llmobs/_integrations/utils.py                                   @DataDog/ml-observability
tests/contrib/google_generativeai/test_google_generativeai.py           @DataDog/ml-observability
tests/snapshots/tests.contrib.google_generativeai.test_google_generativeai.test_gemini_completion.json  @DataDog/ml-observability
tests/snapshots/tests.contrib.google_generativeai.test_google_generativeai.test_gemini_completion_error.json  @DataDog/ml-observability
tests/snapshots/tests.contrib.google_generativeai.test_google_generativeai.test_gemini_completion_image.json  @DataDog/ml-observability
tests/snapshots/tests.contrib.google_generativeai.test_google_generativeai.test_gemini_completion_multiple_messages.json  @DataDog/ml-observability
tests/snapshots/tests.contrib.google_generativeai.test_google_generativeai.test_gemini_completion_stream.json  @DataDog/ml-observability
tests/snapshots/tests.contrib.google_generativeai.test_google_generativeai.test_gemini_completion_system_prompt.json  @DataDog/ml-observability
tests/snapshots/tests.contrib.google_generativeai.test_google_generativeai.test_gemini_completion_tool_stream.json  @DataDog/ml-observability
tests/snapshots/tests.contrib.google_generativeai.test_google_generativeai.test_gemini_tool_chat_completion.json  @DataDog/ml-observability
tests/snapshots/tests.contrib.google_generativeai.test_google_generativeai.test_gemini_tool_completion.json  @DataDog/ml-observability

github-actions · 2025-07-03T16:00:28Z

Bootstrap import analysis

Comparison of import times between this PR and base.

Summary

The average import time from this PR is: 279 ± 3 ms.

The average import time from base is: 281 ± 3 ms.

The import time difference between this PR and base is: -2.3 ± 0.1 ms.

Import time breakdown

The following import paths have shrunk:

ddtrace.auto 2.024 ms (0.73%)

ddtrace.bootstrap.sitecustomize 1.351 ms (0.48%)

ddtrace.bootstrap.preload 1.351 ms (0.48%)

ddtrace.internal.remoteconfig.client 0.655 ms (0.23%)

ddtrace.debugging._import 0.013 ms (0.00%)

ddtrace.debugging._function.discovery 0.013 ms (0.00%)

ddtrace.debugging._function 0.013 ms (0.00%)

ddtrace 0.673 ms (0.24%)

ddtrace.internal._unpatched 0.030 ms (0.01%)

json 0.030 ms (0.01%)

json.decoder 0.030 ms (0.01%)

re 0.030 ms (0.01%)

enum 0.030 ms (0.01%)

types 0.030 ms (0.01%)

pr-commenter · 2025-07-03T16:31:29Z

Benchmarks

Benchmark execution time: 2025-07-14 14:33:49

Comparing candidate commit 9c11c39 in PR branch jsimpher/dac-strip-io-from-gemini with baseline commit c7c99b3 in branch main.

Found 0 performance improvements and 5 performance regressions! Performance is the same for 543 metrics, 2 unstable metrics.

scenario:iastaspects-lstrip_aspect

🟥 execution_time [+736.331ns; +811.666ns] or [+7.071%; +7.794%]

scenario:iastaspects-strip_aspect

🟥 execution_time [+763.810ns; +834.007ns] or [+7.340%; +8.014%]

scenario:iastaspectsospath-ospathbasename_aspect

🟥 execution_time [+737.405ns; +881.015ns] or [+17.411%; +20.802%]

scenario:iastaspectsospath-ospathjoin_aspect

🟥 execution_time [+641.593ns; +855.453ns] or [+10.434%; +13.911%]

scenario:iastaspectsospath-ospathnormcase_aspect

🟥 execution_time [+367.526ns; +438.529ns] or [+10.485%; +12.511%]

ddtrace/llmobs/_integrations/gemini.py

ncybul

LGTM -- left a couple minor comments, but looks clean!

Yun-Kim

Small nits but otherwise LGTM

ddtrace/llmobs/_integrations/gemini.py

jsimpher · 2025-07-14T16:34:47Z

/merge

dd-devflow-routing-codex · 2025-07-14T16:34:51Z

View all feedbacks in Devflow UI.

2025-07-14 16:34:51 UTC ℹ️ Start processing command /merge

2025-07-14 16:35:02 UTC ℹ️ MergeQueue: waiting for PR to be ready

This merge request is not mergeable yet, because of pending checks/missing approvals. It will be added to the queue as soon as checks pass and/or get approvals.
Note: if you pushed new commits since the last approval, you may need additional approval.
You can remove it from the waiting list with /remove command.

2025-07-14 16:35:17 UTC ⚠️ MergeQueue: This merge request was unqueued

jacob.simpher@datadoghq.com unqueued this merge request

jsimpher · 2025-07-14T16:35:06Z

/merge --cancel

dd-devflow-routing-codex · 2025-07-14T16:35:10Z

View all feedbacks in Devflow UI.

2025-07-14 16:35:10 UTC ℹ️ Start processing command /merge --cancel

jsimpher · 2025-07-14T18:55:46Z

/merge

dd-devflow-routing-codex · 2025-07-14T18:55:50Z

View all feedbacks in Devflow UI.

2025-07-14 18:55:49 UTC ℹ️ Start processing command /merge

2025-07-14 18:56:02 UTC ℹ️ MergeQueue: waiting for PR to be ready

This merge request is not mergeable yet, because of pending checks/missing approvals. It will be added to the queue as soon as checks pass and/or get approvals.
Note: if you pushed new commits since the last approval, you may need additional approval.
You can remove it from the waiting list with /remove command.

2025-07-14 19:21:19 UTC ℹ️ MergeQueue: merge request added to the queue

The expected merge time in main is approximately 1h (p90).

2025-07-14 19:44:47 UTC ⚠️ MergeQueue: This merge request build was cancelled

jacob.simpher@datadoghq.com cancelled this merge request build

jsimpher · 2025-07-14T19:44:32Z

/merge --cancel

dd-devflow-routing-codex · 2025-07-14T19:44:37Z

View all feedbacks in Devflow UI.

2025-07-14 19:44:36 UTC ℹ️ Start processing command /merge --cancel

jsimpher · 2025-07-14T19:55:52Z

/merge

dd-devflow-routing-codex · 2025-07-14T19:55:57Z

View all feedbacks in Devflow UI.

2025-07-14 19:55:57 UTC ℹ️ Start processing command /merge

2025-07-14 19:56:08 UTC ℹ️ MergeQueue: pull request added to the queue

The expected merge time in main is approximately 1h (p90).

2025-07-14 21:43:12 UTC ℹ️ MergeQueue: This merge request was merged

jsimpher added 2 commits July 3, 2025 11:28

remove io from google generative ai intergration

d63fc8f

add release note

668e976

black

911b8fc

stream tag

259198f

jsimpher marked this pull request as ready for review July 3, 2025 16:45

jsimpher requested review from a team as code owners July 3, 2025 16:45

jsimpher requested review from Yun-Kim and erikayasuda July 3, 2025 16:45

emmettbutler approved these changes Jul 7, 2025

View reviewed changes

ncybul reviewed Jul 8, 2025

View reviewed changes

ddtrace/llmobs/_integrations/gemini.py Show resolved Hide resolved

ncybul reviewed Jul 8, 2025

View reviewed changes

ddtrace/llmobs/_integrations/gemini.py Outdated Show resolved Hide resolved

ncybul approved these changes Jul 8, 2025

View reviewed changes

Yun-Kim approved these changes Jul 11, 2025

View reviewed changes

ddtrace/llmobs/_integrations/gemini.py Outdated Show resolved Hide resolved

ddtrace/llmobs/_integrations/gemini.py Outdated Show resolved Hide resolved

Yun-Kim mentioned this pull request Jul 11, 2025

chore(llmobs): refactor google util functions #13952

Draft

2 tasks

jsimpher and others added 3 commits July 11, 2025 14:55

feedback

8c3940a

snapshots

82abf40

Merge branch 'main' into jsimpher/dac-strip-io-from-gemini

9c11c39

dd-devflow bot added the mergequeue-status: waiting label Jul 14, 2025

dd-devflow bot added mergequeue-status: removed and removed mergequeue-status: waiting labels Jul 14, 2025

dd-devflow bot added mergequeue-status: waiting mergequeue-status: queued mergequeue-status: in_progress and removed mergequeue-status: removed mergequeue-status: waiting mergequeue-status: queued labels Jul 14, 2025

dd-devflow bot added mergequeue-status: removed and removed mergequeue-status: in_progress labels Jul 14, 2025

dd-devflow bot added mergequeue-status: queued mergequeue-status: in_progress and removed mergequeue-status: removed mergequeue-status: queued labels Jul 14, 2025

dd-mergequeue bot merged commit b02e328 into main Jul 14, 2025
609 of 613 checks passed

dd-mergequeue bot deleted the jsimpher/dac-strip-io-from-gemini branch July 14, 2025 21:43

dd-devflow bot added mergequeue-status: done and removed mergequeue-status: in_progress labels Jul 14, 2025

chore(llmobs): dac strip io from gemini #13874

chore(llmobs): dac strip io from gemini #13874

Uh oh!

Conversation

jsimpher commented Jul 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Checklist

Reviewer Checklist

Uh oh!

github-actions bot commented Jul 3, 2025

Uh oh!

github-actions bot commented Jul 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Bootstrap import analysis

Summary

Import time breakdown

Uh oh!

pr-commenter bot commented Jul 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmarks

scenario:iastaspects-lstrip_aspect

scenario:iastaspects-strip_aspect

scenario:iastaspectsospath-ospathbasename_aspect

scenario:iastaspectsospath-ospathjoin_aspect

scenario:iastaspectsospath-ospathnormcase_aspect

Uh oh!

Uh oh!

Uh oh!

ncybul left a comment

Choose a reason for hiding this comment

Uh oh!

Yun-Kim left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

jsimpher commented Jul 14, 2025

Uh oh!

dd-devflow-routing-codex bot commented Jul 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jsimpher commented Jul 14, 2025

Uh oh!

dd-devflow-routing-codex bot commented Jul 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jsimpher commented Jul 14, 2025

Uh oh!

dd-devflow-routing-codex bot commented Jul 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jsimpher commented Jul 14, 2025

Uh oh!

dd-devflow-routing-codex bot commented Jul 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jsimpher commented Jul 14, 2025

Uh oh!

dd-devflow-routing-codex bot commented Jul 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jsimpher commented Jul 3, 2025 •

edited

Loading

github-actions bot commented Jul 3, 2025 •

edited

Loading

pr-commenter bot commented Jul 3, 2025 •

edited

Loading

dd-devflow-routing-codex bot commented Jul 14, 2025 •

edited

Loading

dd-devflow-routing-codex bot commented Jul 14, 2025 •

edited

Loading

dd-devflow-routing-codex bot commented Jul 14, 2025 •

edited

Loading

dd-devflow-routing-codex bot commented Jul 14, 2025 •

edited

Loading

dd-devflow-routing-codex bot commented Jul 14, 2025 •

edited

Loading