chore: fix/update tracer flares to the format expected #13447

juanjux · 2025-05-19T12:16:00Z

Fixes:

Default value for some remote config version fields (required by system tests).
Add uuid.
Use a strict ordering of the payload.
Check the contents of the config, not the name as per the RFC.
Don't send the API KEY since it will be forwarded by the agent.
Add more tests to check these changes.

Signed-off-by: Juanjo Alvarez juanjo.alvarezmartinez@datadoghq.com## Checklist

PR author has checked that all the criteria below are met
The PR description includes an overview of the change
The PR description articulates the motivation for the change
The change includes tests OR the PR description describes a testing strategy
The PR description notes risks associated with the change, if any
Newly-added code is easy to change
The change follows the library release note guidelines
The change includes or references documentation updates if necessary
Backport labels are set (if applicable)

Reviewer Checklist

Reviewer has checked that all the criteria below are met
Title is accurate
All changes are related to the pull request's stated goal
Avoids breaking API changes
Testing strategy adequately addresses listed risks
Newly-added code is easy to change
Release note makes sense to a user of the library
If necessary, author has acknowledged and discussed the performance implications of this PR as reported in the benchmarks PR comment
Backport labels are set in a manner that is consistent with the release branch maintenance policy

This reverts commit 592e891.

Signed-off-by: Juanjo Alvarez <juanjo.alvarezmartinez@datadoghq.com>

github-actions · 2025-05-19T12:16:35Z

CODEOWNERS have been resolved as:

scripts/trace_flares/test_flare_manual.py                               @DataDog/apm-core-python
ddtrace/internal/flare/flare.py                                         @DataDog/apm-core-python
ddtrace/internal/flare/handler.py                                       @DataDog/apm-core-python
ddtrace/internal/remoteconfig/client.py                                 @DataDog/remote-config @DataDog/apm-core-python
tests/internal/test_tracer_flare.py                                     @DataDog/apm-core-python

github-actions · 2025-05-19T12:38:00Z

Bootstrap import analysis

Comparison of import times between this PR and base.

Summary

The average import time from this PR is: 277 ± 2 ms.

The average import time from base is: 278 ± 2 ms.

The import time difference between this PR and base is: -1.75 ± 0.09 ms.

Import time breakdown

The following import paths have grown:

ddtrace.auto 0.026 ms (0.01%)

ddtrace.bootstrap.sitecustomize 0.026 ms (0.01%)

ddtrace.bootstrap.preload 0.026 ms (0.01%)

ddtrace.internal.flare.flare 0.026 ms (0.01%)

The following import paths have shrunk:

ddtrace.auto 1.860 ms (0.67%)

ddtrace.bootstrap.sitecustomize 1.191 ms (0.43%)

ddtrace.bootstrap.preload 1.191 ms (0.43%)

ddtrace.internal.remoteconfig.client 0.571 ms (0.21%)

ddtrace 0.669 ms (0.24%)

ddtrace.internal._unpatched 0.030 ms (0.01%)

json 0.030 ms (0.01%)

json.decoder 0.030 ms (0.01%)

re 0.030 ms (0.01%)

enum 0.030 ms (0.01%)

types 0.030 ms (0.01%)

pr-commenter · 2025-05-19T13:03:30Z

Benchmarks

Benchmark execution time: 2025-06-25 15:50:39

Comparing candidate commit 7dcabf9 in PR branch juanjux/fix.flares with baseline commit 101a1b1 in branch main.

Found 0 performance improvements and 0 performance regressions! Performance is the same for 541 metrics, 3 unstable metrics.

Signed-off-by: Juanjo Alvarez <juanjo.alvarezmartinez@datadoghq.com>

We make the native extension cache more fine-grained by generating hashes for each single native integration that we build. We then store each generated artifact in the cache according to this FS structure ``` .ext_cache/ # Example: └─ <module_name>/ # "ddtrace.profiling.collector" └─ <source_hash>/ # <sha256 hex digest> └─ <binary> # "collector.cpython-310-darwin.so" ``` We implement a custom command to allow retrieving the source to target mapping from `setup.py` directly, instead of coding the same information elsewhere, which would increase the maintenance cost of the whole build process. ## Checklist - [x] PR author has checked that all the criteria below are met - The PR description includes an overview of the change - The PR description articulates the motivation for the change - The change includes tests OR the PR description describes a testing strategy - The PR description notes risks associated with the change, if any - Newly-added code is easy to change - The change follows the [library release note guidelines](https://ddtrace.readthedocs.io/en/stable/releasenotes.html) - The change includes or references documentation updates if necessary - Backport labels are set (if [applicable](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting)) ## Reviewer Checklist - [ ] Reviewer has checked that all the criteria below are met - Title is accurate - All changes are related to the pull request's stated goal - Avoids breaking [API](https://ddtrace.readthedocs.io/en/stable/versioning.html#interfaces) changes - Testing strategy adequately addresses listed risks - Newly-added code is easy to change - Release note makes sense to a user of the library - If necessary, author has acknowledged and discussed the performance implications of this PR as reported in the benchmarks PR comment - Backport labels are set in a manner that is consistent with the [release branch maintenance policy](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting)

fixes error when we try to concat the string 'truncated' to bytes ## Checklist - [x] PR author has checked that all the criteria below are met - The PR description includes an overview of the change - The PR description articulates the motivation for the change - The change includes tests OR the PR description describes a testing strategy - The PR description notes risks associated with the change, if any - Newly-added code is easy to change - The change follows the [library release note guidelines](https://ddtrace.readthedocs.io/en/stable/releasenotes.html) - The change includes or references documentation updates if necessary - Backport labels are set (if [applicable](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting)) ## Reviewer Checklist - [x] Reviewer has checked that all the criteria below are met - Title is accurate - All changes are related to the pull request's stated goal - Avoids breaking [API](https://ddtrace.readthedocs.io/en/stable/versioning.html#interfaces) changes - Testing strategy adequately addresses listed risks - Newly-added code is easy to change - Release note makes sense to a user of the library - If necessary, author has acknowledged and discussed the performance implications of this PR as reported in the benchmarks PR comment - Backport labels are set in a manner that is consistent with the [release branch maintenance policy](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting) --------- Co-authored-by: Quinna Halim <quinna.halim@datadoghq.com> Co-authored-by: Brett Langdon <brett.langdon@datadoghq.com>

We do a minor setup.py cleanup, focusing on typing correctness. ## Checklist - [x] PR author has checked that all the criteria below are met - The PR description includes an overview of the change - The PR description articulates the motivation for the change - The change includes tests OR the PR description describes a testing strategy - The PR description notes risks associated with the change, if any - Newly-added code is easy to change - The change follows the [library release note guidelines](https://ddtrace.readthedocs.io/en/stable/releasenotes.html) - The change includes or references documentation updates if necessary - Backport labels are set (if [applicable](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting)) ## Reviewer Checklist - [x] Reviewer has checked that all the criteria below are met - Title is accurate - All changes are related to the pull request's stated goal - Avoids breaking [API](https://ddtrace.readthedocs.io/en/stable/versioning.html#interfaces) changes - Testing strategy adequately addresses listed risks - Newly-added code is easy to change - Release note makes sense to a user of the library - If necessary, author has acknowledged and discussed the performance implications of this PR as reported in the benchmarks PR comment - Backport labels are set in a manner that is consistent with the [release branch maintenance policy](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting)

Add telemetry support for the legacy ATO SDK. This will also be tested using system tests DataDog/system-tests#4806 APPSEC-58051 ## Checklist - [x] PR author has checked that all the criteria below are met - The PR description includes an overview of the change - The PR description articulates the motivation for the change - The change includes tests OR the PR description describes a testing strategy - The PR description notes risks associated with the change, if any - Newly-added code is easy to change - The change follows the [library release note guidelines](https://ddtrace.readthedocs.io/en/stable/releasenotes.html) - The change includes or references documentation updates if necessary - Backport labels are set (if [applicable](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting)) ## Reviewer Checklist - [x] Reviewer has checked that all the criteria below are met - Title is accurate - All changes are related to the pull request's stated goal - Avoids breaking [API](https://ddtrace.readthedocs.io/en/stable/versioning.html#interfaces) changes - Testing strategy adequately addresses listed risks - Newly-added code is easy to change - Release note makes sense to a user of the library - If necessary, author has acknowledged and discussed the performance implications of this PR as reported in the benchmarks PR comment - Backport labels are set in a manner that is consistent with the [release branch maintenance policy](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting)

This PR is a workaround for this [bug report](#13700) where LiteLLM users are running into an unhandled exception caused by trying to append a chunk choice to the list of streamed chunks at that choice index. The root of the issue comes down to where the user is setting the parameter for number of choices in the streamed response. If the `n` parameter was set as a kwarg, then there would be no issue. However, if the parameter was set elsewhere (e.g. in the user's proxy config file), then the `_streamed_chunks` list would incorrectly expect 1 choice which would lead to an indexing error. I was able to reproduce the issue with the following config and client request: Config File ``` model_list: - model_name: gpt-3.5-turbo litellm_params: model: openai/gpt-3.5-turbo api_key: "os.environ/OPENAI_API_KEY" n: 2 temperature: 0.2 ``` Client Request ``` import os import litellm import asyncio from litellm import acompletion litellm.api_key = os.environ["OPENAI_API_KEY"] async def acompletion_proxy(): messages = [{ "content": "What color is the sky?","role": "user"}] response = await acompletion(model="gpt-3.5-turbo", messages=messages, api_base="http://0.0.0.0:4000/", stream=True) async for item in response: print(item) if __name__ == "__main__": asyncio.run(acompletion_proxy()) ``` This lead to the following error: ``` Traceback (most recent call last): File "/Users/nicole.cybul/Documents/ML Observability/scripts/integrations/_simple_litellm_script.py", line 14, in <module> for item in response: File "/Users/nicole.cybul/Documents/ML Observability/scripts/.venv/lib/python3.11/site-packages/ddtrace/contrib/internal/litellm/utils.py", line 63, in __iter__ _loop_handler(chunk, self._streamed_chunks) File "/Users/nicole.cybul/Documents/ML Observability/scripts/.venv/lib/python3.11/site-packages/ddtrace/contrib/internal/litellm/utils.py", line 131, in _loop_handler streamed_chunks[choice.index].append(choice) ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^ IndexError: list index out of range ``` The fix in this PR uses a `defaultdict(list)` which should be robust against these types of indexing issues. With this new fix, the same request leads to a successful response and this [trace](https://dd.datad0g.com/llm/traces?query=%40ml_app%3Anicole-test%20%40event_type%3Aspan%20%40parent_id%3Aundefined&agg_m=count&agg_m_source=base&agg_t=count&fromUser=true&llmPanels=%5B%7B%22t%22%3A%22sampleDetailPanel%22%2C%22rEID%22%3A%22AwAAAZeEzsIo5PnrZAAAABhBWmVFenNJb0FBRDZPcUs4ZV94bEFBQUEAAAAkZjE5Nzg0Y2UtZTFlNy00YzI2LTk5MWQtMjg3YmJlNGM2ZTllAAAAIg%22%7D%5D&spanId=15697199987942968683&start=1750278947467&end=1750279847467&paused=false) in the product. ## Checklist - [x] PR author has checked that all the criteria below are met - The PR description includes an overview of the change - The PR description articulates the motivation for the change - The change includes tests OR the PR description describes a testing strategy - The PR description notes risks associated with the change, if any - Newly-added code is easy to change - The change follows the [library release note guidelines](https://ddtrace.readthedocs.io/en/stable/releasenotes.html) - The change includes or references documentation updates if necessary - Backport labels are set (if [applicable](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting)) ## Reviewer Checklist - [x] Reviewer has checked that all the criteria below are met - Title is accurate - All changes are related to the pull request's stated goal - Avoids breaking [API](https://ddtrace.readthedocs.io/en/stable/versioning.html#interfaces) changes - Testing strategy adequately addresses listed risks - Newly-added code is easy to change - Release note makes sense to a user of the library - If necessary, author has acknowledged and discussed the performance implications of this PR as reported in the benchmarks PR comment - Backport labels are set in a manner that is consistent with the [release branch maintenance policy](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting)

Signed-off-by: Juanjo Alvarez <juanjo.alvarezmartinez@datadoghq.com>

scripts/trace_flares/test_flare_manual.py

Signed-off-by: Juanjo Alvarez <juanjo.alvarezmartinez@datadoghq.com>

tests/internal/test_tracer_flare.py

Signed-off-by: Juanjo Alvarez <juanjo.alvarezmartinez@datadoghq.com>

juanjux · 2025-06-23T10:11:10Z

Tests added. Ready for review.

Signed-off-by: Juanjo Alvarez <juanjo.alvarezmartinez@datadoghq.com>

emmettbutler

Linking to the relevant system-tests code would help reviewers be certain that this change conforms to the specification and won't lead to another regression.

ddtrace/internal/flare/flare.py

Signed-off-by: Juanjo Alvarez <juanjo.alvarezmartinez@datadoghq.com>

Fixes: - Default value for some remote config version fields (required by system tests). - Add uuid. - Use a strict ordering of the payload. - Check the contents of the config, not the name as per the RFC. - Don't send the API KEY since it will be forwarded by the agent. - Add more tests to check these changes. Signed-off-by: Juanjo Alvarez <juanjo.alvarezmartinez@datadoghq.com>## Checklist - [X] PR author has checked that all the criteria below are met - The PR description includes an overview of the change - The PR description articulates the motivation for the change - The change includes tests OR the PR description describes a testing strategy - The PR description notes risks associated with the change, if any - Newly-added code is easy to change - The change follows the [library release note guidelines](https://ddtrace.readthedocs.io/en/stable/releasenotes.html) - The change includes or references documentation updates if necessary - Backport labels are set (if [applicable](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting)) ## Reviewer Checklist - [ ] Reviewer has checked that all the criteria below are met - Title is accurate - All changes are related to the pull request's stated goal - Avoids breaking [API](https://ddtrace.readthedocs.io/en/stable/versioning.html#interfaces) changes - Testing strategy adequately addresses listed risks - Newly-added code is easy to change - Release note makes sense to a user of the library - If necessary, author has acknowledged and discussed the performance implications of this PR as reported in the benchmarks PR comment - Backport labels are set in a manner that is consistent with the [release branch maintenance policy](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting) --------- Signed-off-by: Juanjo Alvarez <juanjo.alvarezmartinez@datadoghq.com> Co-authored-by: brettlangdon <brett.langdon@datadoghq.com> Co-authored-by: Gabriele N. Tornetta <P403n1x87@users.noreply.github.com> Co-authored-by: Rachel Yang <rachel.yang@datadoghq.com> Co-authored-by: Quinna Halim <quinna.halim@datadoghq.com> Co-authored-by: Christophe Papazian <114495376+christophe-papazian@users.noreply.github.com> Co-authored-by: Federico Mon <federico.mon@datadoghq.com> Co-authored-by: ncybul <124532568+ncybul@users.noreply.github.com>

brettlangdon and others added 6 commits May 14, 2025 08:55

add logging and fix tracer flare bugs

592e891

Merge remote-tracking branch 'origin/main' into brettlangdon/fix.flares

814beac

Revert "add logging and fix tracer flare bugs"

787ef3c

This reverts commit 592e891.

fix(internal): fix inconsistencies with tracer flare implementation

899f9f3

Merge branch 'main' into brettlangdon/fix.flares

a8ed143

checkpoint

c402141

Signed-off-by: Juanjo Alvarez <juanjo.alvarezmartinez@datadoghq.com>

juanjux and others added 21 commits June 11, 2025 12:14

Merge branch 'main' into juanjux/fix.flares

cfcd411

checkpoint

6cb6e8b

Signed-off-by: Juanjo Alvarez <juanjo.alvarezmartinez@datadoghq.com>

Merge remote-tracking branch 'origin/main' into juanjux/fix.flares

9c12b6c

Update log names

f724d25

Signed-off-by: Juanjo Alvarez <juanjo.alvarezmartinez@datadoghq.com>

Merge remote-tracking branch 'origin/main' into juanjux/fix.flares

7700e0e

update

aee901f

Signed-off-by: Juanjo Alvarez <juanjo.alvarezmartinez@datadoghq.com>

Merge remote-tracking branch 'origin/main' into juanjux/fix.flares

35ff080

checkpoint

3621cf3

Signed-off-by: Juanjo Alvarez <juanjo.alvarezmartinez@datadoghq.com>

Merge remote-tracking branch 'origin/main' into juanjux/fix.flares

a96d7f3

Remove debug stuff

1e379f8

Signed-off-by: Juanjo Alvarez <juanjo.alvarezmartinez@datadoghq.com>

checkpoint

600b754

Signed-off-by: Juanjo Alvarez <juanjo.alvarezmartinez@datadoghq.com>

working

de4b801

Signed-off-by: Juanjo Alvarez <juanjo.alvarezmartinez@datadoghq.com>

refactor(ci_visibility): delete InternalTestId class (#13729)

8cea616

chore(ci_visibility): allow skipping freezegun patch (#13704)

b7c736a

checkpoint

904d6d0

Signed-off-by: Juanjo Alvarez <juanjo.alvarezmartinez@datadoghq.com>

Merge branch 'main' into juanjux/fix.flares

ea279b2

checkpoint

ef33617

Signed-off-by: Juanjo Alvarez <juanjo.alvarezmartinez@datadoghq.com>

datadog-datadog-prod-us1 bot reviewed Jun 20, 2025

View reviewed changes

scripts/trace_flares/test_flare_manual.py Show resolved Hide resolved

juanjux added 2 commits June 20, 2025 18:51

Merge branch 'main' into juanjux/fix.flares

72bf557

Merge branch 'main' into juanjux/fix.flares

b7f03e5

juanjux self-assigned this Jun 20, 2025

juanjux added the changelog/no-changelog A changelog entry is not required for this PR. label Jun 20, 2025

juanjux requested a review from coignetp June 20, 2025 16:56

juanjux changed the title ~~add logging and fix tracer flare bugs~~ chore: fix/update tracer flares to the format expected Jun 20, 2025

Added more tracer flare tests

b9a0127

Signed-off-by: Juanjo Alvarez <juanjo.alvarezmartinez@datadoghq.com>

datadog-datadog-prod-us1 bot reviewed Jun 23, 2025

View reviewed changes

tests/internal/test_tracer_flare.py Outdated Show resolved Hide resolved

juanjux added 2 commits June 23, 2025 12:10

Remove two pokemon exception handlers

c39aa1e

Signed-off-by: Juanjo Alvarez <juanjo.alvarezmartinez@datadoghq.com>

Merge branch 'main' into juanjux/fix.flares

1976f45

juanjux marked this pull request as ready for review June 23, 2025 10:10

juanjux requested review from a team as code owners June 23, 2025 10:10

juanjux requested a review from erikayasuda June 23, 2025 10:10

fmt

d0a00c2

Signed-off-by: Juanjo Alvarez <juanjo.alvarezmartinez@datadoghq.com>

juanjux enabled auto-merge (squash) June 24, 2025 09:14

emmettbutler approved these changes Jun 24, 2025

View reviewed changes

ddtrace/internal/flare/flare.py Outdated Show resolved Hide resolved

juanjux added 5 commits June 25, 2025 14:57

Simplify the code

caea0c5

Signed-off-by: Juanjo Alvarez <juanjo.alvarezmartinez@datadoghq.com>

Simplify the code a little more

92de674

Signed-off-by: Juanjo Alvarez <juanjo.alvarezmartinez@datadoghq.com>

Factorize and simplify some more things

ea65d8e

Signed-off-by: Juanjo Alvarez <juanjo.alvarezmartinez@datadoghq.com>

fmt

e8462cb

Signed-off-by: Juanjo Alvarez <juanjo.alvarezmartinez@datadoghq.com>

Merge branch 'main' into juanjux/fix.flares

7dcabf9

juanjux merged commit 82ca0cf into main Jun 26, 2025
716 of 718 checks passed

juanjux deleted the juanjux/fix.flares branch June 26, 2025 11:08

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

chore: fix/update tracer flares to the format expected #13447

chore: fix/update tracer flares to the format expected #13447

Uh oh!

juanjux commented May 19, 2025 •

edited by gnufede

Loading

Uh oh!

github-actions bot commented May 19, 2025 •

edited

Loading

Uh oh!

github-actions bot commented May 19, 2025 •

edited

Loading

Uh oh!

pr-commenter bot commented May 19, 2025 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

juanjux commented Jun 23, 2025

Uh oh!

emmettbutler left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

chore: fix/update tracer flares to the format expected #13447

chore: fix/update tracer flares to the format expected #13447

Uh oh!

Conversation

juanjux commented May 19, 2025 • edited by gnufede Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Reviewer Checklist

Uh oh!

github-actions bot commented May 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented May 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Bootstrap import analysis

Summary

Import time breakdown

Uh oh!

pr-commenter bot commented May 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmarks

Uh oh!

Uh oh!

Uh oh!

juanjux commented Jun 23, 2025

Uh oh!

emmettbutler left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

juanjux commented May 19, 2025 •

edited by gnufede

Loading

github-actions bot commented May 19, 2025 •

edited

Loading

github-actions bot commented May 19, 2025 •

edited

Loading

pr-commenter bot commented May 19, 2025 •

edited

Loading