Skip to content

feat(llmobs): [MLOB-2681] instrument openai responses with llm #13310

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 108 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
108 commits
Select commit Hold shift + click to select a range
9a982e7
MLOB-2635 add responses endpoint apm span
XG-xin Apr 28, 2025
3b21b55
collect tools metadata
XG-xin Apr 28, 2025
6a3c5e2
fix formatting
XG-xin Apr 29, 2025
6366987
adding back the test_chat_completion_azure
XG-xin Apr 29, 2025
f928b0f
remove azure test
XG-xin Apr 30, 2025
07f0964
[MLOB-2680] handling stream response
XG-xin Apr 30, 2025
3331ba6
fix span missing openai.response tags
XG-xin Apr 30, 2025
936c81a
remove steeam_option from record request
XG-xin May 1, 2025
33a9545
fix formatting
XG-xin May 1, 2025
42bb6b3
add response stream test
XG-xin May 1, 2025
978f45d
add response stream test with filtered API key
XG-xin May 1, 2025
83f1833
[MLOB-2681] add instrumentation for llm
XG-xin May 1, 2025
c9fe05b
remove openai.py
XG-xin May 1, 2025
d5f9a09
[MLOB-2681] add instrumentation for llm
XG-xin May 1, 2025
ac62931
Add back openai.py without any changes
XG-xin May 1, 2025
e30b0e7
handle stream input message
XG-xin May 1, 2025
547e959
remove llm tags
XG-xin May 1, 2025
12745dc
move SPAN_KINE within the stream
XG-xin May 1, 2025
badb6aa
move tests to test_openai file
XG-xin May 1, 2025
f68321c
Remove outdated snapshot test file
XG-xin May 1, 2025
dedb77c
skip tests for lower openai version
XG-xin May 2, 2025
01b33a0
feat/fix/docs/refactor/ci(xxx): commit title here
XG-xin May 2, 2025
1c944cf
fix errors
XG-xin May 2, 2025
359d18d
fix formatting
XG-xin May 2, 2025
afb2c51
handle async request. Also update the token usage extraction
XG-xin May 5, 2025
8041da0
fix typo
XG-xin May 5, 2025
1532c07
update test snapshopt
XG-xin May 5, 2025
6ebfca4
clean up
XG-xin May 5, 2025
ef69087
clean up the code
XG-xin May 6, 2025
1a0ebe0
update tests to only use snapshopts
XG-xin May 6, 2025
74eef11
remove outdated yaml
XG-xin May 6, 2025
e40a643
clean up endpoint_hooks, and update the set_token_metrics
XG-xin May 7, 2025
174cbd3
expend the TracedOpenAIStream helper to handle responses endpoint
XG-xin May 7, 2025
a00b0da
clean up code
XG-xin May 7, 2025
c9da6d6
fix formatting
XG-xin May 7, 2025
d66e2c1
check the chunk to type
XG-xin May 7, 2025
de87e7b
add back tokens tags
XG-xin May 7, 2025
eceafa3
record estimated tokens
XG-xin May 7, 2025
fc821db
clea up
XG-xin May 8, 2025
99371ea
ignore some response metadata
XG-xin May 8, 2025
f7f5817
fix tests
XG-xin May 8, 2025
dd4e861
update set_token_metrics and add test snapshot
XG-xin May 8, 2025
97a5e03
fix typo
XG-xin May 8, 2025
e78e596
fix comments
XG-xin May 9, 2025
2d630ce
fix formatting
XG-xin May 9, 2025
7047926
remove outdated snapshots
XG-xin May 9, 2025
6ad46f3
adding comments
XG-xin May 9, 2025
a3674a0
clean up
XG-xin May 9, 2025
5caa180
Merge branch 'main' into xinyuan/openai-responses-apm-stream
XG-xin May 9, 2025
6b36a7a
add a release note
XG-xin May 12, 2025
64c3132
fix comments and test keys
XG-xin May 12, 2025
d64a051
remove snapshot ignores
XG-xin May 12, 2025
19699b6
Merge branch 'xinyuan/openai-responses-apm-stream' into xinyuan/opena…
XG-xin May 12, 2025
f8eb92d
adding llm I/O tags
XG-xin May 14, 2025
612113e
parsing reasoning and tool calls
XG-xin May 14, 2025
ea912b4
handleing stream
XG-xin May 15, 2025
62e968f
Merge branch 'main' into xinyuan/openai-responses-llm
XG-xin May 15, 2025
4f5679a
revert change
XG-xin May 15, 2025
140081d
revert change
XG-xin May 15, 2025
a8c1571
handling stream text input and output
XG-xin May 17, 2025
d979f0e
handling tool calls
XG-xin May 18, 2025
53d8a23
handling stream reasoning
XG-xin May 19, 2025
6a1cfba
adding comments
XG-xin May 19, 2025
0089482
adding tests
XG-xin May 19, 2025
d86e9f1
adding more tests
XG-xin May 20, 2025
2782a27
update error test
XG-xin May 20, 2025
c8f6470
add instructions to input message
XG-xin May 20, 2025
134504c
use operation_type
XG-xin May 20, 2025
79b9a08
add an async test
XG-xin May 22, 2025
90aa99c
remove the redundant code from openai_set_meta_tags_from_response whe…
XG-xin May 22, 2025
0259c69
reconstruct the stream_chunk
XG-xin May 23, 2025
107e041
apply even suggestions
XG-xin May 23, 2025
f3717ef
Revert "apply even suggestions" changes
XG-xin May 23, 2025
cad2a71
refactor i/o parsing for responses api (#13497)
lievan May 23, 2025
6326f45
update tests
XG-xin May 23, 2025
f67e920
clean up
XG-xin May 23, 2025
98e3983
riotfile
May 29, 2025
f539b36
merge
May 29, 2025
150783d
requirements
May 29, 2025
ed8e50a
fix tests
XG-xin Jun 5, 2025
777071f
improving code
XG-xin Jun 12, 2025
358941a
improving code
XG-xin Jun 12, 2025
d36d64b
Merge branch 'main' into xinyuan/openai-responses-llm
Yun-Kim Jun 12, 2025
7c0c15c
minor updates
XG-xin Jun 12, 2025
bd0dd4f
improve usage parsing
XG-xin Jun 13, 2025
112e566
collect kwarg in opena_get_metadata_from_response
XG-xin Jun 13, 2025
46df490
fix format
XG-xin Jun 13, 2025
4a7da8e
remove json format
XG-xin Jun 13, 2025
6337957
update processed_item
XG-xin Jun 13, 2025
0b98f6f
fix test
XG-xin Jun 13, 2025
870e964
fix response error test
XG-xin Jun 13, 2025
9105a55
Merge branch 'main' into xinyuan/openai-responses-llm
XG-xin Jun 13, 2025
a6c4e45
add riot files
Kyle-Verhoog Jun 13, 2025
68a5223
fmt
Kyle-Verhoog Jun 13, 2025
affdf26
fix response async test
XG-xin Jun 13, 2025
b76c998
add release note
XG-xin Jun 16, 2025
2d807fb
revert openai min version
XG-xin Jun 16, 2025
02f9e90
ensure content is str
XG-xin Jun 16, 2025
1d17ce3
improve the code
XG-xin Jun 16, 2025
1cfd274
Merge branch 'xinyuan/openai-responses-llm' of github.com:DataDog/dd-…
XG-xin Jun 16, 2025
e239c06
revert change
XG-xin Jun 16, 2025
e0184dc
update openai min version
XG-xin Jun 16, 2025
8d106f7
fix arguments to be a dict
XG-xin Jun 16, 2025
6d6d534
Update ddtrace/llmobs/_integrations/utils.py
XG-xin Jun 16, 2025
0039f78
Update ddtrace/llmobs/_integrations/utils.py
XG-xin Jun 16, 2025
05917e0
Update ddtrace/llmobs/_integrations/utils.py
XG-xin Jun 16, 2025
3078f8a
Merge branch 'main' into xinyuan/openai-responses-llm
XG-xin Jun 17, 2025
37aa590
fix typo in release note
XG-xin Jun 17, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
54 changes: 0 additions & 54 deletions .riot/requirements/107d8f2.txt

This file was deleted.

54 changes: 28 additions & 26 deletions .riot/requirements/e648105.txt → .riot/requirements/109d638.txt
Original file line number Diff line number Diff line change
Expand Up @@ -2,49 +2,51 @@
# This file is autogenerated by pip-compile with Python 3.11
# by the following command:
#
# pip-compile --allow-unsafe --no-annotate .riot/requirements/e648105.in
# pip-compile --allow-unsafe --no-annotate .riot/requirements/109d638.in
#
annotated-types==0.7.0
anyio==3.7.1
attrs==25.1.0
certifi==2024.12.14
coverage[toml]==7.6.10
attrs==25.3.0
certifi==2025.4.26
coverage[toml]==7.9.1
distro==1.9.0
h11==0.14.0
httpcore==1.0.7
h11==0.16.0
httpcore==1.0.9
httpx==0.27.2
hypothesis==6.45.0
idna==3.10
iniconfig==2.0.0
mock==5.1.0
multidict==6.1.0
numpy==2.2.2
iniconfig==2.1.0
mock==5.2.0
multidict==6.4.4
numpy==2.3.0
openai[datalib,embeddings]==1.0.0
opentracing==2.4.0
packaging==24.2
pandas==2.2.3
pandas-stubs==2.2.3.241126
packaging==25.0
pandas==2.3.0
pandas-stubs==2.2.3.250527
pillow==9.5.0
pluggy==1.5.0
propcache==0.2.1
pydantic==2.10.6
pydantic-core==2.27.2
pytest==8.3.4
pluggy==1.6.0
propcache==0.3.2
pydantic==2.11.6
pydantic-core==2.33.2
pygments==2.19.1
pytest==8.4.0
pytest-asyncio==0.21.1
pytest-cov==6.0.0
pytest-mock==3.14.0
pytest-cov==6.2.1
pytest-mock==3.14.1
pytest-randomly==3.16.0
python-dateutil==2.9.0.post0
pytz==2024.2
pytz==2025.2
pyyaml==6.0.2
six==1.17.0
sniffio==1.3.1
sortedcontainers==2.4.0
tqdm==4.67.1
types-pytz==2024.2.0.20241221
typing-extensions==4.12.2
tzdata==2025.1
types-pytz==2025.2.0.20250516
typing-extensions==4.14.0
typing-inspection==0.4.1
tzdata==2025.2
urllib3==1.26.20
vcrpy==4.2.1
vcrpy==7.0.0
wrapt==1.17.2
yarl==1.18.3
yarl==1.20.1
48 changes: 0 additions & 48 deletions .riot/requirements/130158f.txt

This file was deleted.

32 changes: 16 additions & 16 deletions .riot/requirements/1e6bd37.txt → .riot/requirements/13c42e3.txt
Original file line number Diff line number Diff line change
Expand Up @@ -2,53 +2,53 @@
# This file is autogenerated by pip-compile with Python 3.8
# by the following command:
#
# pip-compile --no-annotate --resolver=backtracking .riot/requirements/1e6bd37.in
# pip-compile --allow-unsafe --no-annotate .riot/requirements/13c42e3.in
#
annotated-types==0.7.0
anyio==4.5.2
attrs==24.3.0
certifi==2024.12.14
attrs==25.3.0
certifi==2025.4.26
coverage[toml]==7.6.1
distro==1.9.0
exceptiongroup==1.2.2
h11==0.14.0
httpcore==1.0.7
exceptiongroup==1.3.0
h11==0.16.0
httpcore==1.0.9
httpx==0.27.2
hypothesis==6.45.0
idna==3.10
importlib-metadata==8.5.0
iniconfig==2.0.0
mock==5.1.0
iniconfig==2.1.0
mock==5.2.0
multidict==6.1.0
numpy==1.24.4
openai[datalib,embeddings]==1.30.1
opentracing==2.4.0
packaging==24.2
packaging==25.0
pandas==2.0.3
pandas-stubs==2.0.3.230814
pillow==9.5.0
pluggy==1.5.0
propcache==0.2.0
pydantic==2.10.5
pydantic==2.10.6
pydantic-core==2.27.2
pytest==8.3.4
pytest==8.3.5
pytest-asyncio==0.21.1
pytest-cov==5.0.0
pytest-mock==3.14.0
pytest-mock==3.14.1
pytest-randomly==3.15.0
python-dateutil==2.9.0.post0
pytz==2024.2
pytz==2025.2
pyyaml==6.0.2
six==1.17.0
sniffio==1.3.1
sortedcontainers==2.4.0
tomli==2.2.1
tqdm==4.67.1
types-pytz==2024.2.0.20241221
typing-extensions==4.12.2
tzdata==2025.1
typing-extensions==4.13.2
tzdata==2025.2
urllib3==1.26.20
vcrpy==4.2.1
vcrpy==6.0.2
wrapt==1.17.2
yarl==1.15.2
zipp==3.20.2
56 changes: 56 additions & 0 deletions .riot/requirements/1458d7e.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,56 @@
#
# This file is autogenerated by pip-compile with Python 3.9
# by the following command:
#
# pip-compile --allow-unsafe --no-annotate .riot/requirements/1458d7e.in
#
annotated-types==0.7.0
anyio==4.9.0
attrs==25.3.0
certifi==2025.4.26
coverage[toml]==7.9.1
distro==1.9.0
exceptiongroup==1.3.0
h11==0.16.0
httpcore==1.0.9
httpx==0.27.2
hypothesis==6.45.0
idna==3.10
importlib-metadata==8.7.0
iniconfig==2.1.0
mock==5.2.0
multidict==6.4.4
numpy==2.0.2
openai[datalib,embeddings]==1.30.1
opentracing==2.4.0
packaging==25.0
pandas==2.3.0
pandas-stubs==2.2.2.240807
pillow==9.5.0
pluggy==1.6.0
propcache==0.3.2
pydantic==2.11.6
pydantic-core==2.33.2
pygments==2.19.1
pytest==8.4.0
pytest-asyncio==0.21.1
pytest-cov==6.2.1
pytest-mock==3.14.1
pytest-randomly==3.16.0
python-dateutil==2.9.0.post0
pytz==2025.2
pyyaml==6.0.2
six==1.17.0
sniffio==1.3.1
sortedcontainers==2.4.0
tomli==2.2.1
tqdm==4.67.1
types-pytz==2025.2.0.20250516
typing-extensions==4.14.0
typing-inspection==0.4.1
tzdata==2025.2
urllib3==1.26.20
vcrpy==7.0.0
wrapt==1.17.2
yarl==1.20.1
zipp==3.23.0
49 changes: 49 additions & 0 deletions .riot/requirements/164ce6e.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
#
# This file is autogenerated by pip-compile with Python 3.12
# by the following command:
#
# pip-compile --allow-unsafe --no-annotate .riot/requirements/164ce6e.in
#
annotated-types==0.7.0
anyio==4.9.0
attrs==25.3.0
certifi==2025.4.26
charset-normalizer==3.4.2
coverage[toml]==7.9.1
distro==1.9.0
h11==0.16.0
httpcore==1.0.9
httpx==0.28.1
hypothesis==6.45.0
idna==3.10
iniconfig==2.1.0
jiter==0.10.0
mock==5.2.0
multidict==6.4.4
openai==1.76.2
opentracing==2.4.0
packaging==25.0
pillow==11.2.1
pluggy==1.6.0
propcache==0.3.2
pydantic==2.11.6
pydantic-core==2.33.2
pygments==2.19.1
pytest==8.4.0
pytest-asyncio==0.21.1
pytest-cov==6.2.1
pytest-mock==3.14.1
pytest-randomly==3.16.0
pyyaml==6.0.2
regex==2024.11.6
requests==2.32.4
sniffio==1.3.1
sortedcontainers==2.4.0
tiktoken==0.9.0
tqdm==4.67.1
typing-extensions==4.14.0
typing-inspection==0.4.1
urllib3==1.26.20
vcrpy==7.0.0
wrapt==1.17.2
yarl==1.20.1
Loading
Loading