Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore(iast): fix iast span metrics #13075

Merged
merged 5 commits into from
Apr 7, 2025
Merged

Conversation

avara1986
Copy link
Member

Checklist

  • PR author has checked that all the criteria below are met
  • The PR description includes an overview of the change
  • The PR description articulates the motivation for the change
  • The change includes tests OR the PR description describes a testing strategy
  • The PR description notes risks associated with the change, if any
  • Newly-added code is easy to change
  • The change follows the library release note guidelines
  • The change includes or references documentation updates if necessary
  • Backport labels are set (if applicable)

Reviewer Checklist

  • Reviewer has checked that all the criteria below are met
  • Title is accurate
  • All changes are related to the pull request's stated goal
  • Avoids breaking API changes
  • Testing strategy adequately addresses listed risks
  • Newly-added code is easy to change
  • Release note makes sense to a user of the library
  • If necessary, author has acknowledged and discussed the performance implications of this PR as reported in the benchmarks PR comment
  • Backport labels are set in a manner that is consistent with the release branch maintenance policy

@avara1986 avara1986 added changelog/no-changelog A changelog entry is not required for this PR. ASM Application Security Monitoring backport 2.21 backport 3.2 backport 3.3 labels Apr 4, 2025
Copy link
Contributor

github-actions bot commented Apr 4, 2025

CODEOWNERS have been resolved as:

ddtrace/appsec/_iast/_metrics.py                                        @DataDog/asm-python
ddtrace/appsec/_iast/_taint_tracking/_taint_objects.py                  @DataDog/asm-python
ddtrace/appsec/_iast/taint_sinks/ast_taint.py                           @DataDog/asm-python
ddtrace/appsec/_iast/taint_sinks/code_injection.py                      @DataDog/asm-python
ddtrace/appsec/_iast/taint_sinks/command_injection.py                   @DataDog/asm-python
ddtrace/appsec/_iast/taint_sinks/header_injection.py                    @DataDog/asm-python
ddtrace/appsec/_iast/taint_sinks/insecure_cookie.py                     @DataDog/asm-python
ddtrace/appsec/_iast/taint_sinks/path_traversal.py                      @DataDog/asm-python
ddtrace/appsec/_iast/taint_sinks/sql_injection.py                       @DataDog/asm-python
ddtrace/appsec/_iast/taint_sinks/ssrf.py                                @DataDog/asm-python
ddtrace/appsec/_iast/taint_sinks/weak_cipher.py                         @DataDog/asm-python
ddtrace/appsec/_iast/taint_sinks/weak_hash.py                           @DataDog/asm-python
ddtrace/appsec/_iast/taint_sinks/xss.py                                 @DataDog/asm-python
tests/appsec/integrations/django_tests/conftest.py                      @DataDog/asm-python
tests/appsec/integrations/django_tests/test_django_appsec_iast.py       @DataDog/asm-python

Copy link
Contributor

github-actions bot commented Apr 4, 2025

Bootstrap import analysis

Comparison of import times between this PR and base.

Summary

The average import time from this PR is: 249 ± 4 ms.

The average import time from base is: 238 ± 4 ms.

The import time difference between this PR and base is: 11.1 ± 0.2 ms.

Import time breakdown

The following import paths have grown:

ddtrace.auto 9.558 ms (3.84%)
ddtrace 6.423 ms (2.58%)
ddtrace._logger 4.181 ms (1.68%)
ddtrace.internal.telemetry 3.745 ms (1.51%)
ddtrace.internal.telemetry.writer 3.628 ms (1.46%)
http.client 1.152 ms (0.46%)
email.parser 0.555 ms (0.22%)
email.feedparser 0.555 ms (0.22%)
email._policybase 0.555 ms (0.22%)
email.utils 0.443 ms (0.18%)
socket 0.179 ms (0.07%)
_socket 0.058 ms (0.02%)
selectors 0.038 ms (0.02%)
array 0.019 ms (0.01%)
email._parseaddr 0.124 ms (0.05%)
calendar 0.124 ms (0.05%)
locale 0.072 ms (0.03%)
urllib.parse 0.071 ms (0.03%)
random 0.046 ms (0.02%)
math 0.017 ms (0.01%)
email.header 0.094 ms (0.04%)
binascii 0.035 ms (0.01%)
email.base64mime 0.016 ms (0.01%)
base64 0.016 ms (0.01%)
ssl 0.244 ms (0.10%)
_ssl 0.094 ms (0.04%)
email.message 0.035 ms (0.01%)
http 0.035 ms (0.01%)
ddtrace.internal.utils.http 0.794 ms (0.32%)
dataclasses 0.384 ms (0.15%)
inspect 0.323 ms (0.13%)
dis 0.103 ms (0.04%)
opcode 0.051 ms (0.02%)
_opcode 0.025 ms (0.01%)
ast 0.082 ms (0.03%)
ddtrace.internal.http 0.210 ms (0.08%)
ddtrace.internal.runtime 0.148 ms (0.06%)
uuid 0.102 ms (0.04%)
platform 0.033 ms (0.01%)
_uuid 0.032 ms (0.01%)
ddtrace.internal.forksafe 0.046 ms (0.02%)
wrapt 0.018 ms (0.01%)
wrapt.__wrapt__ 0.018 ms (0.01%)
wrapt._wrappers 0.018 ms (0.01%)
ddtrace.internal.runtime.container 0.044 ms (0.02%)
ddtrace.internal.utils 0.038 ms (0.02%)
ddtrace.internal.utils.cache 0.017 ms (0.01%)
ddtrace.internal.constants 0.015 ms (0.01%)
ddtrace.internal.utils.version 0.636 ms (0.26%)
ddtrace.vendor.packaging.version 0.636 ms (0.26%)
ddtrace.vendor.packaging 0.456 ms (0.18%)
ddtrace.vendor 0.456 ms (0.18%)
ddtrace.internal.module 0.456 ms (0.18%)
ddtrace.internal.wrapping.context 0.390 ms (0.16%)
ddtrace.internal.wrapping 0.318 ms (0.13%)
bytecode 0.289 ms (0.12%)
bytecode.bytecode 0.180 ms (0.07%)
bytecode.flags 0.150 ms (0.06%)
bytecode.instr 0.128 ms (0.05%)
bytecode.cfg 0.090 ms (0.04%)
bytecode.concrete 0.043 ms (0.02%)
contextvars 0.031 ms (0.01%)
_contextvars 0.031 ms (0.01%)
ddtrace.internal.encoding 0.326 ms (0.13%)
ddtrace.internal._encoding 0.059 ms (0.02%)
ddtrace.internal.telemetry.data 0.237 ms (0.10%)
ddtrace.internal.packages 0.183 ms (0.07%)
_sysconfigdata__linux_x86_64-linux-gnu 0.078 ms (0.03%)
ddtrace.settings.third_party 0.021 ms (0.01%)
sysconfig 0.054 ms (0.02%)
ddtrace.settings._agent 0.173 ms (0.07%)
ddtrace.settings._core 0.145 ms (0.06%)
ddtrace.internal.native 0.058 ms (0.02%)
ddtrace.internal.native._native 0.058 ms (0.02%)
envier 0.045 ms (0.02%)
envier.env 0.045 ms (0.02%)
ddtrace.settings._telemetry 0.108 ms (0.04%)
ddtrace.settings._inferred_base_service 0.075 ms (0.03%)
pathlib 0.059 ms (0.02%)
ddtrace.internal.periodic 0.076 ms (0.03%)
ddtrace.internal._threads 0.056 ms (0.02%)
ddtrace.internal.service 0.020 ms (0.01%)
ddtrace.internal.atexit 0.030 ms (0.01%)
signal 0.030 ms (0.01%)
ddtrace.internal.telemetry.metrics 0.021 ms (0.01%)
ddtrace.internal.telemetry.logging 0.019 ms (0.01%)
ddtrace.internal.telemetry.constants 0.030 ms (0.01%)
ddtrace.settings._otel_remapper 0.016 ms (0.01%)
logging 0.334 ms (0.13%)
traceback 0.261 ms (0.10%)
contextlib 0.191 ms (0.08%)
linecache 0.042 ms (0.02%)
tokenize 0.042 ms (0.02%)
typing 0.102 ms (0.04%)
ddtrace.trace 0.933 ms (0.37%)
ddtrace._trace.tracer 0.380 ms (0.15%)
ddtrace.internal.dogstatsd 0.160 ms (0.06%)
ddtrace.vendor.dogstatsd 0.160 ms (0.06%)
ddtrace.vendor.dogstatsd.base 0.160 ms (0.06%)
ddtrace.vendor.dogstatsd.container 0.035 ms (0.01%)
queue 0.033 ms (0.01%)
heapq 0.033 ms (0.01%)
_heapq 0.033 ms (0.01%)
ddtrace.vendor.dogstatsd.context 0.019 ms (0.01%)
ddtrace.vendor.dogstatsd.context_async 0.019 ms (0.01%)
ddtrace.internal.processor.endpoint_call_counter 0.073 ms (0.03%)
ddtrace.internal.debug 0.034 ms (0.01%)
ddtrace.internal.peer_service.processor 0.020 ms (0.01%)
ddtrace._trace.filters 0.362 ms (0.15%)
ddtrace._trace.processor 0.341 ms (0.14%)
ddtrace._trace.sampler 0.234 ms (0.09%)
ddtrace._trace.span 0.234 ms (0.09%)
ddtrace.internal.sampling 0.095 ms (0.04%)
ddtrace.internal.rate_limiter 0.044 ms (0.02%)
ddtrace.internal._rand 0.049 ms (0.02%)
pprint 0.030 ms (0.01%)
ddtrace.internal.writer 0.069 ms (0.03%)
ddtrace.internal.writer.writer 0.069 ms (0.03%)
ddtrace._trace.context 0.090 ms (0.04%)
ddtrace._trace._span_link 0.045 ms (0.02%)
ddtrace.settings._config 0.803 ms (0.32%)
ddtrace.internal.gitmetadata 0.289 ms (0.12%)
ddtrace.ext.ci 0.289 ms (0.12%)
ddtrace.ext.git 0.241 ms (0.10%)
shutil 0.103 ms (0.04%)
zlib 0.035 ms (0.01%)
subprocess 0.044 ms (0.02%)
tempfile 0.032 ms (0.01%)
ddtrace.internal._file_queue 0.072 ms (0.03%)
secrets 0.050 ms (0.02%)
hmac 0.050 ms (0.02%)
_hashlib 0.050 ms (0.02%)
ddtrace.internal.schema 0.041 ms (0.02%)
ddtrace.internal.schema.span_attribute_schema 0.041 ms (0.02%)
ddtrace._monkey 0.326 ms (0.13%)
ddtrace.appsec 0.150 ms (0.06%)
ddtrace.internal.core 0.150 ms (0.06%)
ddtrace.internal.core.event_hub 0.104 ms (0.04%)
ddtrace.settings.asm 0.121 ms (0.05%)
ddtrace.appsec._constants 0.053 ms (0.02%)
ddtrace.internal._unpatched 0.180 ms (0.07%)
json 0.136 ms (0.05%)
json.decoder 0.136 ms (0.05%)
re 0.076 ms (0.03%)
enum 0.076 ms (0.03%)
functools 0.028 ms (0.01%)
json.scanner 0.030 ms (0.01%)
_json 0.030 ms (0.01%)
threading 0.043 ms (0.02%)
ddtrace.bootstrap.sitecustomize 3.135 ms (1.26%)
ddtrace.bootstrap.preload 2.562 ms (1.03%)
ddtrace.internal.products 1.308 ms (0.53%)
importlib.metadata 0.559 ms (0.22%)
importlib.abc 0.306 ms (0.12%)
importlib.resources 0.028 ms (0.01%)
importlib.resources._common 0.028 ms (0.01%)
csv 0.071 ms (0.03%)
_csv 0.045 ms (0.02%)
zipfile 0.066 ms (0.03%)
ddtrace.internal.symbol_db.remoteconfig 0.255 ms (0.10%)
ddtrace.internal.symbol_db.symbols 0.173 ms (0.07%)
ddtrace.settings.symbol_db 0.035 ms (0.01%)
ddtrace.internal.remoteconfig._connectors 0.165 ms (0.07%)
ctypes 0.132 ms (0.05%)
_ctypes 0.065 ms (0.03%)
multiprocessing 0.151 ms (0.06%)
multiprocessing.context 0.151 ms (0.06%)
multiprocessing.reduction 0.119 ms (0.05%)
pickle 0.119 ms (0.05%)
_pickle 0.043 ms (0.02%)
ddtrace.settings.dynamic_instrumentation 0.067 ms (0.03%)
ddtrace.internal.remoteconfig.worker 0.057 ms (0.02%)
multiprocessing.sharedctypes 0.054 ms (0.02%)
multiprocessing.heap 0.026 ms (0.01%)
mmap 0.026 ms (0.01%)
ddtrace.settings.profiling 0.711 ms (0.29%)
ddtrace.vendor.psutil 0.568 ms (0.23%)
ddtrace.vendor.psutil._common 0.354 ms (0.14%)
ddtrace.vendor.psutil._pslinux 0.078 ms (0.03%)
ddtrace.vendor.psutil._psutil_linux 0.043 ms (0.02%)
glob 0.035 ms (0.01%)
ddtrace.vendor.psutil._compat 0.032 ms (0.01%)
ddtrace.internal.datadog.profiling.ddup 0.042 ms (0.02%)
ddtrace.internal.datadog.profiling.ddup._ddup 0.042 ms (0.02%)
ddtrace.internal.flare.flare 0.152 ms (0.06%)
logging.handlers 0.062 ms (0.03%)
ddtrace.settings.crashtracker 0.110 ms (0.04%)
ddtrace.internal.datadog.profiling.crashtracker 0.070 ms (0.03%)
ddtrace.internal.datadog.profiling.crashtracker._crashtracker 0.070 ms (0.03%)
ddtrace.appsec._remoteconfiguration 0.050 ms (0.02%)
ddtrace.appsec._capabilities 0.050 ms (0.02%)
ddtrace.internal.core.crashtracking 0.029 ms (0.01%)
ddtrace._trace.trace_handlers 0.269 ms (0.11%)
ddtrace._trace._inferred_proxy 0.136 ms (0.05%)
ddtrace.propagation.http 0.136 ms (0.05%)
ddtrace.internal._tagset 0.043 ms (0.02%)
ddtrace.contrib.trace_utils 0.056 ms (0.02%)
ddtrace.contrib.internal.trace_utils 0.056 ms (0.02%)
ddtrace.appsec._common_module_patches 0.052 ms (0.02%)
ddtrace.appsec._asm_request_context 0.052 ms (0.02%)
shlex 0.037 ms (0.01%)

The following import paths have shrunk:

ddtrace.auto 1.045 ms (0.42%)
ddtrace 0.634 ms (0.25%)
ddtrace.bootstrap.sitecustomize 0.412 ms (0.17%)
ddtrace.bootstrap.preload 0.412 ms (0.17%)
ddtrace.internal.products 0.412 ms (0.17%)
ddtrace.internal.remoteconfig.client 0.412 ms (0.17%)

@pr-commenter
Copy link

pr-commenter bot commented Apr 4, 2025

Benchmarks

Benchmark execution time: 2025-04-07 09:12:16

Comparing candidate commit f191b68 in PR branch avara1986/fix_iast_span_metrics with baseline commit 93d1e12 in branch main.

Found 0 performance improvements and 0 performance regressions! Performance is the same for 498 metrics, 2 unstable metrics.

@avara1986 avara1986 marked this pull request as ready for review April 4, 2025 19:12
@avara1986 avara1986 requested a review from a team as a code owner April 4, 2025 19:12
Copy link
Contributor

@Copilot Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot reviewed 15 out of 15 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (1)

ddtrace/appsec/taint_sinks/code_injection.py:84

  • The variable 'reported' is initialized but never updated even when a vulnerability is reported. Consider setting reported = True after a successful call to CodeInjection.report to accurately reflect that a vulnerability was detected.
def _iast_report_code_injection(code_string: Text):

@avara1986 avara1986 merged commit 9c8c6d0 into main Apr 7, 2025
457 of 458 checks passed
@avara1986 avara1986 deleted the avara1986/fix_iast_span_metrics branch April 7, 2025 09:25
Copy link
Contributor

github-actions bot commented Apr 7, 2025

The backport to 2.21 failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add .worktrees/backport-2.21 2.21
# Navigate to the new working tree
cd .worktrees/backport-2.21
# Create a new branch
git switch --create backport-13075-to-2.21
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 9c8c6d0eb66c67ae261329c0f098dd32b6e67b21
# Push it to GitHub
git push --set-upstream origin backport-13075-to-2.21
# Go back to the original working tree
cd ../..
# Delete the working tree
git worktree remove .worktrees/backport-2.21

Then, create a pull request where the base branch is 2.21 and the compare/head branch is backport-13075-to-2.21.

Copy link
Contributor

github-actions bot commented Apr 7, 2025

The backport to 3.2 failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add .worktrees/backport-3.2 3.2
# Navigate to the new working tree
cd .worktrees/backport-3.2
# Create a new branch
git switch --create backport-13075-to-3.2
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 9c8c6d0eb66c67ae261329c0f098dd32b6e67b21
# Push it to GitHub
git push --set-upstream origin backport-13075-to-3.2
# Go back to the original working tree
cd ../..
# Delete the working tree
git worktree remove .worktrees/backport-3.2

Then, create a pull request where the base branch is 3.2 and the compare/head branch is backport-13075-to-3.2.

Copy link
Contributor

github-actions bot commented Apr 7, 2025

The backport to 3.3 failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add .worktrees/backport-3.3 3.3
# Navigate to the new working tree
cd .worktrees/backport-3.3
# Create a new branch
git switch --create backport-13075-to-3.3
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 9c8c6d0eb66c67ae261329c0f098dd32b6e67b21
# Push it to GitHub
git push --set-upstream origin backport-13075-to-3.3
# Go back to the original working tree
cd ../..
# Delete the working tree
git worktree remove .worktrees/backport-3.3

Then, create a pull request where the base branch is 3.3 and the compare/head branch is backport-13075-to-3.3.

avara1986 added a commit that referenced this pull request Apr 7, 2025
- [x] PR author has checked that all the criteria below are met
- The PR description includes an overview of the change
- The PR description articulates the motivation for the change
- The change includes tests OR the PR description describes a testing
strategy
- The PR description notes risks associated with the change, if any
- Newly-added code is easy to change
- The change follows the [library release note
guidelines](https://ddtrace.readthedocs.io/en/stable/releasenotes.html)
- The change includes or references documentation updates if necessary
- Backport labels are set (if
[applicable](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting))

- [x] Reviewer has checked that all the criteria below are met
- Title is accurate
- All changes are related to the pull request's stated goal
- Avoids breaking
[API](https://ddtrace.readthedocs.io/en/stable/versioning.html#interfaces)
changes
- Testing strategy adequately addresses listed risks
- Newly-added code is easy to change
- Release note makes sense to a user of the library
- If necessary, author has acknowledged and discussed the performance
implications of this PR as reported in the benchmarks PR comment
- Backport labels are set in a manner that is consistent with the
[release branch maintenance
policy](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting)

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
(cherry picked from commit 9c8c6d0)
chojomok pushed a commit that referenced this pull request Apr 7, 2025
## Checklist
- [x] PR author has checked that all the criteria below are met
- The PR description includes an overview of the change
- The PR description articulates the motivation for the change
- The change includes tests OR the PR description describes a testing
strategy
- The PR description notes risks associated with the change, if any
- Newly-added code is easy to change
- The change follows the [library release note
guidelines](https://ddtrace.readthedocs.io/en/stable/releasenotes.html)
- The change includes or references documentation updates if necessary
- Backport labels are set (if
[applicable](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting))

## Reviewer Checklist
- [x] Reviewer has checked that all the criteria below are met 
- Title is accurate
- All changes are related to the pull request's stated goal
- Avoids breaking
[API](https://ddtrace.readthedocs.io/en/stable/versioning.html#interfaces)
changes
- Testing strategy adequately addresses listed risks
- Newly-added code is easy to change
- Release note makes sense to a user of the library
- If necessary, author has acknowledged and discussed the performance
implications of this PR as reported in the benchmarks PR comment
- Backport labels are set in a manner that is consistent with the
[release branch maintenance
policy](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting)

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
avara1986 added a commit that referenced this pull request Apr 9, 2025
backport #13075 to 2.21

Partial backport of #13046 to align the functions parameters.


(cherry picked from commit 9c8c6d0)

## Checklist
- [x] PR author has checked that all the criteria below are met
- The PR description includes an overview of the change
- The PR description articulates the motivation for the change
- The change includes tests OR the PR description describes a testing
strategy
- The PR description notes risks associated with the change, if any
- Newly-added code is easy to change
- The change follows the [library release note
guidelines](https://ddtrace.readthedocs.io/en/stable/releasenotes.html)
- The change includes or references documentation updates if necessary
- Backport labels are set (if
[applicable](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting))

## Reviewer Checklist
- [x] Reviewer has checked that all the criteria below are met 
- Title is accurate
- All changes are related to the pull request's stated goal
- Avoids breaking
[API](https://ddtrace.readthedocs.io/en/stable/versioning.html#interfaces)
changes
- Testing strategy adequately addresses listed risks
- Newly-added code is easy to change
- Release note makes sense to a user of the library
- If necessary, author has acknowledged and discussed the performance
implications of this PR as reported in the benchmarks PR comment
- Backport labels are set in a manner that is consistent with the
[release branch maintenance
policy](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ASM Application Security Monitoring backport 2.21 backport 3.2 backport 3.3 changelog/no-changelog A changelog entry is not required for this PR.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants