Skip to content

fix(standalone): stamp _dd.apm.enabled on every exported chunk - #9483

Merged
BridgeAR merged 1 commit into
masterfrom
ishabi/apm-enabled-outbound-span
Jul 22, 2026
Merged

BridgeAR merged 1 commit into
masterfrom
ishabi/apm-enabled-outbound-span

Conversation

@IlyasShabi

@IlyasShabi IlyasShabi commented Jul 22, 2026 •

Copy link
Copy Markdown
Contributor

What does this PR do?

Moves the _dd.apm.enabled:0 billing marker from a span-start hook to the chunk export path in SpanProcessor.
When apmTracingEnabled === false, the first span of every exported chunk is stamped with the marker. The old start-time hook in standalone/index.js is removed (redundancy)

Motivation

Standalone ASM customers (DD_APM_TRACING_ENABLED=false) rely on the _dd.apm.enabled:0 marker to opt out of APM host billing. The intake emits an APM billing point for a trace chunk unless at least one span in that chunk carries this marker.

Current behavior: At span start, we stamped _dd.apm.enabled:0 only on spans with no parent or a remote parent. The assumption was that this local root would always be present in every exported chunk.

Issue with outbound spans: When a child span outlives its parent, the parent flushes first in its own chunk (with the marker), and the child flushes later in a separate chunk. That second chunk contains no local root — the child had a local parent at start, so the guard skipped it. The intake sees a chunk with no marker and charges APM host billing.

Reported case: a fastify request span finished immediately, while its outbound http.request child kept running for 10s. The outbound arrived alone, unmarked, and triggered APM billing on an ASM-only service.

Solution: Make the decision at chunk export instead of span start. The export loop is the one place that unambiguously knows the chunk boundaries, so stamping the first span there guarantees every exported chunk carries the marker.

Additional Notes

  • This is a hotfix for an ongoing incident
  • asm_stanalone ST scenario works well

@dd-octo-sts

dd-octo-sts Bot commented Jul 22, 2026

Copy link
Copy Markdown
Contributor

Overall package size

Self size: 7.51 MB
Deduped: 8.17 MB
No deduping: 8.17 MB

Dependency sizes | name | version | self size | total size | |------|---------|-----------|------------| | import-in-the-middle | 3.3.2 | 124.41 kB | 440.65 kB | | opentracing | 0.14.7 | 194.81 kB | 194.81 kB | | dc-polyfill | 0.1.11 | 25.74 kB | 25.74 kB |

🤖 This report was automatically generated by heaviest-objects-in-the-universe

@datadog-official

datadog-official Bot commented Jul 22, 2026 •

Copy link
Copy Markdown

Tests

🎉 All green!

🧪 All tests passed
❄️ No new flaky tests detected

🔄 Datadog retried 1 test - 1 passed on retry View in Datadog

🎯 Code Coverage (details)
• Patch Coverage: 100.00%
• Overall Coverage: 98.44% (-0.00%)

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: bacd947 | Docs | Datadog PR Page | Give us feedback!

@IlyasShabi
IlyasShabi marked this pull request as ready for review July 22, 2026 08:58
@IlyasShabi
IlyasShabi requested review from a team as code owners July 22, 2026 08:58
@IlyasShabi
IlyasShabi requested review from khanayan123 and removed request for a team July 22, 2026 08:58
@codecov

codecov Bot commented Jul 22, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 98.44%. Comparing base (293cf61) to head (bacd947).
⚠️ Report is 2 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #9483      +/-   ##
==========================================
+ Coverage   96.54%   98.44%   +1.89%     
==========================================
  Files         927      943      +16     
  Lines      126652   127248     +596     
  Branches    10431    10827     +396     
==========================================
+ Hits       122274   125266    +2992     
+ Misses       4378     1982    -2396     
Flag Coverage Δ
aiguard 57.84% <100.00%> (-0.02%) ⬇️
aiguard-integration 56.22% <100.00%> (-0.02%) ⬇️
apm-bucket-0 57.70% <100.00%> (-4.62%) ⬇️
apm-bucket-1 64.00% <100.00%> (-0.74%) ⬇️
apm-bucket-2 62.76% <100.00%> (-0.02%) ⬇️
apm-bucket-3 60.12% <100.00%> (-0.02%) ⬇️
apm-capabilities-tracing 64.49% <100.00%> (-0.01%) ⬇️
apm-integrations-aerospike 56.71% <100.00%> (-0.02%) ⬇️
apm-integrations-confluentinc-kafka-javascript 61.64% <100.00%> (-0.02%) ⬇️
apm-integrations-couchbase 57.13% <100.00%> (-0.03%) ⬇️
apm-integrations-http 62.79% <100.00%> (-0.02%) ⬇️
apm-integrations-kafkajs 62.20% <100.00%> (-0.02%) ⬇️
apm-integrations-next 59.24% <100.00%> (-0.02%) ⬇️
apm-integrations-prisma 58.72% <100.00%> (-0.02%) ⬇️
apm-integrations-sharedb_integrations-tedious ?
appsec 72.93% <100.00%> (-0.01%) ⬇️
appsec-express_fastify_graphql 70.43% <100.00%> (-0.02%) ⬇️
appsec-integration 51.39% <100.00%> (+0.01%) ⬆️
appsec-kafka_ldapjs_lodash 64.00% <100.00%> (-0.02%) ⬇️
appsec-mongodb-core_mongoose_mysql 67.72% <100.00%> (-0.02%) ⬇️
appsec-next 57.56% <100.00%> (-0.02%) ⬇️
appsec-node-serialize ?
appsec-node-serialize_passport_postgres 67.40% <100.00%> (-0.02%) ⬇️
appsec-passport ?
appsec-sourcing_stripe_template 65.74% <100.00%> (-0.02%) ⬇️
debugger 64.90% <100.00%> (-0.05%) ⬇️
instrumentations-bucket-0 51.66% <20.00%> (-0.82%) ⬇️
instrumentations-bucket-1 60.25% <100.00%> (-3.05%) ⬇️
instrumentations-bucket-10 61.82% <100.00%> (-0.02%) ⬇️
instrumentations-bucket-11 56.73% <100.00%> (-0.02%) ⬇️
instrumentations-bucket-12 52.03% <20.00%> (-0.03%) ⬇️
instrumentations-bucket-13 52.05% <20.00%> (-0.03%) ⬇️
instrumentations-bucket-2 53.35% <20.00%> (-0.03%) ⬇️
instrumentations-bucket-3 53.65% <20.00%> (-0.03%) ⬇️
instrumentations-bucket-4 59.25% <100.00%> (-0.02%) ⬇️
instrumentations-bucket-5 50.06% <20.00%> (-0.03%) ⬇️
instrumentations-bucket-6 60.96% <100.00%> (-0.02%) ⬇️
instrumentations-bucket-7 58.40% <100.00%> (-0.02%) ⬇️
instrumentations-bucket-8 59.49% <100.00%> (-0.01%) ⬇️
instrumentations-bucket-9 52.04% <20.00%> (-0.03%) ⬇️
instrumentations-instrumentation-couchbase 51.10% <20.00%> (-0.03%) ⬇️
instrumentations-instrumentation-router ?
instrumentations-instrumentation-zlib 51.69% <20.00%> (-0.03%) ⬇️
instrumentations-integration-esbuild 34.11% <20.00%> (-0.01%) ⬇️
llmobs-ai_anthropic_bedrock 62.60% <100.00%> (-0.02%) ⬇️
llmobs-bucket-1 61.56% <100.00%> (-0.02%) ⬇️
llmobs-openai 62.73% <100.00%> (-0.02%) ⬇️
llmobs-sdk 66.23% <100.00%> (-0.02%) ⬇️
llmobs-vertex-ai 59.22% <100.00%> (-0.02%) ⬇️
master-coverage 98.44% <100.00%> (?)
openfeature 56.21% <60.00%> (+<0.01%) ⬆️
openfeature-unit 53.47% <20.00%> (-0.03%) ⬇️
platform-core_esbuild_instrumentations-misc 40.40% <20.00%> (-0.02%) ⬇️
platform-integration 61.50% <60.00%> (-0.01%) ⬇️
platform-shimmer_unit-guardrails_webpack 39.10% <20.00%> (-0.02%) ⬇️
platform-unit-guardrails ?
plugins-axios_body-parser_bullmq ?
plugins-bucket-0 57.11% <100.00%> (-0.02%) ⬇️
plugins-bucket-1 54.43% <60.00%> (+<0.01%) ⬆️
plugins-bucket-11 62.35% <100.00%> (-0.02%) ⬇️
plugins-bucket-14 ?
plugins-bucket-15 ?
plugins-bucket-16 ?
plugins-bucket-17 ?
plugins-bucket-18 62.03% <100.00%> (-0.02%) ⬇️
plugins-bucket-19 60.13% <100.00%> (-0.02%) ⬇️
plugins-bucket-2 ?
plugins-bucket-20 62.14% <100.00%> (-0.02%) ⬇️
plugins-bucket-4 58.63% <100.00%> (-0.02%) ⬇️
plugins-bullmq_cassandra_cookie 61.82% <100.00%> (-0.02%) ⬇️
plugins-cassandra_cookie_cookie-parser ?
plugins-cookie-parser_crypto_dd-trace-api 56.81% <100.00%> (-0.02%) ⬇️
plugins-fastify_fetch_fs ?
plugins-fetch_fs_generic-pool 58.84% <100.00%> (-0.06%) ⬇️
plugins-generic-pool_grpc_handlebars ?
plugins-google-cloud-pubsub_grpc_handlebars 64.76% <100.00%> (-0.02%) ⬇️
plugins-hapi_hono_ioredis 60.29% <100.00%> (-0.02%) ⬇️
plugins-jest_knex_langgraph 55.67% <100.00%> (-0.02%) ⬇️
plugins-jest_knex_ldapjs ?
plugins-ldapjs_light-my-request_limitd-client 58.57% <100.00%> (-0.02%) ⬇️
plugins-light-my-request_limitd-client_lodash ?
plugins-lodash_mariadb_memcached 58.07% <100.00%> (-0.02%) ⬇️
plugins-mariadb_memcached_mercurius ?
plugins-microgateway-core_moleculer_mongodb ?
plugins-moleculer_mongodb_mongodb-core 61.99% <100.00%> (-0.02%) ⬇️
plugins-mongodb-core_mongoose_multer ?
plugins-mongoose_multer_mysql 59.09% <100.00%> (-0.02%) ⬇️
plugins-mysql2_nats_node-serialize 60.79% <100.00%> (-0.02%) ⬇️
plugins-mysql_mysql2_nats ?
plugins-opensearch_passport-http_pino 59.61% <100.00%> (-0.02%) ⬇️
plugins-passport-http_pino_postgres ?
plugins-postgres_process_pug 58.35% <100.00%> (-0.02%) ⬇️
plugins-process_pug_redis ?
plugins-redis_router_sequelize 62.12% <100.00%> (-0.02%) ⬇️
plugins-test-and-upstream-rhea_undici_url 61.70% <100.00%> (-0.02%) ⬇️
plugins-url_valkey_vm ?
plugins-valkey_vm_winston 58.08% <100.00%> (?)
plugins-winston_ws ?
plugins-ws 59.65% <100.00%> (-0.02%) ⬇️
profiling 62.22% <100.00%> (?)
serverless-aws-sdk-aws-sdk 55.13% <100.00%> (-0.02%) ⬇️
serverless-aws-sdk-bedrockruntime 54.83% <100.00%> (-0.02%) ⬇️
serverless-aws-sdk-client 56.49% <100.00%> (-0.02%) ⬇️
serverless-aws-sdk-dynamodb 55.73% <100.00%> (-0.02%) ⬇️
serverless-aws-sdk-eventbridge 49.44% <20.00%> (-0.03%) ⬇️
serverless-aws-sdk-kinesis 59.42% <100.00%> (-0.02%) ⬇️
serverless-aws-sdk-lambda 57.46% <100.00%> (-0.02%) ⬇️
serverless-aws-sdk-s3 55.68% <100.00%> (-0.02%) ⬇️
serverless-aws-sdk-serverless-peer-service 59.82% <100.00%> (-0.02%) ⬇️
serverless-aws-sdk-sns 60.21% <100.00%> (-0.02%) ⬇️
serverless-aws-sdk-sqs 60.64% <100.00%> (-0.02%) ⬇️
serverless-aws-sdk-stepfunctions 55.66% <100.00%> (-0.02%) ⬇️
serverless-aws-sdk-util 51.55% <20.00%> (-0.03%) ⬇️
serverless-bucket-0 54.33% <60.00%> (+<0.01%) ⬆️
serverless-bucket-1 59.33% <100.00%> (+5.78%) ⬆️
test-optimization-cucumber 71.91% <100.00%> (+0.01%) ⬆️
test-optimization-cypress 65.81% <60.00%> (+0.07%) ⬆️
test-optimization-jest 73.23% <100.00%> (-0.06%) ⬇️
test-optimization-mocha 73.49% <100.00%> (+7.51%) ⬆️
test-optimization-playwright-playwright-atr 60.39% <60.00%> (+0.53%) ⬆️
test-optimization-playwright-playwright-efd 60.58% <60.00%> (+0.16%) ⬆️
test-optimization-playwright-playwright-final-status 60.70% <60.00%> (+0.38%) ⬆️
test-optimization-playwright-playwright-impacted-tests 60.28% <60.00%> (-0.02%) ⬇️
test-optimization-playwright-playwright-reporting 61.56% <60.00%> (-0.13%) ⬇️
test-optimization-playwright-playwright-test-management 61.09% <60.00%> (-0.13%) ⬇️
test-optimization-playwright-playwright-test-span 60.49% <60.00%> (?)
test-optimization-selenium 60.03% <60.00%> (?)
test-optimization-testopt 58.47% <60.00%> (-1.89%) ⬇️
test-optimization-vitest 70.28% <100.00%> (+3.84%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@BridgeAR BridgeAR left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, just left a nit. I believe the test case could be a bit more robust, but it should be fine

Comment on lines +65 to +66
if (isFirstSpanInChunk && stampApmDisabled) {
formattedSpan.metrics[APM_TRACING_ENABLED_KEY] = 0

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit (the single variable is better to optimize away)

Suggested change
if (isFirstSpanInChunk && stampApmDisabled) {
formattedSpan.metrics[APM_TRACING_ENABLED_KEY] = 0
if (stampApmDisabled) {
formattedSpan.metrics[APM_TRACING_ENABLED_KEY] = 0
stampApmDisabled = false

@BridgeAR
BridgeAR merged commit 42819b5 into master Jul 22, 2026
661 checks passed
@BridgeAR
BridgeAR deleted the ishabi/apm-enabled-outbound-span branch July 22, 2026 16:56
@dd-octo-sts dd-octo-sts Bot mentioned this pull request Jul 22, 2026
This was referenced Jul 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants