Skip to content

feat(datadog_agent): support v3 series metrics intake - #26264

Open
arfa79 wants to merge 22 commits into
vectordotdev:masterfrom
arfa79:feat/datadog-agent-v3-series
Open

arfa79 wants to merge 22 commits into
vectordotdev:masterfrom
arfa79:feat/datadog-agent-v3-series

Conversation

@arfa79

@arfa79 arfa79 commented Aug 30, 2026

Copy link
Copy Markdown
Contributor

Summary

Adds support for the Datadog Agent v3 series metrics intake format used by default by Datadog Agent 7.81.0 and newer.

The datadog_agent source now accepts the dictionary-encoded protobuf payload at /api/intake/metrics/v3/series and /api/intake/metrics/v3beta/series, translates it into Vector metrics, and preserves metric names, tags, resources, units, intervals, timestamps, values, and origin metadata.

The Datadog metrics e2e matrix also restores the floating Agent 7 test leg so current Agent versions exercise the v3 path.

References

Closes: #25790

Vector configuration

No new configuration is required. Existing datadog_agent source configurations accept the new intake endpoint automatically.

How did you test this PR?

  • cargo check --lib --no-default-features --features sources-datadog_agent
  • cargo test --lib --no-default-features --features sources-datadog_agent datadog_agent::
  • cargo clippy --lib --no-default-features --features sources-datadog_agent -- -D warnings
  • cargo fmt --all -- --check

Is this a breaking change?

  • Yes
  • No

Does this PR include user facing changes?

  • Yes. Please add a changelog fragment based on our guidelines.
  • No. A maintainer will apply the no-changelog label to this PR.

@arfa79
arfa79 requested a review from a team as a code owner August 30, 2026 12:43
Copilot AI lite review requested due to automatic review settings August 30, 2026 12:43
@github-actions github-actions Bot added the domain: sources Anything related to the Vector's sources label Aug 30, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds Datadog Agent v3 series metrics intake support to the datadog_agent source so Vector can ingest metrics from Agent versions that default to the v3 endpoint, and updates tests to exercise the new path.

Changes:

  • Add Warp routing + decoding for /api/intake/metrics/v3/series and /api/intake/metrics/v3beta/series, translating v3 payloads into Vector metrics.
  • Add unit test coverage for decoding a v3 series payload.
  • Restore the floating Datadog Agent v7 leg in the Datadog metrics E2E test matrix and add the v3 proto to codegen.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 3 comments.

Show a summary per file
File Description
tests/e2e/datadog-metrics/config/test.yaml Unpins the v7 agent matrix leg to exercise current Agent behavior (v3 default).
src/sources/datadog_agent/tests.rs Adds a v3 endpoint constant, a raw-bytes sender helper, and a v3 decode test.
src/sources/datadog_agent/mod.rs Wires in the generated v3 metrics protobuf module.
src/sources/datadog_agent/metrics.rs Adds v3 series route and decoding/translation logic.
proto/vector/dd_metric_v3.proto Introduces the v3 intake proto definition used for decoding.
changelog.d/25790_datadog_agent_v3_series.feature.md Adds a changelog fragment documenting the user-facing feature.
build.rs Adds the v3 proto to the build-time proto compilation inputs.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread src/sources/datadog_agent/metrics.rs Outdated
Comment thread src/sources/datadog_agent/metrics.rs Outdated
Comment thread src/sources/datadog_agent/metrics.rs Outdated

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 5330fba0d3

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread src/sources/datadog_agent/metrics.rs Outdated
Comment thread src/sources/datadog_agent/metrics.rs Outdated
Comment thread src/sources/datadog_agent/metrics.rs Outdated
Comment thread src/sources/datadog_agent/metrics.rs Outdated
@github-actions github-actions Bot added domain: sinks Anything related to the Vector's sinks domain: core Anything related to core crates i.e. vector-core, core-common, etc labels Aug 31, 2026

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 8be2d900a4

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread src/sources/datadog_agent/metrics.rs Outdated
Comment thread src/sources/datadog_agent/metrics.rs Outdated
Comment thread src/sources/datadog_agent/metrics.rs Outdated

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 584d5bcbee

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread src/sources/datadog_agent/metrics.rs Outdated
Comment thread src/sources/datadog_agent/metrics.rs Outdated
Comment thread src/sources/datadog_agent/metrics.rs Outdated

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: e2463c4444

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread src/sources/datadog_agent/metrics.rs Outdated
Comment thread src/sources/datadog_agent/metrics.rs Outdated
Comment thread src/sources/datadog_agent/metrics.rs Outdated
Comment thread src/sources/datadog_agent/metrics.rs Outdated

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 42d4e8684f

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread src/sources/datadog_agent/metrics.rs Outdated
Comment thread src/sources/datadog_agent/metrics.rs
Comment thread src/sources/datadog_agent/metrics.rs Outdated
Comment thread src/sources/datadog_agent/metrics.rs

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 068c2391cb

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread src/sources/datadog_agent/metrics.rs
Comment thread src/sources/datadog_agent/metrics.rs
Comment thread src/sources/datadog_agent/metrics.rs Outdated

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 4000346324

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread src/sources/datadog_agent/metrics.rs
Comment thread src/sources/datadog_agent/metrics.rs Outdated
Comment thread src/sources/datadog_agent/metrics.rs
@pront

This comment was marked as outdated.

@pront

pront commented Sep 2, 2026

Copy link
Copy Markdown
Member

Hi @arfa79, thank you for this PR. Ignore my previous message.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 8402a3b92a

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread src/sources/datadog_agent/metrics.rs
@arfa79

arfa79 commented Sep 7, 2026

Copy link
Copy Markdown
Contributor Author

@pront This branch is current with master and mergeable. Since the last automated review, commits 40003463, 068c2391, 6fa6c2d8, and dddaf830 addressed the remaining untrusted-payload allocation and CPU amplification findings, with focused regression coverage for event expansion, metadata/string/numeric predecode allocation, duplicate metadata tags, empty tagsets, and valid-string cloning. When you have capacity, could you please take a focused maintainer review? CI workflows for this fork PR may still require maintainer approval to run.

Comment thread proto/vector/dd_metric.proto Outdated
@arfa79

arfa79 commented Sep 12, 2026 •

Copy link
Copy Markdown
Contributor Author

@pront The compatibility fix is pushed and the branch is now current with master at commit 9b2c36cf. The updated fork workflows require maintainer approval before they can start. Could you please approve those runs when convenient? Local validation against current master passes: Buf breaking compatibility, Rust formatting, all 22 focused decode_v3 tests, and the v3 HTTP endpoint test. I will continue monitoring the full pipeline and address any PR-caused failures.

Comment thread tests/e2e/datadog-metrics/config/test.yaml

@pront pront left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @arfa79, thank you for this PR!

Please remove all code around v3beta.

Comment thread src/sources/datadog_agent/metrics.rs Outdated
Comment thread src/sources/datadog_agent/metrics.rs Outdated
@pront pront added the meta: awaiting author Pull requests that are awaiting their author. label Sep 25, 2026
Resolve metadata protobuf and Datadog e2e matrix conflicts. Refactor v3 decoding per maintainer feedback by documenting packed type flags, isolating reference and point cursors, centralizing pre-allocation accounting, and caching dictionary entry sizes.
@github-actions github-actions Bot removed the meta: awaiting author Pull requests that are awaiting their author. label Sep 26, 2026
…v3-series

# Conflicts:
#	lib/datadog-proto/proto/datadog/metrics/intake_v3.proto
#	lib/datadog-proto/src/lib.rs
…v3-series

# Conflicts:
#	src/sinks/datadog/metrics/encoder.rs
#	tests/e2e/datadog-metrics/config/test.yaml
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 12, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-12T11:52:37.018955Z d3369ad New commits
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@arfa79

arfa79 commented Oct 12, 2026

Copy link
Copy Markdown
Contributor Author

@pront Thanks again for the earlier review and for requesting removal of the v3beta route; that change has been addressed. I have now also resolved the latest master conflicts in commit d3369add, and GitHub reports the PR as mergeable. Local validation passes: Rust formatting, all 22 focused decode_v3 tests, the v3 HTTP endpoint test, and the affected Datadog encoder test. The PR has been open since August 30 and has been awaiting follow-up maintainer approval since the last review activity in late September. When you have capacity, could you please take another look? The remaining merge blocker is the required maintainer review; the full CI suite is still pending.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: d3369adda5

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment on lines +703 to +706
unit: event_metadata
.datadog_metric_unit()
.unwrap_or_default()
.to_string(),

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Forward preserved units through the v3 sink

When a unit-bearing Agent metric is forwarded with the datadog_metrics sink's default series_api_version: v3, this assignment only affects the v2 protobuf encoder; src/sinks/datadog/metrics/encoder_v3.rs:224-270 explicitly leaves the unit unset and never reads datadog_metric_unit, so the round trip still drops the unit. Fresh evidence after the earlier fix is the now-present v3 sink path, which retains that explicit omission; pass the metadata unit to V3Writer as well.

Useful? React with 👍 / 👎.

pass_through.category().or(Some(ORIGIN_CATEGORY_VALUE)),
pass_through.service().or(Some(no_value)),
)
.with_metric_type(pass_through.metric_type()),

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Emit the preserved no-index flag through the v3 sink

When a flagNoIndex Agent metric is forwarded with the default v3 sink, carrying metric_type through this helper does not affect the wire encoding: src/sinks/datadog/metrics/encoder_v3.rs:252-260 obtains the returned origin but calls builder.set_origin(..., false), so the output becomes indexable. Fresh evidence after the earlier fix is that the current v3 serializer still hard-codes this boolean; derive it from metric_type == 9 when emitting the series.

Useful? React with 👍 / 👎.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

domain: core Anything related to core crates i.e. vector-core, core-common, etc domain: sinks Anything related to the Vector's sinks domain: sources Anything related to the Vector's sources

Projects

None yet

Development

Successfully merging this pull request may close these issues.

datadog_agent source: support Datadog v3 series metrics intake API

3 participants