feat(metrics): add TTL-based cache for metric sets and add expire_metrics_secs for Prometheus remote write sink

[GreyLilac09]

Summary

• Add a TTL-based cache for metrics sets
• Add a expire_metrics_secs config for Prometheus remote write sink which uses the TTL-based cache
• This fixes an issue where incremental metrics are preserved for the lifetime of Vector's runtime, which causes indefinite memory growth

Vector configuration

How did you test this PR?

Example showing vector_component_allocated_bytes running for 12 hours before and after the change setting expire_metrics_secs to 120:

with the following sink:

sinks:
  prometheus:
    type: prometheus_remote_write
    inputs:
      - remap_metrics_to_prom
    endpoint: <...>
    batch:
      max_events: 5000
    buffer:
      type: memory
      max_events: 100000
      when_full: drop_newest
    **expire_metrics_secs: 120**
    healthcheck:
      enabled: false

Change Type

• Bug fix
• New feature
• Non-functional (chore, refactoring, docs)
• Performance

Is this a breaking change?

• Yes
• No

Does this PR include user facing changes?

• Yes. Please add a changelog fragment based on our guidelines.
• No. A maintainer will apply the no-changelog label to this PR.

References

• Closes: Prometheus remote write sink causes indefinite memory growth because it converts incremental to absolute values #23160
• Closes: prometheus_remote_write memory leak #20470

Notes

• Please read our Vector contributor resources.
• Do not hesitate to use @vectordotdev/vector to reach out to us regarding this PR.
• Some CI checks run only after we manually approve them.
  • We recommend adding a pre-push hook, please see this template.
  • Alternatively, we recommend running the following locally before pushing to the remote branch:
    • cargo fmt --all
    • cargo clippy --workspace --all-targets -- -D warnings
    • cargo nextest run --workspace (alternatively, you can run cargo test --all)
• After a review is requested, please avoid force pushes to help us review incrementally.
  • Feel free to push as many commits as you want. They will be squashed into one before merging.
  • For example, you can run git merge origin master and git push.
• If this PR introduces changes Vector dependencies (modifies Cargo.lock), please
  run cargo vdev build licenses to regenerate the license inventory and commit the changes (if any). More details here.

[bits-bot]

All committers have signed the CLA.

[GreyLilac09]

FYI I haven't tested this E2E as I can't get it to build on my M4 Mac.. would love if someone could get a Dockerfile up (alpine) so I can test it on my Kube cluster

[pront]

Thanks, great PR

[GreyLilac09]

@pront would you be able to run/upload a build of it so I can do a smoke test on my kube cluster?

[pront]

@pront would you be able to run/upload a build of it so I can do a smoke test on my kube cluster?

Not sure what is the ask here, do you want a custom docker image from this PR? You should be able to run scripts/build-docker.sh with a custom target repo.

[GreyLilac09]

@pront Was able to test it and it looks good (see the screenshots in the PR description), should be good to merge

[pront]

Thank you!

[GreyLilac09]

@pront hey sorry can you approve it again? not sure how this process works but it looks like the auto-merge disabled itself

[GreyLilac09]

@pront why are the tests not running when you've approved it and set to auto-merge?

[pront]

@pront why are the tests not running when you've approved it and set to auto-merge?

Each time you push a commit it requires manual approval from a maintainer. It is a security measure. It should be merged today.

[GreyLilac09]

@pront This issue also makes this issue pretty easy to resolve: #23018 (comment)

Options:

1. adding an optional incremental -> absolute conversion with expiring interval in statsd
2. on the file output plugin by adding a new codec that does incremental -> absolute conversions, or adding the option to existing codecs
3. adding a new transform entirely that just does incremental -> absolute conversions with optional expire_metrics_secs

Curious what you think what be the best approach here

[GreyLilac09]

^ will move convo to the other thread

[pront]

Adding this to the release highlights: #23586