[prometheusremotewrite] Otel keep sending metrics for 5 more minutes after last datapoint

[mpostument]

Component(s)

exporter/prometheusremotewrite

What happened?

Description

I was sending metrics to otel collector through otlp receiver and export them with prometheusremotewrite to Amazon Managed Prometheus. And after i stopped my agent for next 5 minutes i still saw metrics is sent to prometheus with values of last received datapoint

Steps to Reproduce

Send metrics to otlp receiver using telegraf
Stop telegraf

Expected Result

No metrics is exported to prometheus when agent is stopped

Actual Result

Metrics exported to prometheus for 5 more minutes when telegraf agent is stopped

Collector version

0.88.0

Environment information

Environment

EKS

OpenTelemetry Collector configuration

exporters:
  debug: {}
  elasticsearch/logv2:
    endpoints:
    - ENDPOINT
    logs_dynamic_index:
      enabled: true
    logs_index: otel-logs
    password: $ELASTIC_PASSWORD_V2
    user: $ELASTIC_USER_V2
  logging: {}
  otlp/o11yv2:
    endpoint: ENDPOINT
    tls:
      insecure_skip_verify: true
  prometheusremotewrite:
    auth:
      authenticator: sigv4auth
    endpoint: ENDPOINT/api/v1/remote_write
    resource_to_telemetry_conversion:
      enabled: true
    timeout: 30s
extensions:
  health_check: {}
  memory_ballast:
    size_in_percentage: 40
  sigv4auth:
    region: us-east-1
    service: aps
processors:
  attributes:
    actions:
    - action: insert
      key: env
      value: dev
  batch: {}
  memory_limiter:
    check_interval: 5s
    limit_percentage: 80
    spike_limit_percentage: 25
receivers:
  jaeger:
    protocols:
      grpc:
        endpoint: ${env:MY_POD_IP}:14250
      thrift_compact:
        endpoint: ${env:MY_POD_IP}:6831
      thrift_http:
        endpoint: ${env:MY_POD_IP}:14268
  otlp:
    protocols:
      grpc:
        endpoint: ${env:MY_POD_IP}:4317
        max_recv_msg_size_mib: 100
      http:
        endpoint: ${env:MY_POD_IP}:4318
  prometheus:
    config:
      scrape_configs:
      - job_name: opentelemetry-collector
        metric_relabel_configs:
        - replacement: gateway
          target_label: type
        scrape_interval: 10s
        static_configs:
        - targets:
          - ${MY_POD_IP}:8888
  zipkin:
    endpoint: ${env:MY_POD_IP}:9411
service:
  extensions:
  - sigv4auth
  - health_check
  pipelines:
    logs:
      exporters:
      - elasticsearch/logv2
      processors:
      - memory_limiter
      - batch
      - attributes
      receivers:
      - otlp
    metrics:
      exporters:
      - prometheusremotewrite
      processors:
      - memory_limiter
      - batch
      - attributes
      receivers:
      - otlp
      - prometheus
    traces:
      exporters:
      - otlp/o11yv2
      processors:
      - memory_limiter
      - batch
      - attributes
      receivers:
      - otlp
      - jaeger
      - zipkin
  telemetry:
    metrics:
      address: ${env:MY_POD_IP}:8888

Log output

No response

Additional context

No response

[github-actions]

Pinging code owners:

• exporter/googlemanagedprometheus: @aabmass @dashpole @jsuereth @punya @damemi @psx95

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[mpostument]

/label -exporter/googlemanagedprometheus

[mpostument]

/label exporter/prometheusremotewrite added wrong label

[github-actions]

Pinging code owners for exporter/prometheusremotewrite: @Aneurysm9 @rapphil. See Adding Labels via Comments if you do not have permissions to add labels yourself.

[dashpole]

See #27893 (comment)

[crobert-1]

I'm going to close as a duplicate of #27893, but please let us know if this is incorrect or if you have any other questions.