[exporter/awsemf] Exporter does not read the config flag to not ignore the first batch of a metric send to cloudwatch

[roseo1]

Component(s)

exporter/awsemf

What happened?

Description

The awsemf exporter was not sending data for the first values of metrics, so we bumped to a version higher than v0.73.0 where retain_initial_value_of_delta_metric was added to be able to keep those values.

But we found that we still did not get the initial value.

The flag is on metadata on the datapoint slice which is used for the check

opentelemetry-collector-contrib/exporter/awsemfexporter/datapoint.go

Line 133 in 7576b11

if !retained && dps.retainInitialValueForDelta {

Which is passed in the grouping

opentelemetry-collector-contrib/exporter/awsemfexporter/grouped_metric.go

Line 43 in 7576b11

dps := getDataPoints(pmd, metadata, logger)

But the retain flag is not passed from the config to the metric metadata

opentelemetry-collector-contrib/exporter/awsemfexporter/metric_translator.go

Line 122 in 7576b11

groupedMetricMetadata: groupedMetricMetadata{

Steps to Reproduce

Believe the steps to reproduce are the same as #16218.

I was using the aws-otel-collector docker example https://github.com/aws-observability/aws-otel-collector/tree/main/examples/docker, but with the retain initial value flag set in the config, and additionally with a sample app exporting prometheus metrics.

Expected Result

We expect the log-group to contain an entry for the initial metric value.

Actual Result

Initial metric value was not exported

Collector version

v0.76.3

Environment information

Environment

OS: (e.g., "Ubuntu 20.04")
Compiler(if manually compiled): (e.g., "go 14.2")

OpenTelemetry Collector configuration

receivers:
  prometheus:
    config:
      global:
        scrape_interval: 1m
        scrape_timeout: 10s
      scrape_configs:
        - job_name: 'prometheus-roseo'
          metrics_path: /metrics
          static_configs:
            - targets: [ ot-sample-app:9128 ]
exporters:
  logging:
    loglevel: debug
  awsemf:
    region: 'eu-west-2'
    namespace: roseo
    log_group_name: /aws/ecs/roseo
    log_stream_name: prometheus
    retain_initial_value_of_delta_metric: true
    dimension_rollup_option: "NoDimensionRollup"
processors:
  batch:
extensions:
  health_check:
service:
  pipelines:
    metrics:
      receivers: [prometheus]
      exporters: [awsemf]
  telemetry:
    logs:
      level: debug

Log output

Did some testing with a modified awsemf exporter to add some log messages (roseo1@59ce7ff)

(With a sample application just incrementing a counter)

From this saw it logged that the initial value was not retained, and that the value of retainInitialValueForDelta on the data point slice did not reflect that it had been enabled in the config ("RetainInitial": "false" in the last log line).

2023/05/12 16:42:44 ADOT Collector version: v0.28.0
2023/05/12 16:42:44 found no extra config, skip it, err: open /opt/aws/aws-otel-collector/etc/extracfg.txt: no such file or directory
2023/05/12 16:42:44 attn: users of the prometheus receiver, prometheus exporter or prometheusremotewrite exporter please refer to https://github.com/aws-observability/aws-otel-collector/issues/2043 in regards to an ADOT Collector v0.31.0 breaking change
2023-05-12T16:42:44.699Z  info    service/telemetry.go:92                               Setting up own telemetry...
2023-05-12T16:42:44.699Z  info    service/telemetry.go:149                              Serving Prometheus metrics                  {"address": ":8888", "level": "Basic"}
2023-05-12T16:42:44.699Z  debug   exporter@v0.76.1/exporter.go:284                      Beta component. May change in the future.   {"kind": "exporter", "data_type": "metrics", "name": "awsemf"}
2023-05-12T16:42:44.700Z  debug   awsutil@v0.76.3/conn.go:60                            Using proxy address:                        {"kind": "exporter", "data_type": "metrics", "name": "awsemf", "proxyAddr": ""}
2023-05-12T16:42:44.700Z  debug   awsutil@v0.76.3/conn.go:137                           Fetch region from commandline/config file   {"kind": "exporter", "data_type": "metrics", "name": "awsemf", "region": "eu-west-2"}
2023-05-12T16:42:44.700Z  debug   receiver@v0.76.1/receiver.go:305                      Beta component. May change in the future.   {"kind": "receiver", "name": "prometheus", "data_type": "metrics"}
2023-05-12T16:42:44.700Z  info    service/service.go:129                                Starting aws-otel-collector...              {"Version": "v0.28.0", "NumCPU": 4}
2023-05-12T16:42:44.700Z  info    extensions/extensions.go:41                           Starting extensions...
2023-05-12T16:42:44.700Z  info    prometheusreceiver@v0.76.3/metrics_receiver.go:243    Scrape job added                            {"kind": "receiver", "name": "prometheus", "data_type": "metrics", "jobName": "prometheus-roseo"}
2023-05-12T16:42:44.700Z  debug   discovery/manager.go:289                              Starting provider                           {"kind": "receiver", "name": "prometheus", "data_type": "metrics", "provider": "static/0", "subs": "map[prometheus-roseo:{}]"}
2023-05-12T16:42:44.700Z  info    service/service.go:146                                Everything is ready. Begin running and processing data.
2023-05-12T16:42:44.700Z  info    prometheusreceiver@v0.76.3/metrics_receiver.go:255    Starting discovery manager                  {"kind": "receiver", "name": "prometheus", "data_type": "metrics"}
2023-05-12T16:42:44.700Z  info    prometheusreceiver@v0.76.3/metrics_receiver.go:289    Starting scrape manager                     {"kind": "receiver", "name": "prometheus", "data_type": "metrics"}
2023-05-12T16:42:44.701Z  debug   discovery/manager.go:323                              Discoverer channel closed                   {"kind": "receiver", "name": "prometheus", "data_type": "metrics", "provider": "static/0"}
2023-05-12T16:44:07.330Z  info    awsemfexporter@v0.76.3/emf_exporter.go:101            Start processing resource metrics           {"kind": "exporter", "data_type": "metrics", "name": "awsemf", "labels": {"http.scheme":"http","net.host.name":"ot-sample-app","net.host.port":"9128","service.instance.id":"ot-sample-app:9128","service.name":"prometheus-roseo"}}
2023-05-12T16:44:07.330Z  warn    awsemfexporter@v0.76.3/datapoint.go:142               Did not retain initial                      {"kind": "exporter", "data_type": "metrics", "name": "awsemf", "Metric": "http_server_requests_total", "Retained": "false", "RetainInitial": "false"}

Additional context

This relates to #16218

[github-actions]

Pinging code owners:

• exporter/awsemf: @Aneurysm9 @shaochengwang @mxiamxia

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[roseo1]

PR #21899 went stale but PR #24051 is open

[bryan-aguilar]

Closed with #24051