Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

AWS metric EMF exporter fails to report http.route for Cloudwatch metrics #2135

Closed
Ryang20718 opened this issue Jun 12, 2023 · 4 comments
Closed
Labels

Comments

@Ryang20718
Copy link

Describe the bug
AWS metric EMF exporter fails to report http.route for Cloudwatch metrics when otelhttp is used as middleware

Steps to reproduce
Spin up the otel collector via docker,

instrument some code with otelhttp

import(
	"go.opentelemetry.io/contrib/instrumentation/github.com/gin-gonic/gin/otelgin"
   "go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp"
)


       // tracing setup here for otel
       
       // tracing middleware
       
	router := gin.New()
   router.Use(otelgin.Middleware(config.OtelServiceName))

	mux := http.NewServeMux()
	mux.Handle("/api/", router)

	otelHTTPHandler := otelhttp.NewHandler(mux, "")

       httpServer := &http.Server{
           ReadHeaderTimeout: 1 * time.Minute,
           Addr:              ":8080",
           Handler:           otelHTTPHandler,
       }
       return httpServer.ListenAndServe()

What did you expect to see?
http.route as part of the metrics getting sent

What did you see instead?
no http.route was getting sent

{
    "OTelLib": "go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp",
    "Version": "1",
    "_aws": {
        "CloudWatchMetrics": [
            {
                "Namespace": "testing",
                "Dimensions": [
                    [
                        "OTelLib",
                        "http.scheme",
                        "http.status_code",
                        "net.sock.peer.port",
                        "net.host.name",
                        "net.host.port",
                        "net.sock.peer.addr",
                        "http.flavor",
                        "http.method",
                        "http.user_agent"
                    ],
                    [
                        "OTelLib"
                    ],
                    [
                        "OTelLib",
                        "http.status_code"
                    ],
                    [
                        "OTelLib",
                        "net.sock.peer.port"
                    ],
                    [
                        "OTelLib",
                        "http.scheme"
                    ],
                    [
                        "OTelLib",
                        "http.method"
                    ],
                    [
                        "OTelLib",
                        "http.user_agent"
                    ],
                    [
                        "OTelLib",
                        "net.host.name"
                    ],
                    [
                        "OTelLib",
                        "net.host.port"
                    ],
                    [
                        "OTelLib",
                        "net.sock.peer.addr"
                    ],
                    [
                        "OTelLib",
                        "http.flavor"
                    ]
                ],
                "Metrics": [
                    {
                        "Name": "http.server.duration"
                    }
                ]
            }
        ],
        "Timestamp": 1686603548366
    },
    "http.flavor": "1.1",
    "http.method": "GET",
    "http.scheme": "http",
    "http.server.duration": {
        "Max": 1.26,
        "Min": 0.14,
        "Count": 3,
        "Sum": 2.43
    },
    "http.status_code": "304",
    "http.user_agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36",
    "net.host.name": "localhost",
    "net.host.port": "8080",
    "net.sock.peer.addr": "172.19.0.1",
    "net.sock.peer.port": "55952"
}

Environment
Collector config file

extensions:
  health_check:
  pprof:
    endpoint: 0.0.0.0:1777

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317

processors:
  batch:

exporters:
  logging:
    loglevel: debug
  awsemf:
    region: 'us-west-2'

service:
  pipelines:
    metrics:
      receivers: [otlp]
      exporters: [awsemf]

  extensions: [pprof]
  telemetry:
    logs:
      level: debug

Additional context
Exact command used to spin up collector was using the example in this repo

     docker run --rm -p 4317:4317 -p 55680:55680 -p 8889:8888 \
       -e AWS_REGION=us-west-2 \
       -e AWS_PROFILE=default \
       -v ~/.aws:/root/.aws \
       -v "${PWD}/examples/docker/config-test.yaml":/otel-local-config.yaml \
       --name awscollector public.ecr.aws/aws-observability/aws-otel-collector:latest \
       --config otel-local-config.yaml;
    ```
@Ryang20718
Copy link
Author

@rapphil
Copy link
Member

rapphil commented Jun 22, 2023

The issue is that http.route is not being populated by the instrumentation: https://github.com/open-telemetry/opentelemetry-go-contrib/blob/main/instrumentation/net/http/otelhttp/internal/semconvutil/httpconv.go#L269

It is not being populated because net/http does not have the notion of route. It only deals with the path, which has high cardinality.

Please see note [1] about http.route in the semantic conventions: https://github.com/open-telemetry/semantic-conventions/blob/main/specification/metrics/semantic_conventions/http-metrics.md?plain=1#L75

Related ticket: open-telemetry/opentelemetry-go-contrib#3743

@github-actions
Copy link
Contributor

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 30 days.

@github-actions github-actions bot added the stale label Aug 27, 2023
@github-actions
Copy link
Contributor

github-actions bot commented Oct 1, 2023

This issue was closed because it has been marked as stale for 30 days with no activity.

@github-actions github-actions bot closed this as not planned Won't fix, can't repro, duplicate, stale Oct 1, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants