Prometheus Metrics export failed

[banandh]

Component(s)

No response

Describe the issue you're reporting

I used https://github.com/Bhoopesh123/OpenTelemetry for setting up Open telemetry on my EKS. Only prometheus metrics are failing. Im able to check the traces in jaeger.

info MetricsExporter {"kind": "exporter", "data_type": "metrics", "name": "debug", "resource metrics": 20, "metrics": 40, "data points": 180}
2024-05-22T11:23:55.757Z error exporterhelper/common.go:95 Exporting failed. Dropping data. {"kind": "exporter", "data_type": "metrics", "name": "otlphttp/prometheus", "error": "not retryable error: Permanent error: error exporting items, request to http://opentelemetry-demo-prometheus-server:9090/api/v1/otlp/v1/metrics responded with HTTP Status Code 404", "dropped_items": 180}
go.opentelemetry.io/collector/exporter/otlphttpexporter.createMetricsExporter.WithQueue.func5.1

Prometheus server is running
banandh@MCHICL652W60 opentelemetry-app % k logs opentelemetry-demo-prometheus-server-59bd64f76-w8tg6 -n otel-demo
ts=2024-05-21T05:33:37.140Z caller=main.go:178 level=info msg="Experimental in-memory exemplar storage enabled"
ts=2024-05-21T05:33:37.779Z caller=main.go:585 level=info msg="Starting Prometheus Server" mode=server version="(version=2.46.0, branch=HEAD, revision=cbb69e51423565ec40f46e74f4ff2dbb3b7fb4f0)"
ts=2024-05-21T05:33:37.794Z caller=main.go:590 level=info build_context="(go=go1.20.6, platform=linux/arm64, user=root@42454fc0f41e, date=20230725-12:37:33, tags=netgo,builtinassets,stringlabels)"
ts=2024-05-21T05:33:37.798Z caller=main.go:591 level=info host_details="(Linux 6.6.14-0-virt Bhoopesh123/OpenTelemetry#1 SMP Fri, 26 Jan 2024 11:08:07 +0000 aarch64 opentelemetry-demo-prometheus-server-59bd64f76-w8tg6 (none))"
ts=2024-05-21T05:33:37.798Z caller=main.go:592 level=info fd_limits="(soft=1048576, hard=1048576)"
ts=2024-05-21T05:33:37.798Z caller=main.go:593 level=info vm_limits="(soft=unlimited, hard=unlimited)"
ts=2024-05-21T05:33:37.934Z caller=web.go:563 level=info component=web msg="Start listening for connections" address=0.0.0.0:9090
ts=2024-05-21T05:33:37.982Z caller=main.go:1026 level=info msg="Starting TSDB ..."

Im able to access prometheus in localhost:9090 after port-forward. Pls help

[JaredTan95]

you need move this issue into https://github.com/Bhoopesh123/OpenTelemetry repo first, It looks like there is a configuration error. The address of the otlp exportaer was incorrectly configured for prometheus; however, prometheus is not able to receive otlp metrics directly. https://prometheus.io/docs/prometheus/latest/storage/#remote-storage-integrations

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.