[BUG] Using Prometheus datasource causes internal server error

[StefanSchuhart]

Describe the bug

Trying to get metrics from prometheus datasource via app/observability-metrics#/ shows nothing and Browser developer tools listing a XHR with: POST "api/ppl/search?dataSourceMDSId=" and 500 Internal Server Error

Related component

Plugins

To Reproduce

1. Go to Management > Dashboards Management > Data sources
2. Click on 'Create direct query connection'
3. Add Prometheus Data source with URI: http://localhost:9090 OR
   create one via dev tools:

POST _plugins/_query/_datasources 
{
    "name" : "my_prometheus",
    "connector": "prometheus",
    "properties" : {
        "prometheus.uri" : "http://localhost:9090"
    }
}

5. Click on: query data
6. No metrics available

Prometheus is reachable via http://localhost:9090.

Expected behavior

Available metrics are listed like shown here: https://opensearch.org/docs/latest/observing-your-data/prometheusmetrics/, no "Internal Server error" XHR requests show up inside developer tools.

Additional Details

Plugins
SQL Plugin

Log messages from Opensearch-dashboards

{"type":"response","@timestamp":"2024-11-26T15:57:26Z","tags":[],"pid":1,"method":"post","statusCode":500,"req":{"url":"/api/ppl/search?dataSourceMDSId=","method":"post","headers":{"host":"XXX","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36","content-length":"72","accept":"*/*","accept-encoding":"gzip, deflate, br, zstd","accept-language":"de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7","cache-control":"no-cache","content-type":"application/json","origin":"https://XXX","osd-version":"2.18.0","osd-xsrf":"osd-fetch","pragma":"no-cache","referer":"https://XXX/app/observability-metrics","sec-ch-ua":"\"Not_A Brand\";v=\"8\", \"Chromium\";v=\"120\", \"Google Chrome\";v=\"120\"","sec-ch-ua-mobile":"?0","sec-ch-ua-platform":"\"Windows\"","sec-fetch-dest":"empty","sec-fetch-mode":"cors","sec-fetch-site":"same-origin","x-forwarded-for":"141.91.18.142","x-forwarded-host":"XXX","x-forwarded-port":"443","x-forwarded-proto":"https","x-forwarded-server":"monitoring-dependencies-traefik-9887cbfbc-wdg6m","x-real-ip":"XXX","securitytenant":"admin_tenant"},"remoteAddress":"10.0.6.37","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36","referer":"https://XXX/app/observability-metrics"},"res":{"statusCode":500,"responseTime":302,"contentLength":9},"message":"POST /api/ppl/search?dataSourceMDSId= 500 302ms - 9.0B"}
  response: '{\n' +
    '  "error": {\n' +
    '    "reason": "There was internal problem at backend",\n' +
    '    "details": "Error while fetching metric list for from prometheus: Failed to connect to localhost/[0:0:0:0:0:0:0:1]:9090",\n' +
    '    "type": "RuntimeException"\n' +
    '  },\n' +
    '  "status": 500\n' +
    '}',
  toString: [Function (anonymous)],
  toJSON: [Function (anonymous)]
}
PPL query fetch err:  StatusCodeError: Internal Server Error
    at respond (/usr/share/opensearch-dashboards/node_modules/elasticsearch/src/lib/transport.js:349:15)
    at checkRespForFailure (/usr/share/opensearch-dashboards/node_modules/elasticsearch/src/lib/transport.js:306:7)
    at HttpConnector.<anonymous> (/usr/share/opensearch-dashboards/node_modules/elasticsearch/src/lib/connectors/http.js:173:7)
    at IncomingMessage.wrapper (/usr/share/opensearch-dashboards/node_modules/lodash/lodash.js:4991:19)
    at IncomingMessage.emit (node:events:529:35)
    at IncomingMessage.emit (node:domain:489:12)
    at endReadableNT (node:internal/streams/readable:1400:12)
    at processTicksAndRejections (node:internal/process/task_queues:82:21) {
  status: 500,
  displayName: 'InternalServerError',
  path: '/_plugins/_ppl',
  query: { format: 'viz' },
  body: '{\n' +
    '  "error": {\n' +
    '    "reason": "There was internal problem at backend",\n' +
    '    "details": "Error while fetching metric list for from prometheus: Failed to connect to localhost/[0:0:0:0:0:0:0:1]:9090",\n' +
    '    "type": "RuntimeException"\n' +
    '  },\n' +
    '  "status": 500\n' +
    '}',
  statusCode: 500,
  response: '{\n' +
    '  "error": {\n' +
    '    "reason": "There was internal problem at backend",\n' +
    '    "details": "Error while fetching metric list for from prometheus: Failed to connect to localhost/[0:0:0:0:0:0:0:1]:9090",\n' +
    '    "type": "RuntimeException"\n' +
    '  },\n' +
    '  "status": 500\n' +
    '}',
  toString: [Function (anonymous)],
  toJSON: [Function (anonymous)]
}
{"type":"error","@timestamp":"2024-11-26T15:57:26Z","tags":[],"pid":1,"level":"error","error":{"message":"Internal Server Error","name":"Error","stack":"Error: Internal Server Error\n    at HapiResponseAdapter.toError (/usr/share/opensearch-dashboards/src/core/server/http/router/response_adapter.js:127:19)\n    at HapiResponseAdapter.toHapiResponse (/usr/share/opensearch-dashboards/src/core/server/http/router/response_adapter.js:83:19)\n    at HapiResponseAdapter.handle (/usr/share/opensearch-dashboards/src/core/server/http/router/response_adapter.js:79:17)\n    at Router.handle (/usr/share/opensearch-dashboards/src/core/server/http/router/router.js:175:34)\n    at processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at handler (/usr/share/opensearch-dashboards/src/core/server/http/router/router.js:140:50)\n    at exports.Manager.execute (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/toolkit.js:60:28)\n    at Object.internals.handler (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/handler.js:46:20)\n    at exports.execute (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/handler.js:31:20)\n    at Request._lifecycle (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/request.js:371:32)\n    at Request._execute (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/request.js:281:9)"},"url":"http://XXX/api/ppl/search?dataSourceMDSId=","message":"Internal Server Error"}

[dblock]

[Catch All Triage - 1, 2, 3]

[Swiddis]

@vmmusings can you take a look at this? Related to #3197 that you're also working on (no Prometheus metrics list available)