Prometheus unreachable on docker network

[kevinmatthe]

NOTE

If this case is urgent, please subscribe to SUBNET so that our 24/7 support team may help you faster.

When deploying minio with docker compose, configure prometheus will always display "Prometheus unreachable"

Expected Behavior

Show prometheus metrics on monitoring panel.

Current Behavior

Showing "Prometheus unreachable"

Possible Solution

Currently no solution found.

Steps to Reproduce (for bugs)

Notice: the prometheus job_name == 'minio-job'
Start both service, and the error show up.

Try attaching to the minio container, use curl:

curl http://prom:9090/-/healthy

It works fine and prove that the network is definitely REACHABLE.

Context

Regression

Your Environment

Here's my example docker-compose.yaml:
minio:

services:
  minio:
    command: 'server /data --console-address ":9001"'
    image: minio/minio
    restart: always
    environment:
      - MINIO_ROOT_PASSWORD=Heyuheng1.22.3
      - MINIO_ROOT_USER=kevinmatt
      - MINIO_API_CORS_ALLOW_ORIGIN=*
      - MINIO_PROMETHEUS_AUTH_TYPE=public
      - MINIO_PROMETHEUS_URL="http://prom:9090"
      - MINIO_PROMETHEUS_JOB_ID=minio-job
      - MINIO_LOG_LEVEL=debug
    volumes:
      - '/mnt/StorageMirror/Storage/minio/data:/data'
    container_name: minio
    ports:
      - 29001:9001
      - 29000:9000
    expose:
      - 9001
      - 9000

networks:
  default:
    external: true
    name: homelab

prometheus:

version: '3.7'
services:
  prometheus:
    image: prom/prometheus:main
    container_name: prom
    restart: always
    user: root:root
    command:
      - "--config.file=/etc/prometheus/prometheus.yml"
      - "--storage.tsdb.path=/prometheus"
      - "--web.console.libraries=/usr/share/prometheus/console_libraries"
      - "--web.console.templates=/usr/share/prometheus/consoles"
      - "--storage.tsdb.retention.time=1y"
    volumes:
      - ./prometheus/data:/prometheus
      - ./prometheus/config:/etc/prometheus
    ports:
      - 9090:9090
networks:
  default:
    external: true
    name: homelab

• MinIO version used (minio --version): RELEASE.2024-05-10T01-41-38Z
• Server setup and configuration: as the compose file defined
• Operating System and version (uname -a): Linux truenas 6.6.20-production+truenas Profiling endpoints for mcs #1 SMP PREEMPT_DYNAMIC Tue Apr 23 01:22:22 UTC 2024 x86_64 GNU/Linux

[pjuarezd]

Can you check on the hostname of the prometheus container? I am not sure the hostname is reachable as you defined in the env variable

MINIO_PROMETHEUS_URL="http://prom:9090"

[allanrogerr]

@kevinmatthe
"http://prom:9090"/-/healthy is not reachable; http://prom:9090/-/healthy is.
Redefine MINIO_PROMETHEUS_URL=http://prom:9090 without the quotes and try again.

[pjuarezd]

Thank you @allanrogerr, if after redefine the env variable the problem persist please let us know @kevinmatthe