Prometheus metrics are not collected #488
Comments
|
@yujinchoi-94 |
|
@Gallardot Here's my configuration for prometheus operator kube-prometheus-stack:
fullnameOverride: "kube-prometheus-stack"
grafana:
service:
type: NodePort
ingress:
enabled: true
annotations:
kubernetes.io/ingress.class: alb
alb.ingress.kubernetes.io/scheme: internal
alb.ingress.kubernetes.io/certificate-arn: ***
alb.ingress.kubernetes.io/security-groups: 'common-internal-alb-sg'
alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS": 443}]'
alb.ingress.kubernetes.io/healthcheck-path: /api/health
alb.ingress.kubernetes.io/success-codes: '200'
alb.ingress.kubernetes.io/backend-protocol: HTTP
hosts:
- ***
pathType: Prefix
path: /*
defaultDashboardsTimezone: Asia/Seoul
additionalDataSources:
- name: Thanos
type: prometheus
url: http://thanos-query-frontend.monitor:9090
uid: Thanos
- name: Thanos(auto-resolution)
type: prometheus
url: http://thanos-query-frontend.monitor:9090
uid: ThanosAutoResolution
jsonData:
customQueryParameters: max_source_resolution=auto
persistence:
enabled: true
type: pvc
size: 1Gi
storageClassName: efs
accessModes: ["ReadWriteOnce"]
plugins: []
kubeApiServer:
enabled: false
kubelet:
enabled: true
kubeControllerManager:
enabled: false
coreDns:
enabled: false
kubeEtcd:
enabled: false
kubeScheduler:
enabled: false
kubeProxy:
enabled: false
kubeStateMetrics:
enabled: true
nodeExporter:
enabled: false
prometheus:
serviceAccount:
create: true
name: "kube-prometheus-stack-prometheus"
annotations:
eks.amazonaws.com/role-arn: ***
thanosService:
enabled: true
thanosServiceMonitor:
enabled: true
prometheusSpec:
scrapeInterval: "1m"
scrapeTimeout: "5s"
configMaps:
- bucket
serviceMonitorSelector:
matchExpressions:
- {key: release, operator: In, values: [kube-prometheus-stack, dev-kube-prometheus-stack]}
shards: 3
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: spec
operator: In
values:
- 4xlarge
storageSpec:
volumeClaimTemplate:
spec:
storageClassName: efs
accessModes: ["ReadWriteOnce"]
resources:
requests:
storage: 50Gi
## Resource limits & requests
##
resources:
requests:
cpu: 8
memory: 8G
limits:
cpu: 8
memory: 8G
additionalScrapeConfigs:
###
thanos:
baseImage: quay.io/thanos/thanos
version: v0.30.1
objectStorageConfigFile: /etc/prometheus/configmaps/bucket/bucket.yaml
volumeMounts:
- name: configmap-bucket # prometheus.prometheusSpec.configMaps 설정에 의해 등록된 volume 참조
mountPath: /etc/prometheus/configmaps/bucketKubernetes Version
(For your information, I've installed apisix and prometheus using argocd.) |
|
@yujinchoi-94 Thanks. I also need the helm charts parameters for when you install apisix. I tried to reproduce it |
|
@Gallardot You can check below. (What I provided in the first comment) apisix:
gateway:
type: NodePort
ingress:
enabled: true
annotations:
kubernetes.io/ingress.class: alb
alb.ingress.kubernetes.io/scheme: internal
alb.ingress.kubernetes.io/certificate-arn: ***
alb.ingress.kubernetes.io/security-groups: 'common-internal-alb-sg'
alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS": 443}]'
alb.ingress.kubernetes.io/healthcheck-path: /
alb.ingress.kubernetes.io/success-codes: '404' # 404 Route Not Found
alb.ingress.kubernetes.io/backend-protocol: HTTP
hosts:
- host: ***
paths:
- /*
discovery:
enabled: true
registry:
eureka:
host:
- ***
- ***
prefix: "/eureka/"
fetch_interval: 30
weight: 100
timeout:
connect: 2000
send: 2000
read: 5000
# Observability configuration.
# ref: https://apisix.apache.org/docs/apisix/plugins/prometheus/
serviceMonitor:
enabled: true
labels:
release: kube-prometheus-stack
dashboard:
enabled: true
config:
conf:
etcd:
endpoints:
- dev-apisix-etcd:2379
service:
type: NodePort
port: 80
ingress:
enabled: true
annotations:
kubernetes.io/ingress.class: alb
alb.ingress.kubernetes.io/scheme: internal
alb.ingress.kubernetes.io/certificate-arn: ***
alb.ingress.kubernetes.io/security-groups: 'common-internal-alb-sg'
alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS": 443}]'
alb.ingress.kubernetes.io/healthcheck-path: /ping
alb.ingress.kubernetes.io/success-codes: '200'
alb.ingress.kubernetes.io/backend-protocol: HTTP
hosts:
- host: ***
paths:
- /*
ingress-controller:
enabled: true
config:
apisix:
serviceName: dev-apisix-admin
serviceNamespace: ingress-apisix
plugins:
- api-breaker
- prometheus
- forward-auth |
|
@yujinchoi-94 an example |
|
@Gallardot Thank you for responding so quickly :) I've installed them in different namespaces. |
|
@Gallardot Thank you for your help :) |
|
@yujinchoi-94 |
Hi, I'm trying to collect some metrics using apisix prometheus plugin.
I've found this PR and followed the document.
(I manually configured prometheus port based on this)
However, prometheus is not collecting any metrics from apisix service monitor.
I found that
kubernetes_sd_confighas been configured for apisix in Prometheus UI.Prometheus configuration (through Prometheus UI)
By the way, there was no target for apisix.
Moreover, in service discovery page, there were some discovered metrics but it was dropped.
FYI, Prometheus has deployed in another namespace so I set this option
serviceMonitorSelectorNilUsesHelmValuesto false.Here's what I've configured.
Apisix Helm
Gateway Service
ServiceMonitor
Prometheus
Thank you in advance :)
The text was updated successfully, but these errors were encountered: