Support Prometheus PodMonitor Deployment

[ordukhanian]

Hello,
Could you please support Prometheus PodMonitor configuration and deployment for Spark Operator?
Currently the Spark GCP Operator exposes metrics endpoint, but there is no ability to configure Prometheus integration.

apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  name: spark-operator-gcp-monitor
  labels:
    <pod monitor labels>
spec:
  podMetricsEndpoints:
    - interval: 5s
      port: metrics
      path: /metrics
      scheme: http
  jobLabel: spark-operator-gcp-monitor
  namespaceSelector:
    matchNames:
      - spark-operator-gcp
  selector:
    matchLabels:
      app.kubernetes.io/name: sparkoperator

[hagaibarel]

Hi,
prometheus-operator integration is usually done using a ServiceMonitor, any specific reason to use a PodMonitor instead?

[ordukhanian]

GCP Spark Operator does not deploy a Service when Prometheus metrics are enabled. That's why we decided to use a PodMonitor.

[nicholas-fwang]

hello
I'm also trying to apply pod monitor to spark application.
it's PR for adding container port naming of jmx exporter for pod monitor, but it's not merged currently.
#1099

and I'm developing pod monitor integration with spark operator
#1101

[ordukhanian]

@fisache , thank you for the response.
Could you please clarify, you're adding PodMonitor for a Spark application(driver and executor pods) or for the operator's pod?

[anikin-aa]

Hi!

GCP Spark Operator does not deploy a Service when Prometheus metrics are enabled

May be we should think of creation of such service ?

it's PR for adding container port naming of jmx exporter for pod monitor, but it's not merged currently.

It will be nice to optionally submit pod(service)monitor for spark operator during deployment via helm chart.

[nicholas-fwang]

@fisache , thank you for the response.
Could you please clarify, you're adding PodMonitor for a Spark application(driver and executor pods) or for the operator's pod?

for a Spark application(driver and executor pods)

[ordukhanian]

@fisache , thank you for the response.
Could you please clarify, you're adding PodMonitor for a Spark application(driver and executor pods) or for the operator's pod?

for a Spark application(driver and executor pods)

Right, but our suggestion is to support pod/service monitor configuration (and optional submission via helm chart) for the Spark operator itself, not for the applications.

[anikin-aa]

@liyinan926 , do you see some value in adding pod(service) monitor for operator pod ? If so, we can open a PR with changes.

[nicholas-fwang]

@fisache , thank you for the response.
Could you please clarify, you're adding PodMonitor for a Spark application(driver and executor pods) or for the operator's pod?

for a Spark application(driver and executor pods)

Right, but our suggestion is to support pod/service monitor configuration (and optional submission via helm chart) for the Spark operator itself, not for the applications.

I was confused. The operator's monitoring using prometheus-operator seems to be essential :)