docs: add prometheus + grafana deployment guide

[EyalPazz]

/fixes #747

[netlify]

✅ Deploy Preview for gateway-api-inference-extension ready!

Name	Link
🔨 Latest commit	5d60f5d
🔍 Latest deploy log	https://app.netlify.com/projects/gateway-api-inference-extension/deploys/6890e75fa982f200086dfb01
😎 Deploy Preview	https://deploy-preview-1019--gateway-api-inference-extension.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify project configuration.

[EyalPazz]

@nirrozenbaum @danehans

[nirrozenbaum]

after running this step, I'm running the command from the helm output to get prometheus pod:

Get the Prometheus server URL by running these commands in the same shell:
  export POD_NAME=$(kubectl get pods --namespace monitoring -l "app.kubernetes.io/name=prometheus,app.kubernetes.io/instance=prometheus" -o jsonpath="{.items[0].metadata.name}")
  kubectl --namespace monitoring port-forward $POD_NAME 9090

when running the export POD_NAME=... I get the following error:

error: error executing jsonpath "{.items[0].metadata.name}": Error executing template: array index out of bounds: index 0, length 0. Printing more information for debugging the template:
	template was:
		{.items[0].metadata.name}
	object given to jsonpath engine was:
		map[string]interface {}{"apiVersion":"v1", "items":[]interface {}{}, "kind":"List", "metadata":map[string]interface {}{"resourceVersion":""}}

no pod matches the command. maybe something is missing from values.yaml?

[EyalPazz]

This is relying on the merge to main, it's currently pointing to the main branch despite it's not there yet

You can currently test using:
https://raw.githubusercontent.com/eyalpazz/gateway-api-inference-extension/add-prometheus-guide/config/manifests/prometheus/values.yaml

[nirrozenbaum]

this is the error after I copied values.yaml locally and used it

[nirrozenbaum]

do you mind trying the instructions on a fresh kind cluster?

[EyalPazz]

Of course, i'll try to do it later tonight

[danehans]

@EyalPazz pls rebase.

[EyalPazz]

Rebased, sry for the delay

[EyalPazz]

@nirrozenbaum did it work for you after we talked in private?

[nirrozenbaum]

@nirrozenbaum did it work for you after we talked in private?

@EyalPazz sorry for the delayed response! we had some stressed milestone and I wasn't available.
unfortunately, this PR is not working for me (therefore cannot approve it as is).
I think it would be good if you document the env you're using, like version of the cluster, what type of cluster (k8s, kind, openshift, etc), in order to isolate the problem. it might help identifying the differences between your env and mine.

[EyalPazz]

Hi!! all good
I'm not currently able to start a fresh kind cluster as something with the crd's seems to be off (i wrote about it in the slack channel), i think i know what the problem is though, the Service Account, RoleBinding etc, they are all being created in the default namespace, i forgot to change / mention it in the guide. I hope i'll be able to retest it soon, but if you have the time to check if that's actually the problem i'd greatly appreciate it

@nirrozenbaum

[nirrozenbaum]

@EyalPazz are you able to work with release 0.5 branch? seems like main quickstart is broken

[EyalPazz]

0.5 branch was also kinda sketchy, but i fixed the problem
I forgot to mention the RBAC Resources at the top of the page should be applied at the monitoring namespace :(
I thought it would be better to add it to the config directory for this specific use case and not change it for everybody, should be good to go, sry for the delay @nirrozenbaum

[EyalPazz]

@nirrozenbaum the tests are failing because of kubectl-validate that is being run on the manifests directory, and the values.yaml doesn't pass it as it isn't a resource, and i didn't find an option to ignore this file elegantly, should i create a new directory instead of placing it in manifests? and if so, what should be the name of it

[kfswain]

I think the values.yaml file should be pulled out of the manifests dir since its not a k8s manifest. We can probably put this under its own directory within config, as far as name of the dir, we can do like observability or something like that. i dont think we need to overthink this one

[EyalPazz]

Should be good to go @nirrozenbaum

[danehans]

ClusterRoleBinding is a cluster-scoped resource, so remove the namespace field (to avoid confusion).

[EyalPazz]

btw in the current guide it has a namespace so i just went along with it, i can open another PR to fix it, in order to not "complicate" this one

[danehans]

btw in the current guide it has a namespace so i just went along with it, i can open another PR to fix it, in order to not "complicate" this one

@EyalPazz, that would be much appreciated.

[danehans]

Most of these resources are already covered in https://gateway-api-inference-extension.sigs.k8s.io/guides/metrics-and-observability/#scrape-metrics-pprof-profiles. To avoid confusion and misconfiguration, this file should only contain ServiceAccount and an updated ClusterRoleBinding that includes an additional subjects entry for the new ServiceAccount:

- kind: ServiceAccount
  name: inference-gateway-sa-metrics-reader
  namespace: monitoring

Maybe use kubectl to patch the ClusterRoleBinding with this entry, so this file only contains the ServiceAccount.

[danehans]

Simplify with a single command that port-forwards the grafana deployment:

kubectl -n monitoring port-forward deploy/grafana 3000

Since Grafana is not configured to use the default admin/admin login info, add a step to get the password from the secret. For example:

kubectl -n monitoring get secret grafana \
  -o go-template='{{ index .data "admin-password" | base64decode }}'

Add a note such as "You can now access the Grafana UI from http://127.0.0.1"

[danehans]

nit: s/Necessary/the necessary/ and s/Resources/resources/

or if you update rbac.yaml to only include the SA:

s/Create Necessary ServiceAccount and RBAC Resources:/Create the necessary ServiceAccount resource:/

and then include the kubectl patch command to patch the ClusterRoleBinding with a subject for the monitoring ServiceAccount.

[danehans]

@EyalPazz PTAL and my review comments above. Let me know if you have any questions.

[EyalPazz]

@danehans wdyt?

[danehans]

/lgtm
/approve

@EyalPazz, thanks for your patience in getting this PR merged.

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: danehans, EyalPazz

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [danehans]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment