Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add README section on configuring Prometheus to query multiple iDRACs #1

Merged
merged 2 commits into from
Feb 21, 2022

Conversation

dstndstn
Copy link
Contributor

It wasn't obvious to me how to get Prometheus to query a whole list of iDRACs. This uses the trick shown in https://prometheus.io/docs/guides/multi-target-exporter/
to achieve that.

I would also love to add some documentation on how to show in Grafana the idrac_sel_entry lines, but I couldn't figure it out!

@mrlhansen
Copy link
Owner

Hi @dstndstn

Thanks for the addition, I will merge it now :) Regarding Grafana you need to show it as a table. I tried making an example (might have a few labels specific to my setup). Below is the JSON output for the panel, hopefully that can be useful for you.

You probably want to remove the hostname label in the query and you need to set the correct data source.

{
  "type": "table",
  "title": "System Event Log",
  "gridPos": {
    "x": 5,
    "y": 0,
    "w": 15,
    "h": 15
  },
  "id": 2,
  "targets": [
    {
      "expr": "sort_desc(idrac_sel_entry{hostname=\"nodeaa-10\"}*1000)",
      "legendFormat": "",
      "interval": "",
      "refId": "A",
      "instant": true,
      "format": "table"
    }
  ],
  "options": {
    "showHeader": true,
    "sortBy": []
  },
  "datasource": "Infrastructure",
  "fieldConfig": {
    "defaults": {
      "custom": {
        "align": null,
        "filterable": false
      },
      "thresholds": {
        "mode": "absolute",
        "steps": [
          {
            "value": null,
            "color": "green"
          },
          {
            "value": 80,
            "color": "red"
          }
        ]
      },
      "mappings": []
    },
    "overrides": [
      {
        "matcher": {
          "id": "byName",
          "options": "Value"
        },
        "properties": [
          {
            "id": "unit",
            "value": "dateTimeAsIso"
          }
        ]
      },
      {
        "matcher": {
          "id": "byName",
          "options": "severity"
        },
        "properties": [
          {
            "id": "custom.width",
            "value": 129
          }
        ]
      },
      {
        "matcher": {
          "id": "byName",
          "options": "component"
        },
        "properties": [
          {
            "id": "custom.width",
            "value": 273
          }
        ]
      },
      {
        "matcher": {
          "id": "byName",
          "options": "message"
        },
        "properties": [
          {
            "id": "custom.width",
            "value": 915
          }
        ]
      }
    ]
  },
  "pluginVersion": "7.2.0",
  "timeFrom": null,
  "timeShift": null,
  "transformations": [
    {
      "id": "organize",
      "options": {
        "excludeByName": {
          "__name__": true,
          "Time": true,
          "id": true,
          "instance": true,
          "job": true,
          "hostname": true
        },
        "indexByName": {
          "Time": 0,
          "__name__": 1,
          "hostname": 2,
          "id": 3,
          "instance": 4,
          "job": 5,
          "severity": 6,
          "component": 7,
          "message": 8,
          "Value": 9
        },
        "renameByName": {}
      }
    }
  ]
}

@mrlhansen mrlhansen merged commit 212d09e into mrlhansen:master Feb 21, 2022
@dstndstn
Copy link
Contributor Author

Thanks for the tip (and this module -- it's great!)! It looks like your example panel here fetches the logs for a single node -- do you know of a way to, say, get the most recent log message for each node? I know this is a generic Prometheus query question, and I'm very new at that (and finding it very unintuitive!), so don't feel obliged to answer.

@mrlhansen
Copy link
Owner

Say that you want the Top 5 most recent entries for each host, you could do something like this (Prometheus query):

topk(5, idrac_sel_entry) by (hostname)

Here hostname is a unique label for each iDRAC instance. This is a label I have in my setup, you can use something else, such as instance.

The other option is to define a variable in your Grafana dashboard that selects a host to show the entries for, like it's done in the node_exporter dashboard, if you are familiar with that.

@dstndstn
Copy link
Contributor Author

Thank you so much for taking the time to answer!

The other thing I was missing was, in the Grafana panel settings, under the Prometheus query, you have to set "Format" to "Table" (not "Time series"), otherwise it only shows the timestamps.

Again, many thanks!

@mrlhansen
Copy link
Owner

You're welcome :) Happy to help!

@dstndstn
Copy link
Contributor Author

And here's the result! (You can see that our data center is a bit on the hot side...)

1

@dstndstn dstndstn deleted the readme-prom branch February 24, 2022 14:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants