How to integrate Karma with Thanos?

[marcdavoli]

Hello,

I'm trying to figure out how to see alerts coming from a central Thanos Query instance in Karma.
I've seen a few people mention they use Thanos, but the details of how it works are unclear.

Here is a summary of my current setup:

• Karma deployed in a Monitoring Cluster
• Thanos components deployed in the same Monitoring Cluster
• I point Karma's config file to thanos-query-frontend like this:

alertmanager:
  interval: 60s
  servers:
    - cluster: local
      name: thanos
      uri: http://thanos-query-frontend.thanos:9090
      timeout: 20s
      proxy: true

• When Karma starts up, I see these logs:

karma-7d6d6fb9c5-fkkqb karma level=info msg="Collecting alerts and silences" alertmanager=thanos
karma-7d6d6fb9c5-fkkqb karma level=info msg="GET request" timeout=20 uri=http://thanos-query-frontend.thanos:9090/metrics
karma-7d6d6fb9c5-fkkqb karma level=error msg="Error while discovering version" error="alertmanager_build_info{version=...} metric is not exported from alertmanger" alertmanager=thanos
karma-7d6d6fb9c5-fkkqb karma level=debug msg="Probed alertmanager version" alertmanager=thanos version=
karma-7d6d6fb9c5-fkkqb karma level=error msg="Collection failed" error="[GET /status] getStatus (status 404): {}" alertmanager=thanos try=1/2
...
karma-7d6d6fb9c5-fkkqb karma level=info msg="Collection completed"

That metric alertmanager_build_info is of course not available at http://thanos-query-frontend.thanos:9090/metrics, Thanos has its own version of that thanos_build_version, but of course, Karma doesn't care about that.

Side note: I have a Grafana instance that uses that same Thanos Query instance

and I see all the alerts that bubble up to Thanos Query in the Grafana UI

Any guidance on how to integrate Karma with Thanos would be appreciated.
I mostly need help understanding which component Karma needs to talk to and how.
e.g.: Do I need to deploy another Prometheus in between Thanos and Karma?

Thanks!

[marcdavoli]

I dug into this a little bit and in retrospect, it seems obvious that this won't work as-is because Karma expects an Alertmanager, which I've since deployed.

Now I'm trying to figure out the Thanos side of things and from my research, it seems I may need the Thanos Ruler component and/or may need to make a major architecture change to allow alerts to bubble up to the Alertmanager.

I'm still working on this and would appreciate guidance if anyone has ever integrated Karma with Thanos.