Allow filtering via query parameters

[fabxc]

For federation we can provide match[] query parameters with metric selectors to ask for specific metrics. By default none are shown.

Regular scrape targets should provide the same options but show all metrics by default.

[beorn7]

This has to be sync'd with other client libraries, obviously.
Perhaps a topic for the roadmap meeting at PromCon.

[fabxc]

Seems very hard to get adopted. Per-exporter parameters to enable certain
collectors seems more useful an independent in general.

So far number of metrics in collection didn't seem to be a bottleneck, no?

On Sat, Aug 20, 2016, 12:03 AM Björn Rabenstein notifications@github.com
wrote:

This has to be sync'd with other client libraries, obviously.
Perhaps a topic for the roadmap meeting at PromCon.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
#135 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AEuA8tUQfzNMnbdSnpKgAxF_eAMgEAJpks5qhigcgaJpZM4E_4JK
.

[grobie]

Using flags to control active collectors will probably stay the way to go for one-purpose exporters. Where this feature will become interesting is when someone wants to scrape different metrics in different scrape intervals and the other collectors incur a substantial overhead. The MySQL exporter would be a good example for that.

[brian-brazil]

Yeah, I want to be able to do that and I know @SuperQ wants it too.

The main question for me is whether these parameters would be the names of collectors or timeseries.

[SuperQ]

I think the easiest thing to do is simply pass URL params as key/value pairs to the Collector interface.

[brian-brazil]

If we pass URL parameters to collectors directly, there will be little standardisation.

My thinking is more that it'd work via Descs, so that only collectors that match the given timeseries names get called.

[beorn7]

The implementation details will be fairly easy to figure out once we have settled on the HTTP API (which should be the same across client libraries). Since the choice there is not obvious, we need to find a common way first.

[SuperQ]

We already lack standards for URL params. snmp_exporter uses address and blackbox_exporter uses target. I think we need the flexibility to allow for arbitrary collector params, and things should be "standardized" via community best practices docs, like we do for metric names and labels.

[fabxc]

I agree with that. Whatever I'm monitoring, I'll have to look up valid values for whatever parameters anyway – might just as well use parameters given in the docs. "Collectors" are no first-class concept in Prometheus and shooting for more than best practices and guidance doesn't seem like well-invested effort.

Selecting by time series is another approach. But at the instrumentation level efforts are rather separated along collectors and inferring these from time series descs seems rather fragile as metrics can be added and dropped.
That's what I was referring to in my last comment: selecting only certain time series seems to optimize a problem currently addressed by metric_relabel_configs. I'm not sure it's an actual problem at this point. It only indirectly addresses the issue of some collectors being expensive.

[SuperQ]

Selecting based on what amounts to collector-side metric_relabel_configs is problematic when dealing with pass-through Const metrics, like node_exporter or mysqld_exporter. I need to have params such that I can toggle specific features on and off depending on my scrape need. Even being able to adjust some params of the collector at scrape time would be useful.

[brian-brazil]

I think standard should go beyond that. Saying things are already slightly non-standard for a similar use case so therefore we can't have enforced standards would prevent us from ever having hard standards.

I think we should start from how this should work for direct instrumentation (as that's what most instrumentation will be), and then extend that to custom collectors. And then extend that to custom collectors that don't know what time series they'll be exporting, as they're rare and we shouldn't have the tail wagging the dog.

I'm against arbitrary data being passed to custom collectors, there's too much scope for it being abused.

I'm not sure it's an actual problem at this point.

It's an actual problem. We've had several users looking for it.

[brian-brazil]

I just had a thought while pondering the WMI exporter.

What if we made Desc automatic?

I see three classes of collector:

1. Desc is known statically (e.g. direct instrumentation, disk stats in node exporter)
2. Desc depends on the system, but is basically static at runtime (e.g. mysql exporter, netstat in node exporter)
3. Desc is dynamic (e.g. jmx exporter)

For the first two classes, why not call Collect at registration time and extract the Desc that way?

For the 3rd case, we're unlikely to ever get query filtering working in a generic fashion and these collectors may be expensive so we'd need to make this opt-out/opt-in/overridable.

There'd also be cases where Collect is expensive or wasteful (e.g. an ephemeral registry for using the pushgateway). We could handle this via the Collector opt-out/opt-in or have it as a per-registry setting that's enabled on the default registry.

What all this gives us is the information we need inside the registry to call exactly the collectors we require, and then filter down to just the timeseres the user wants. It also gives us a simpler API for users to write custom collectors as in most cases they never need to worry about Descs.

[beorn7]

I have been pondering the idea of automatic "description by collection" for quite some time (since December 2013 or so). Sometimes, it will work great, but sometimes it would defeat the whole idea of the Desc. That's the case for any MetricVec pre-initialization of any values. (Yes, we encourage to initialize all the children beforehand, but that's not always possible.) Such a MetricVec might collect nothing, although the Desc is perfectly known.

What I came up with is a helper function DescribeByCollect that would make it easy to implement Describe by calling Collect and reap the Descs from the results. But it has to be opt-in.

All of this is, however, not directly related to the issue at hand here. Unfortunately, we didn't get to talk about it during the Saturday symposium. I could imagine both ways, either specify the names of the metrics to retrieve in some pattern matching way (globs or regexp — full PromQL matchers would be nice but too much to ask from client libraries) or pass through some opaque URL parameters (which would show up at some point in the API of the client libraries; those would be quite specific to each exporter etc.).

Are there any use cases that could not be solved with the former approach?

[brian-brazil]

We just need something that always works for direct instrumentation, and covers the majority of custom collector use cases out of the box. For example with the standard use of Python style ConstMetrics you'll still get an empty metric when you call collect on something that has no label values yet.

I think we should do this by passing in time series names/regexes. I think labels would be a bit much in practice.

All of this is, however, not directly related to the issue at hand here.

It's a key point, as without collector information we can't know which collectors to use.

Are there any use cases that could not be solved with the former approach?

Yes, and I believe we shouldn't support them. Anything that would require opaque information should be handled by the exporter itself, with no assistance from the client library. Users implementing snmp-exporter like mechanisms via parameters meant for filtering collectors doesn't seem like it would lead to a sane ecosystem.

[beorn7]

I think we should do this by passing in time series names/regexes. I think labels would be a bit much in practice.

Yes, that's what I would go for. Just fishing for contrary opinions before investing any effort in implementations.

All of this is, however, not directly related to the issue at hand here.

It's a key point, as without collector information we can't know which collectors to use.

But that's already solved. You were just talking about how to make it more convenient to implement.