Attribute querying approach doesn't scale

[rhuss]

The approach to first query, read all attributes and finally apply the rule set on each attribute individually doesn't scale at all in the real world.

It is not uncommon that a Java server has hundreds MBean registered, each with five to ten attributes. Reading an attribute is normally cheap if only member variables are stored, but many MBean calculate the attribute value also on the fly so that an average readtime of 1-2ms is a good average for many MBeans. Considering this, a run only for reading all attributes can take e.g. 400 MBeans * 8  attributes * 2ms = 6.4 seconds

On each of this 3200 attributes the whole rule set is applied individually. Assuming that people tend to generate quite generic and large rulesets so that they can have a uniform agent installation everywhere, this multiples to the number above.

Considering a standard scrape interval of 15s, I think its fair to say that performing monitoring that way is intrusive.

I've prepared a demo project to demonstrate this. It is only about reading attributes, not applying rule sets. The sample project consist of an MBean which mimics the algorithm of jmx_exporter. Time measurement is done with metrics and exported also via MBeans. See the README for details.

The result for an vanilla JBoss Wildfly 8.2.0 with only this test war deployed is:

• MBeans: 880
• Attributes total: 4137
• Average attribute read time: 1.97 ms
• Reading all MBeans and attributes: 11 s

And again, all that without any rule applied, only reading of the attribute. And believe me, Wildfly is not the most populated server I've seen. Also it is not uncommon in integration scenarios to have many application specific MBeans, e.g one for each Camel Route which will add to this basic number.

My suggestion is to apply the rule before and use it for the query instead of this sledge hammer approach.

[brian-brazil]

Assuming that people tend to generate quite generic and large rulesets so that they can have a uniform agent installation everywhere so that they can have a uniform agent installation everywhere, this multiples to the number above.

The assumption is that you have a per-service ruleset as is shown in the example configs. Do you have benchmarks that indicate that this will be a problem compared to mBean slowness?

Time measurement is done with metrics and exported also via MBeans.

In general I wouldn't use that for benchmarks, as it's an exponentially decaying number (and also about 10x slower compared to simpler types of metrics). I'd suggest simple logging, JMH, or something like the simpleclient Summary/Histogram as they'll give you exact numbers.

My suggestion is to apply the rule before and use it for the query

The challenge with that is that with the features offered we need the full attribute with it's value in order to apply the rule.
I do expect a blacklist on mBean name to be added at some point, but I don't think this helps you. I guess we could extend that to attribute name too.
I'd also expect most rulesets to end up exposing most attributes, so I'm not sure you'll save much.

The fundamental issue is that JMX is slow as your benchmarks show, do you have any idea for ways to make that better?
If this is code you control I suggest switching your instrumentation to the java simpleclient which should be much faster and easier to use (see https://github.com/prometheus/client_java/blob/master/benchmark/README.md). The JMX exporter (as with all exporters) is intended for cases where you're transitioning to Prometheus or you don't have the option to use Prometheus more directly so we are limited to what other systems offer us in terms of metrics and performance.

[rhuss]

The assumption is that you have a per-service ruleset as is shown in the example configs. Do you have benchmarks that indicate that this will be a problem compared to mBean slowness?

That not but we had the following use case in mind: We want to create a base Java Docker image similar to this one, which can install and configure a Java agent during startup. It would have been nice to provide some base, very open, ruleset to match all possible application components (camel, jms, jdbc pools, ...) where rules applied only when the corresponding is applied and are ignored otherwise. This would ease maintenance, but of course, if the rules are not used in the MBean selection process, that doesn't make much sense.

In general I wouldn't use that for benchmarks, as it's an exponentially decaying number (and also about 10x slower compared to simpler types of metrics). I'd suggest simple logging, JMH, or something like the simpleclient Summary/Histogram as they'll give you exact numbers.

You are right, that was just a quick hack to demonstrate the problem, which hopefully becomes clear: Many MBeans usually, many attributes. Its not a benchmark.

The challenge with that is that with the features offered we need the full attribute with it's value in order to apply the rule.

Whats about making this a multi stage process: Use the MBean part of the rule to filter out MBeans in the query, then filter on the attributes and only then read it and filter on the value in a loop ?

The fundamental issue is that JMX is slow as your benchmarks show, do you have any idea for ways to make that better?

I don't think that JMX per se is slow, it depends on the amount of requests (4500 attribute reads/run in this example) and of course, how the MBeans are implemented (what you can't really influence). If you could filter out the 'expensive' reads already when querying for the MBeans that would help quite a bit. I'm pretty sure that your simpleclient easily becomes slow, too, if he had to read expensive (i.e. on-the-fly calculated) attributes, too. It uses MXBeans internally as well for some metrics, so there is no much difference here.

Personally, I don't consider JMX to be legacy (albeit quite old), but you get tons of metrics already for free on many platforms. Also with tools like hawtio there is some sort of JMX revival these days. A good and fast JMX exporter/bridge/acess ... for prometheus would be a good thing IMO, not only for transitioning.

To summarize: I don't think that JMX is slow, its the way how values are fetched here.

[brian-brazil]

This would ease maintenance, but of course, if the rules are not used in the MBean selection process, that doesn't make much sense.

It still makes sense I think.

Whats about making this a multi stage process: Use the MBean part of the rule to filter out MBeans in the query, then filter on the attributes and only then read it and filter on the value in a loop ?

As MBeans are used in such a wide variety of way, the way we generically support this is by putting everything into a string and letting you throw regexes at it. Accordingly we can't split out the MBean name as that'd require reverse engineering the regexes.

A blacklist of some form is an option, to deal with particularly slow or problematic mbeans/attributes.

I don't think that JMX per se is slow, it depends on the amount of requests (4500 attribute reads/run in this example)

4.5k metrics isn't particularly large. Doing a quick test with the Python client (which is less performant than the Java client), 4.5k metrics takes 60ms to render.

I'm pretty sure that your simpleclient easily becomes slow, too, if he had to read expensive (i.e. on-the-fly calculated) attributes, too.

It does permit that for special cases (e.g. that how the jmx exporter works), but in the general case we encourage usage such that updating metrics takes 10-15ns. On-the-fly calculations are discouraged, as they tend to be slow, harder to implement and there's usually a better way.

To summarize: I don't think that JMX is slow, its the way how values are fetched here.

It sounds like it may be how the values are generated. Something like this should take nanoseconds or microseconds per read, not milliseconds.

[rhuss]

The pattern you mention which prevents for optimisations, is this something Prometheus specific or specific to this exporter ? If the latter is the case, and if you would agree that optimisation on the query level for JMX makes sense, why not change the configuration syntax into something a bit more restrictive (but optimizable) ? E.g. a separate MBean pattern (which you can use in MBeanServer.query(), Attribute pattern (where you can iterate over before fetching the value), Value pattern (finally the pattern on the value) ?

[brian-brazil]

The pattern you mention which prevents for optimisations, is this something Prometheus specific or specific to this exporter ?

It's for this exporter. JMX is sufficiently non-standardised that a generic and highly configurable approach is called for.

If the latter is the case, and if you would agree that optimisation on the query level for JMX makes sense, why not change the configuration syntax into something a bit more restrictive (but optimizable) ?

That would make it less usable for the user, as they could no longer use the matching groups from a single regexp to determine details of a sample.

It's my belief that wanting to not read certain attributes for latency is an edge case, so I don't want to sacrifice usability for it.
What sort of speedups are you expecting from this?

[rhuss]

If I want to track 5000 attributes per server than none. If I want to concentrate on the 20 most important value than I expect a speedup of at least 100. Most of the values exposed via JMX are configuration values anyway which typically won't change every 15 seconds (the "M" in JMX).

Sorry, I think we come from different angles, so I don't want to bother you any longer.

Some final remarks, though:

• I really don't see many uses cases with filtering on the exact value (maybe on thresholds, though)
• Patterns with matching groups as you do can be easily done on MBean names and attribute names. E.g. a ^org.apache.cassandra.metrics<type=(\w+), name=(\w+)><count>Value: can be easily translated into MBeanServer.queryNames(new ObjectName("org.apache.cassandra.metrics:type=*,name=*,*"), iterate over the result, check for atributes named "count" and only then fetch the value. The only restriction for this to work is that capturing groups doesn't cross bracket boundaries. Such a big sacrifice to usability ?
• You know that the order of key-value props in an ObjectName is insignificant ? (there is a 'natural' and 'canonical' ordering, but in fact the order of the key-value pair doesn't matter from a JMX perspective). I wonder because you select for the default format the first key to have a special meaning in naming.
• Somewhat unrelated but might be helpful to you: Many appservers (older JBoss, especially Weblogic and Websphere), don't use the PlatformMBeanServer but their own implementation and sometimes not only a single one (Weblogic uses up to three). If you are interested which server uses which MBeanServer you might want to have a look into these classes

[brian-brazil]

If I want to track 5000 attributes per server than none. If I want to concentrate on the 20 most important value than I expect a speedup of at least 100. Most of the values exposed via JMX are configuration values anyway which typically won't change every 15 seconds (the "M" in JMX).

You indicated above that it was things being calculated on the fly that were slowing things down, so accordingly I'd expect static configuration values to be quite fast to process.

My suspicion is that there's a handful of mBeans/attributes that are really slow. Do you have a better breakdown of where time is being spent?

With the jmx servers I've worked with so far (apache things), static configuration is very much in the minority and I've wanted almost all the beans coming back as metrics.

Sorry, I think we come from different angles, so I don't want to bother you any longer.

I'm happy to accept adding a blacklist, or if JMX general performance is really that bad a whitelist.

I really don't see many uses cases with filtering on the exact value (maybe on thresholds, though)

It's mainly for extracting the value to put in a label as prometheus doesn't do strings, or where there's an enum filtering to convert it to a more useful set of variables.

The only restriction for this to work is that capturing groups doesn't cross bracket boundaries. Such a big sacrifice to usability ?

Yes, as it'd require a custom parser for the replacement strings and that wouldn't look like a normal regex substitution string.

I wonder because you select for the default format the first key to have a special meaning in naming.

That format pre-dates making the jmx exporter configurable, and I didn't want to break existing users so it remained as-is. The algorithm does seem to provide reasonably okay metrics in most cases.

Many appservers (older JBoss, especially Weblogic and Websphere), don't use the PlatformMBeanServer but their own implementation and sometimes not only a single one (Weblogic uses up to three).

Good to know thanks. This aspect of JMX was not documented that I could find.
This is something jmx_exporter should probably support in some form.

[rhuss]

FYI: I just was enable to run jmx_exporter on a vanilla Wildfly 8.2.0.Final without any rules and deployed application (needs some tweeks on the bootclasspath for wildfly):

# HELP jmx_scrape_duration_seconds Time this JMX scrape took, in seconds.
# TYPE jmx_scrape_duration_seconds gauge
jmx_scrape_duration_seconds 2.438419079
# HELP jmx_scrape_error Non-zero if this scrape failed.
# TYPE jmx_scrape_error gauge
jmx_scrape_error 1.0

[brian-brazil]

Speed wise that doesn't sound too bad (Cassandra takes about 12s), your run bailed out with an error though so probably didn't get to read everything.

[jimmidyson]

@brian-brazil Thing is without any rules this should only be exporting JVM stats & this should be super fast, right? Saying it's faster than something else that is really slow doesn't seem like a sane argument to me. This could very easily lead to overlapping scrapes.

Is there any chance we could reopen this issue & think about a solution to this together?

[rhuss]

you mean, it stops after an read error occurs ? Read error happen quite frequently because many MBeans are generated automatically and things like 'IllegalArgumentException' or 'UnsupportedOperationException' are quite common.

I suggest to simply to continue (but I think that is what jmx_exporter does anyway)

[jimmidyson]

@brian-brazil Our use case for jmx_exporter is to provide a way to export JMX metrics to Prometheus without the app having any dependency on any other libraries than it needs to function, including simpleclient.

Using jmx_exporter as a jvm agent is perfect for this use case as we can simply attach it to JVMs with appropriate config & we get JMX metrics ready to be scraped by Prometheus.

[brian-brazil]

Thing is without any rules this should only be exporting JVM stats & this should be super fast, right?

It'll export everything in PlatformMBeanServer by default. For the JVM stats the hope would be to add them to the simpleclient_hotspot collector so that all jvm users can benefit from them.

Is there any chance we could reopen this issue & think about a solution to this together?

Sure, I'm happy to look for a solution. Without knowing why things are slow it's hard to select the right solution.

I suggest to simply to continue (but I think that is what jmx_exporter does anyway)

It'll continue on common errors:
https://github.com/prometheus/jmx_exporter/blob/master/collector/src/main/java/io/prometheus/jmx/JmxScraper.java#L89
If there's some missing we can add them, though I wonder if we should just catch Exception.

Using jmx_exporter as a jvm agent is perfect for this use case as we can simply attach it to JVMs with appropriate config & we get JMX metrics ready to be scraped by Prometheus.

That's the intended use case.