[Stack Monitoring] [APM] Visualization differs from real data

[simitt]

The visualized data for APM monitoring seem to diverge from the amount of data ingested. Breaking this down to only having 1 ingestion request containing 1 event, the visualization shows 6 requests/m and 6 events/m (see screenshot):

Looking at the indexed documents the data seem legit (increasing according counter values by 1).

Used version:
tested with 7.0 and master (Kibana, ES and APM Server)

[simitt]

As pointed out by @chrisronline, this doesn't seem to be a bug but related to normalization over selected time.

[chrisronline]

I just want to add some observations here.

The time picker for the above screenshot appears to have a range of 2m. We use this delta to auto calculate a bucket interval size, which would be 10s (technically, this range would incur whatever the xpack.monitoring.min_interval_seconds config is but the default is 10s).

So that means we will have ~12 buckets. For each bucket, we calculate the max of the field and then perform a derivative (for all but the first bucket).

For some reason, when we did the APM monitoring code, we decided to use a value of 1m for the derivative unit (which goes against the default 1s for the rest of monitoring).

Because of that, the derivative values, normalized out from 10s buckets to 1m, are actually derivative_value * 6.

The graphs are labeled as /m to indicate the units of minutes, but maybe this is wrong? Maybe we should use the standard 1s derivative unit as the rest of monitoring?

[simitt]

Sorry for the late response. The @elastic/apm-server team would be fine with changing the unit to the standard derivative units as used for the rest of the monitoring, but not urgent from our side.