Description
APM Server version (apm-server version
): 9.0, 8.19
Description of the problem including expected versus actual behavior:
In a prior issue #17104 the reporting interval for the following metrics was fixedapm-server.sampling.tail.storage.lsm_size
and apm-server.sampling.tail.storage.value_log_size
.
But the second problem with all sampling configs never being reported was not addressed.
Also,
apm-server.sampling.tail.storage.lsm_size
andapm-server.sampling.tail.dynamic_service_groups
are never reported together in 9.0.
Testing locally this is caused because both sets of metrics use the same namespace apm-server.sampling
but they are created using different instances of a Meter (meterProvider.Meter()
). Changing apm-server.sampling.tail.storage.lsm_size
to apm-server.storage.lsm_size
allow both to be reported.
Example stats
{
"apm-server": {
"agentcfg": {
"elasticsearch": {
"cache": {
"entries": {
"count": 0
},
"refresh": {
"successes": 1
}
}
}
},
"otlp": {
"grpc": {
"logs": {
"request": {
"count": 1
},
"response": {
"count": 1,
"valid": {
"count": 1
}
}
},
"metrics": {
"request": {
"count": 1
},
"response": {
"count": 1,
"valid": {
"count": 1
}
}
},
"traces": {
"request": {
"count": 3
},
"response": {
"count": 3,
"valid": {
"count": 3
}
}
}
}
},
"processor": {
"error": {
"transformations": 1
},
"log": {
"transformations": 2
},
"metric": {
"transformations": 2
}
},
"sampling": {
"tail": {
"dynamic_service_groups": 1,
"events": {
"processed": 3,
"stored": 3
}
}
},
"storage": {
"lsm_size": 831932,
"value_log_size": 0
}
},
"libbeat": {
"config": {
"module": {
"running": 0,
"starts": 0,
"stops": 0
},
"reloads": 0,
"scans": 0
},
"output": {
"events": {
"acked": 5,
"active": 0,
"batches": 1,
"failed": 0,
"toomany": 0,
"total": 5
},
"type": "elasticsearch",
"write": {
"bytes": 704
}
},
"pipeline": {
"events": {
"total": 5
}
}
},
"output": {
"elasticsearch": {
"bulk_requests": {
"available": 65,
"completed": 1
},
"indexers": {
"active": 1,
"created": 0,
"destroyed": 0
}
}
}
}
Steps to reproduce:
- In apm-server.yml set http.enabled: true and enable TBS
- Send data using
./sendotlp -insecure -endpoint=http://localhost:8200 -api-key=
- Observe lsm_size and dynamic_service_groups are never reported together with stats endpoint (default localhost:5066/stats)
Possible Solutions:
- Rename metrics so the
lsm_size
anddynamic_service_groups
- Refactor the instrumentation so that a single instance of a meter creates all metrics under the
apm-server.sampling
namespace