Skip to content

monitoring: TBS metrics are not reported together #17342

Closed
@isaacaflores2

Description

@isaacaflores2

APM Server version (apm-server version): 9.0, 8.19

Description of the problem including expected versus actual behavior:

In a prior issue #17104 the reporting interval for the following metrics was fixedapm-server.sampling.tail.storage.lsm_size and apm-server.sampling.tail.storage.value_log_size.

But the second problem with all sampling configs never being reported was not addressed.

Also, apm-server.sampling.tail.storage.lsm_size and apm-server.sampling.tail.dynamic_service_groups are never reported together in 9.0.

Testing locally this is caused because both sets of metrics use the same namespace apm-server.sampling but they are created using different instances of a Meter (meterProvider.Meter()). Changing apm-server.sampling.tail.storage.lsm_size to apm-server.storage.lsm_size allow both to be reported.

Example stats
{
	"apm-server": {
		"agentcfg": {
			"elasticsearch": {
				"cache": {
					"entries": {
						"count": 0
					},
					"refresh": {
						"successes": 1
					}
				}
			}
		},
		"otlp": {
			"grpc": {
				"logs": {
					"request": {
						"count": 1
					},
					"response": {
						"count": 1,
						"valid": {
							"count": 1
						}
					}
				},
				"metrics": {
					"request": {
						"count": 1
					},
					"response": {
						"count": 1,
						"valid": {
							"count": 1
						}
					}
				},
				"traces": {
					"request": {
						"count": 3
					},
					"response": {
						"count": 3,
						"valid": {
							"count": 3
						}
					}
				}
			}
		},
		"processor": {
			"error": {
				"transformations": 1
			},
			"log": {
				"transformations": 2
			},
			"metric": {
				"transformations": 2
			}
		},
		"sampling": {
			"tail": {
				"dynamic_service_groups": 1,
				"events": {
					"processed": 3,
					"stored": 3
				}
			}
		},
		"storage": {
			"lsm_size": 831932,
			"value_log_size": 0
		}
	},
	"libbeat": {
		"config": {
			"module": {
				"running": 0,
				"starts": 0,
				"stops": 0
			},
			"reloads": 0,
			"scans": 0
		},
		"output": {
			"events": {
				"acked": 5,
				"active": 0,
				"batches": 1,
				"failed": 0,
				"toomany": 0,
				"total": 5
			},
			"type": "elasticsearch",
			"write": {
				"bytes": 704
			}
		},
		"pipeline": {
			"events": {
				"total": 5
			}
		}
	},
	"output": {
		"elasticsearch": {
			"bulk_requests": {
				"available": 65,
				"completed": 1
			},
			"indexers": {
				"active": 1,
				"created": 0,
				"destroyed": 0
			}
		}
	}
}

Steps to reproduce:

  1. In apm-server.yml set http.enabled: true and enable TBS
  2. Send data using ./sendotlp -insecure -endpoint=http://localhost:8200 -api-key=
  3. Observe lsm_size and dynamic_service_groups are never reported together with stats endpoint (default localhost:5066/stats)

Possible Solutions:

  1. Rename metrics so the lsm_size and dynamic_service_groups
  2. Refactor the instrumentation so that a single instance of a meter creates all metrics under the apm-server.sampling namespace

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions