Revert "[8.19] (backport #17154) tbs: Update storage metrics to be reported synchronously in the existing runDiskUsageLoop method"

[isaacaflores2]

These changes attempted to fix reporting issue for lsm_size metrics where we believed were caused by the use of Observable OTEL metrics. The changes introduced a regression where we observed the lsm_size metrics less frequently than before.

After updating metrics collection to remove conflicts (#17525), we determined this was the root issue that was contributing to the infrequent lsm_size metrics (not the Observable OTEL metrics). The changes in this PR fixed the regression mention above.

The prior implementation with Observable metrics is preferred so we will revert #17275

Draft PR:

• Validate stats endpoint locally
• Add manual test steps

How to test these changes

1. Run apm-server with TBS enabled
2. Send data to the server ./sendotlp -insecure -endpoint=http://localhost:8200 -secret-token=<token>
3. View metrics via the stats endpoint (http://localhost:5066/stats?pretty) . storage should be visible along with other metrics under apm-server.sampling.tail.

• Call the stats endpoint multiple times. All sets of metrics should be consistently visible

"sampling": {
		"tail": {
			"dynamic_service_groups": 0,
			"events": {
				"processed": 3,
				"sampled": 6,
				"stored": 3
			},
			"storage": {
				"lsm_size": 7431,
				"value_log_size": 0
			}
		}
	}

apm server config

apm-server:
  host: "127.0.0.1:8200"

  sampling.tail:
    enabled: true
    policies:
      - sample_rate: 1.0

output.elasticsearch:
  hosts: ["https://https://cloud-url:443"]
  username: "elastic"
  password: "<REDACTED>"

http:
  enabled: true
  host: localhost
  port: 5066
  
monitoring.enabled: true
monitoring.elasticsearch:
  hosts: [ "https://cloud-url:443" ]
  username: "elastic"
  password: "<REDACTED>"  

[github-actions]

🤖 GitHub comments

Expand to view the GitHub comments

Just comment with:

• run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)