[tmpnet] Deploy collectors with golang to simplify cross-repo use

[maru-ava]

Why this should be merged

Previously, prometheus and promtail for collecting from local tmpnet networks were installed and launched by bash scripts. Migrating installation to nix and launch to golang simplifies reuse from subnet-evm and hypersdk. No more having to copy and maintain copies of the scripts in multiple repos.

How this works

• migrates scripts launching prometheus and promtail to golang
• configures collector start via the tmpnet e2e flag --start-collectors
• enables start and stop of collectors via tmpnetctl

How this was tested

• CI for regression
• Manually checked that metrics and logs were collected from local and CI networks with the chnages
  • Automatic checks implemented in [tmpnet] Add check for collection of logs and metrics to custom github action #3740

Need to be documented in RELEASES.md?

N/A

TODO

• Merge [testing] Replace script-based tool installation with nix #3691
• Refactor for review
• Merge [ci] Simplify tmpnet monitoring action #3736

[aaronbuchwald]

I believe you can create the command and set Stdout and Stderr to nil directly (according to the docs)

[maru-ava]

Please give it a try and get back to me, because my attempts to create a child process that outlives its parent and configured with stdout and stderr redirected to a file created and opened by that parent were entirely unsuccessful.

[maru-ava]

I've added a comment detailing why I took this approach, PTAL.

[aaronbuchwald]

I may be wrong here, just remembered reading the documentation around this, so thought I'd provide the suggestion. May not behave as I expected from reading the docs (either due to a misread or incorrect docs).

[maru-ava]

Re-reading your original comment, what is the intended result of setting Stdout and Stderr to nil? I need to send the output to a file, otherwise in CI I won't be able to collect the logs for troubleshooting (from the same tmpnet dir artifacts that contain the networks).

[aaronbuchwald]

I think that I thought you were trying to ignore the output here:

avalanchego/tests/fixture/tmpnet/start_collectors.go

Line 182 in 15c496b

// TODO(marun) Figure out a way to redirect stdout and stderr of a detached child process without a bash shell

, but you actually wanted to write it to the specified file. I think you could still do this by setting Stdout and Stderr to the corresponding file and avoid constructing a bash command.

[maru-ava]

I attempted to do just that, but the following combination did not result in a log file that the child was writing to:

• child process that will outlive its parent
• child process configured with stderr and stdout set to a file opened by the parent

[maru-ava]

Rebased