[chore] k8s e2e test enhancement

[fatsheep9146]

Component(s)

internal/k8stest

Is your feature request related to a problem? Please describe.

The k8s e2e testing module encountered a critical bug #25849 a few days ago, which caused many CI processes to fail and significantly impacted the release schedule of the entire project. Although the root cause of the problem has been identified #25889 (though it is not a solution to all issues), this incident highlighted the need for enhancements in the e2e module.

• On one hand, we need to reduce the probability of similar issues occurring again,
• On the other hand, we should preserve more context to troubleshoot flaky tests if they occur again.

So I open this issue to track some follow-ups needed to enhance the e2e test, some of them is already discussed in slack channel

here are some follow-ups I came up

• add e2e test for telemetrygen
• add more error log or event when collector could not start in time
• add more error log or event when telemetrygen could not start in time

Describe alternatives you've considered

No response

Additional context

No response

[github-actions]

Pinging code owners:

• internal/k8stest: @crobert-1

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[crobert-1]

/label cmd/telemetrygen

[TylerHelmuth]

I don't really want to add e2e tests for telemetrygen since its a utility, but they would've caught this issue so it is probably the optimal solution. If we can get a good enough solution with better logging/events in the k8s e2e tests I am good with that.

[crobert-1]

/label -needs-triage

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

• cmd/telemetrygen: @mx-psi @codeboten
• internal/k8stest: @crobert-1

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[atoulme]

We at least have some support for regression testing with #26668 now, can we close this or maybe split this issue?