Fix race condition in otelcol component wrappers

[thampiotr]

PR Description

We have a general issue with OTel components where consumers may be used before the Start functions in OTel have finished running. This is because in OTel Start functions are non-blocking and sometimes do work to set things up, like it was the case for batch_processor. In Alloy, however we have Run function that is blocking for the lifetime of the component. As soon as it's called, we consider the component Running. In OTel, however, the Start function should be called and exit to consider a component running.

The solution here is to pause the Consumer until we are sure that the OTel component scheduler has called Start on all OTel components. The consumer will block any attempts to feed data to it.

Which issue(s) this PR fixes

Notes to the Reviewer

thampiotr@MacWork ~/w/alloy (update-otel-112-fix-race)> go test ./internal/component/otelcol/internal/scheduler/ -count 50 -race
ok  	github.com/grafana/alloy/internal/component/otelcol/internal/scheduler	3.543s
thampiotr@MacWork ~/w/alloy (update-otel-112-fix-race)> go test ./internal/component/otelcol/internal/lazyconsumer/ -count 50 -race
ok  	github.com/grafana/alloy/internal/component/otelcol/internal/lazyconsumer	17.495s

PR Checklist

• CHANGELOG.md updated
• Documentation added
• Tests updated
• Config converters updated

[wildum]

Thanks for working on this, I think that's a step in the right direction.
The logic was not super clear to me at first in the scheduler, the idea is that the components are created in the paused state so they are not paused on the first run but are always resumed after being started because they are paused when the scheduler stops running, right? Maybe we could have this explanation in the scheduler code

[wildum]

If pause is called by accident on a paused consumer or resume on a running consumer, it will be dramatic. Should the consumer holds its state (smth like "isPaused") to avoid locking if its already paused or avoid resuming if it's already running for extra safety?

[thampiotr]

Yeah, I was thinking about this too, I wouldn't want to have any dramatic panics either. I've changed the implementation to handle multiple calls to Resume and Pause - it's a bit more complex, but I also wrote a lot of tests.

[wildum]

if the component was created with New() instead of NewPaused() then this will panic because on the first run you will try to unlock the mut that's not locked

[thampiotr]

Now it will no longer panic as calls to Resume can be repeated.

When writing code we still need to take care to use the correct paused or resumed consumer, but the consequences of a mistake shouldn't be catastrophic.

[clayton-cornell]

Just some edits to align with the same changes in #2012 Might also be "fixed" up when the PR conflicts are resolved?