misconfigured component is created twice

[korniltsev]

What's wrong?

Sometimtes when a component is misconfigured (component creation returns error), the component is recreated second time which may lead to unexpected panics

The easiest to reproduce is to run this branch https://github.com/grafana/alloy/compare/korniltsev/alloy-component-recreation-bug2?expand=1
Try to run this branch multiple times and observe a random "new new new " + panic

I expect a component to be created only once.
It does not make sense to recreate the component if the configuration is not changed.
I expect prom Registerer to be fresh and do not expect the component to handle registry duplicate errors due to unexpected component recreation with a non pristine registrer

Steps to reproduce

misconfigure component

System information

linux amd64

Software version

https://github.com/grafana/alloy/compare/korniltsev/alloy-component-recreation-bug2?expand=1

Configuration

logging {
    format = "logfmt"
    level = "debug"
}
pyroscope.ebpf "profiler" {
    forward_to = [pyroscope.write.localhost.receiver]
}

pyroscope.write "localhost" {
    endpoint {
        url = "http://localhost:4040"
    }
}

Logs

GOROOT=/home/korniltsev/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.7.linux-amd64 #gosetup
GOPATH=/home/korniltsev/go #gosetup
/home/korniltsev/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.7.linux-amd64/bin/go build -tags pyroscope_ebpf,pyroscope_ebpf_no_link -o /home/korniltsev/.cache/JetBrains/IntelliJIdea2024.3/tmp/GoLand/___10ebpf . #gosetup
/home/korniltsev/.cache/JetBrains/IntelliJIdea2024.3/tmp/GoLand/___10ebpf run /home/korniltsev/p/alloy/data-alloy/ebpf.txt #gosetup
ts=2025-03-10T04:09:06.105615503Z level=info "boringcrypto enabled"=false
ts=2025-03-10T04:09:06.105084003Z level=info source=/home/korniltsev/go/pkg/mod/github.com/!kim!machine!gun/automemlimit@v0.6.1/memlimit/memlimit.go:170 msg="memory is not limited, skipping" package=github.com/KimMachineGun/automemlimit/memlimit
ts=2025-03-10T04:09:06.105651942Z level=info msg="no peer discovery configured: both join and discover peers are empty" service=cluster
ts=2025-03-10T04:09:06.105655199Z level=info msg="running usage stats reporter"
ts=2025-03-10T04:09:06.105657253Z level=info msg="starting complete graph evaluation" controller_path=/ controller_id="" trace_id=4f7449f3f5d46f781b322d67dce81b69
ts=2025-03-10T04:09:06.105664887Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=4f7449f3f5d46f781b322d67dce81b69 node_id=livedebugging duration=6.753µs
ts=2025-03-10T04:09:06.105668745Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=4f7449f3f5d46f781b322d67dce81b69 node_id=tracing duration=2.384µs
ts=2025-03-10T04:09:06.105672612Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=4f7449f3f5d46f781b322d67dce81b69 node_id=logging duration=69.654µs
ts=2025-03-10T04:09:06.105682361Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=4f7449f3f5d46f781b322d67dce81b69 node_id=labelstore duration=3.366µs
ts=2025-03-10T04:09:06.105830194Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=4f7449f3f5d46f781b322d67dce81b69 node_id=pyroscope.write.localhost duration=142.774µs
ts=2025-03-10T04:09:06.105882765Z level=debug msg="set targets" component_path=/ component_id=pyroscope.ebpf.profiler count=0
ts=2025-03-10T04:09:06.105887635Z level=debug msg="created targets" component_path=/ component_id=pyroscope.ebpf.profiler cid2target=0 pid2target=0
ts=2025-03-10T04:09:06.106021862Z level=error msg="failed to evaluate config" controller_path=/ controller_id="" trace_id=4f7449f3f5d46f781b322d67dce81b69 node=pyroscope.ebpf.profiler err="building component: panic repro"
ts=2025-03-10T04:09:06.106034025Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=4f7449f3f5d46f781b322d67dce81b69 node_id=pyroscope.ebpf.profiler duration=194.403µs
ts=2025-03-10T04:09:06.106100212Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=4f7449f3f5d46f781b322d67dce81b69 node_id=remotecfg duration=60.627µs
ts=2025-03-10T04:09:06.10611438Z level=info msg="applying non-TLS config to HTTP server" service=http
ts=2025-03-10T04:09:06.106118006Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=4f7449f3f5d46f781b322d67dce81b69 node_id=http duration=10.901µs
ts=2025-03-10T04:09:06.106124509Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=4f7449f3f5d46f781b322d67dce81b69 node_id=ui duration=1.022µs
ts=2025-03-10T04:09:06.106130711Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=4f7449f3f5d46f781b322d67dce81b69 node_id=cluster duration=1.303µs
ts=2025-03-10T04:09:06.106136121Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=4f7449f3f5d46f781b322d67dce81b69 node_id=otel duration=862ns
ts=2025-03-10T04:09:06.106140199Z level=info msg="finished complete graph evaluation" controller_path=/ controller_id="" trace_id=4f7449f3f5d46f781b322d67dce81b69 duration=611.524µs
Error: /home/korniltsev/p/alloy/data-alloy/ebpf.txt:5:1: Failed to build component: building component: panic repro

4 |   }
5 |   pyroscope.ebpf "profiler" {
  |  _^^^^^^^^^^^^^^^^^^^^^^^^^^^
6 | |     forward_to = [pyroscope.write.localhost.receiver]
7 | | }
  | |_^
8 |   
interrupt received
ts=2025-03-10T04:09:06.106257314Z level=error msg="failed to start reporter" err="context canceled"
ts=2025-03-10T04:09:06.106296449Z level=debug msg="set targets" component_path=/ component_id=pyroscope.ebpf.profiler count=0
ts=2025-03-10T04:09:06.106330504Z level=debug msg="created targets" component_path=/ component_id=pyroscope.ebpf.profiler cid2target=0 pid2target=0
ts=2025-03-10T04:09:06.106440285Z level=debug msg="finished node evaluation" controller_path=/ controller_id="" node_id=pyroscope.ebpf.profiler duration=249.759µs
panic: duplicate metrics collector registration attempted

goroutine 214 [running]:
github.com/prometheus/client_golang/prometheus.(*wrappingRegisterer).MustRegister(0xc001a7cab0, {0xc001f06660?, 0x0?, 0x0?})
	/home/korniltsev/go/pkg/mod/github.com/prometheus/client_golang@v1.20.5/prometheus/wrap.go:104 +0x12b
github.com/grafana/pyroscope/ebpf/metrics.NewSymtabMetrics({0xc876ea8, 0xc001a7cab0})
	/home/korniltsev/go/pkg/mod/github.com/grafana/pyroscope/ebpf@v0.4.8/metrics/symtab.go:43 +0x88d
github.com/grafana/pyroscope/ebpf/metrics.New({0xc876ea8, 0xc001a7cab0})
	/home/korniltsev/go/pkg/mod/github.com/grafana/pyroscope/ebpf@v0.4.8/metrics/metrics.go:12 +0x25
github.com/grafana/alloy/internal/component/pyroscope/ebpf.newMetrics({0xc876ea8, 0xc001a7cab0})
	/home/korniltsev/p/alloy/internal/component/pyroscope/ebpf/metrics.go:49 +0x459
github.com/grafana/alloy/internal/component/pyroscope/ebpf.New({{0x782748df3f40, 0xc001a7c990}, {0xc001a26678, 0x17}, {0xc7e0660, 0xc001a50b90}, {0xc0013cb860, 0x22}, 0xc001a373c0, {0xc876ea8, ...}, ...}, ...)
	/home/korniltsev/p/alloy/internal/component/pyroscope/ebpf/ebpf_linux.go:45 +0x17f
github.com/grafana/alloy/internal/component/pyroscope/ebpf.init.0.func1({{0x782748df3f40, 0xc001a7c990}, {0xc001a26678, 0x17}, {0xc7e0660, 0xc001a50b90}, {0xc0013cb860, 0x22}, 0xc001a373c0, {0xc876ea8, ...}, ...}, ...)
	/home/korniltsev/p/alloy/internal/component/pyroscope/ebpf/ebpf_linux.go:34 +0x138
github.com/grafana/alloy/internal/runtime/internal/controller.(*BuiltinComponentNode).evaluate(0xc00134afc8, 0xc00069a6d8)
	/home/korniltsev/p/alloy/internal/runtime/internal/controller/node_builtin_component.go:285 +0x4b6
github.com/grafana/alloy/internal/runtime/internal/controller.(*BuiltinComponentNode).Evaluate(0xc00134afc8, 0xb171700?)
	/home/korniltsev/p/alloy/internal/runtime/internal/controller/node_builtin_component.go:258 +0x1c
github.com/grafana/alloy/internal/runtime/internal/controller.(*Loader).concurrentEvalFn(0xc001a2a820, {0x782748df3f60, 0xc00134afc8}, {0xc8a6868, 0xc00177e5d0}, {0xc80e4f0, 0xc00177e540}, 0xc001e8c6f0)
	/home/korniltsev/p/alloy/internal/runtime/internal/controller/loader.go:833 +0x5d5
github.com/grafana/alloy/internal/runtime/internal/controller.(*Loader).EvaluateDependants.func2()
	/home/korniltsev/p/alloy/internal/runtime/internal/controller/loader.go:770 +0x33
github.com/grafana/alloy/internal/runtime/internal/worker.(*workQueue).emitNextTask.func1()
	/home/korniltsev/p/alloy/internal/runtime/internal/worker/worker_pool.go:181 +0x5c
github.com/grafana/alloy/internal/runtime/internal/worker.(*fixedWorkerPool).start.func1()
	/home/korniltsev/p/alloy/internal/runtime/internal/worker/worker_pool.go:87 +0x67
created by github.com/grafana/alloy/internal/runtime/internal/worker.(*fixedWorkerPool).start in goroutine 1
	/home/korniltsev/p/alloy/internal/runtime/internal/worker/worker_pool.go:80 +0x25
new new new <<<< 

new new new <<<< 

Process finished with the exit code 2

[wildum]

hi, thanks for reporting this bug. Although it's only triggered when the component fails to build on the first load, and that Alloy will exit anyway, it's still confusing for the user.

Here is what's happening:

The ebpf depends on the write component, so the write component will be evaluated first. At creation, it's updating its exports which has the effect of pushing its dependants (the ebfp component here) in the concurrent evaluation queue.

But the evaluation of the ebpf component was already planned. This means that the component will be evaluated twice. The mutex here prevents the evaluation from running concurrently so they will just be done one right after the other.

What should happen is that the 1st evaluation builds the component and evaluates it, and then the 2nd evaluation just re-evaluates the component. But because the 1st eval fails to build it, the 2nd will also attempt to build it before Alloy exits. This is why you see a double attempt to create the component.

A potential fix would be to prevent the concurrent evaluation to be carried out on nodes that have not been through their initial evaluation. This fix would also improve startup performance by removing these unnecessary re-evaluations.

[github-actions]

This issue has not had any activity in the past 30 days, so the needs-attention label has been added to it.
If the opened issue is a bug, check to see if a newer release fixed your issue. If it is no longer relevant, please feel free to close this issue.
The needs-attention label signals to maintainers that something has fallen through the cracks. No action is needed by you; your issue will be kept open and you do not have to respond to this comment. The label will be removed the next time this job runs if there is new activity.
Thank you for your contributions!