codspeed-go is unable to run benchmarks on Go modules with local replaces

[mx-psi]

What is the issue?

Following the "Testing the benchmarks locally" section on the pdata/pprofile folder of the open-telemetry/opentelemetry-collector repository causes the following error:

Error: Failed to install codspeed-go dependency: go: go.opentelemetry.io/collector/featuregate@v1.45.0 (replaced by ../../featuregate): reading /featuregate/go.mod: open /featuregate/go.mod: no such file or directory

This is seemingly related to the local replace directive on the pdata/pprofile/go.mod file: https://github.com/open-telemetry/opentelemetry-collector/blob/c503bb11739a62f1d8ff4b1a40342bbe524835f4/pdata/pprofile/go.mod#L34.

Versions

• Commit with which I reproduced this: open-telemetry/opentelemetry-collector@c503bb1
• codspeed --version output: codspeed-runner 4.3.4
• go version output: go version go1.24.0 linux/amd64

Repro steps

Assumes you have gh, go and codspeed installed and have completed the auth steps:

Click to expand

❯ cd $(mktemp -d)
❯ gh repo clone open-telemetry/opentelemetry-collector
Cloning into 'opentelemetry-collector'...
remote: Enumerating objects: 145987, done.
remote: Counting objects: 100% (1972/1972), done.
remote: Compressing objects: 100% (688/688), done.
remote: Total 145987 (delta 1637), reused 1284 (delta 1284), pack-reused 144015 (from 4)
Receiving objects: 100% (145987/145987), 63.54 MiB | 2.85 MiB/s, done.
Resolving deltas: 100% (115821/115821), done.
❯ cd opentelemetry-collector/pdata/pprofile/
❯ codspeed run --mode walltime --skip-upload -- go test -bench=.

►►► Preparing the environment 
Tool already installed, skipping installation
Environment ready

►►► Running the benchmarks 
Sudo privileges are required to continue. Please enter your password.
[sudo] password for [REDACTED]: 
[INFO  codspeed_go_runner] Discovered 1 packages
[INFO  codspeed_go_runner] Total benchmarks discovered: 13
[INFO  codspeed_go_runner] Found BenchmarkFromAttributeIndices  in "attributes_test.go"
[INFO  codspeed_go_runner] Found BenchmarkSetAttribute          in "attributes_test.go"
[INFO  codspeed_go_runner] Found BenchmarkSetFunction           in "functions_test.go"
[INFO  codspeed_go_runner] Found BenchmarkSetLink               in "links_test.go"
[INFO  codspeed_go_runner] Found BenchmarkFromLocationIndices   in "locations_test.go"
[INFO  codspeed_go_runner] Found BenchmarkSetLocation           in "locations_test.go"
[INFO  codspeed_go_runner] Found BenchmarkSetMapping            in "mappings_test.go"
[INFO  codspeed_go_runner] Found BenchmarkProfilesFromProto     in "pb_test.go"
[INFO  codspeed_go_runner] Found BenchmarkProfilesToProto       in "pb_test.go"
[INFO  codspeed_go_runner] Found BenchmarkProfilesMarshalJSON   in "profiles_test.go"
[INFO  codspeed_go_runner] Found BenchmarkProfilesUsage         in "profiles_test.go"
[INFO  codspeed_go_runner] Found BenchmarkSetStack              in "stacks_test.go"
[INFO  codspeed_go_runner] Found BenchmarkSetString             in "string_table_test.go"
[INFO  codspeed_go_runner] Generating custom runner for package: pprofile
Error: Failed to install codspeed-go dependency: go: go.opentelemetry.io/collector/featuregate@v1.45.0 (replaced by ../../featuregate): reading /featuregate/go.mod: open /featuregate/go.mod: no such file or directory

Error: failed to execute the benchmark process: exit status: 1

Output not using codspeed

Click to expand

❯ go test -bench=.
goos: linux
goarch: amd64
pkg: go.opentelemetry.io/collector/pdata/pprofile
cpu: 12th Gen Intel(R) Core(TM) i9-12900H
BenchmarkFromAttributeIndices-20         1801542               643.5 ns/op           192 B/op          7 allocs/op
BenchmarkSetAttribute/with_a_new_attribute-20           61439257                17.40 ns/op            0 B/op          0 allocs/op
BenchmarkSetAttribute/with_an_existing_attribute-20     60663662                16.75 ns/op            0 B/op          0 allocs/op
BenchmarkSetAttribute/with_a_duplicate_attribute-20     67133756                16.13 ns/op            0 B/op          0 allocs/op
BenchmarkSetAttribute/with_a_hundred_locations_to_loop_through-20               43356356                25.58 ns/op             0 B/op          0 allocs/op
BenchmarkSetFunction/with_a_new_function-20                                     142362600                8.435 ns/op            0 B/op          0 allocs/op
BenchmarkSetFunction/with_an_existing_function-20                               150590827                8.393 ns/op            0 B/op          0 allocs/op
BenchmarkSetFunction/with_a_duplicate_function-20                               149041969                7.732 ns/op            0 B/op          0 allocs/op
BenchmarkSetFunction/with_a_hundred_functions_to_loop_through-20                100000000               11.69 ns/op             0 B/op          0 allocs/op
BenchmarkSetLink/with_a_new_link-20                                             107809521               11.90 ns/op            0 B/op          0 allocs/op
BenchmarkSetLink/with_an_existing_link-20                                       146690708                8.209 ns/op           0 B/op          0 allocs/op
BenchmarkSetLink/with_a_duplicate_link-20                                       147888278                8.703 ns/op           0 B/op          0 allocs/op
BenchmarkSetLink/with_a_hundred_links_to_loop_through-20                        121421527                9.857 ns/op           0 B/op          0 allocs/op
BenchmarkFromLocationIndices-20                                                  2841943               446.7 ns/op           248 B/op          6 allocs/op
BenchmarkSetLocation/with_a_new_location-20                                     88166844                17.85 ns/op            0 B/op          0 allocs/op
BenchmarkSetLocation/with_an_existing_location-20                               25558282                41.03 ns/op            0 B/op          0 allocs/op
BenchmarkSetLocation/with_a_duplicate_location-20                               75429883                17.26 ns/op            0 B/op          0 allocs/op
BenchmarkSetLocation/with_a_hundred_locations_to_loop_through-20                 1605705               698.8 ns/op             0 B/op          0 allocs/op
BenchmarkSetMapping/with_a_new_mapping-20                                       100000000               11.33 ns/op            0 B/op          0 allocs/op
BenchmarkSetMapping/with_an_existing_mapping-20                                 100000000               15.58 ns/op            0 B/op          0 allocs/op
BenchmarkSetMapping/with_a_duplicate_mapping-20                                 71525786                16.03 ns/op            0 B/op          0 allocs/op
BenchmarkSetMapping/with_a_hundred_mappings_to_loop_through-20                  56233498                18.43 ns/op            0 B/op          0 allocs/op
BenchmarkProfilesToProto-20                                                       336004              3500 ns/op
BenchmarkProfilesFromProto-20                                                      61065             20657 ns/op           15080 B/op        144 allocs/op
BenchmarkProfilesUsage-20                                                          98110             12000 ns/op             688 B/op         26 allocs/op
BenchmarkProfilesMarshalJSON-20                                                    93840             12312 ns/op            2234 B/op          8 allocs/op
BenchmarkSetStack/with_a_new_stack-20                                           90152194                13.34 ns/op            0 B/op          0 allocs/op
BenchmarkSetStack/with_an_existing_stack-20                                     84413710                12.74 ns/op            0 B/op          0 allocs/op
BenchmarkSetStack/with_a_duplicate_stack-20                                     88680921                12.61 ns/op            0 B/op          0 allocs/op
BenchmarkSetStack/with_a_hundred_stacks_to_loop_through-20                       1920799               638.3 ns/op             0 B/op          0 allocs/op
BenchmarkSetString/with_a_new_value-20                                          100000000               10.35 ns/op            0 B/op          0 allocs/op
BenchmarkSetString/with_an_existing_value-20                                    100000000               10.76 ns/op            0 B/op          0 allocs/op
BenchmarkSetString/with_a_duplicate_value-20                                    100000000               13.27 ns/op            0 B/op          0 allocs/op
BenchmarkSetString/with_a_hundred_values_to_loop_through-20                      3975289               277.3 ns/op             0 B/op          0 allocs/op
PASS
ok      go.opentelemetry.io/collector/pdata/pprofile    40.410s

[not-matthias]

Hey, thanks for the detailed bug report! I'll take a look and fix it ASAP.

[not-matthias]

Hey, I've implemented a fix. We previously only copied the module which was benchmarked to /tmp, which obviously fails if it depends on parent packages.

My coworkers are currently out-of-office, so we'll review and merge this tomorrow. If you want to use my fixes, you can manually install the Go runner which will override the version in our action.

      - uses: moonrepo/setup-rust@v1
        with:
          channel: stable

      - name: Install latest version of the codspeed-go-runner
        run: cargo install --git https://github.com/CodSpeedHQ/codspeed-go/ --branch cod-1666-codspeed-go-is-unable-to-run-benchmarks-on-go-modules-with

      - name: Run the benchmarks
        uses: CodSpeedHQ/action@v4
        with:
          working-directory: pdata/pprofile/
          mode: walltime
          run: go test -bench=. -benchtime=3s

[mx-psi]

Seems to work great locally, thank you for the quick turnaround!!

[adriencaccia]

Hey @mx-psi, I just released v0.5.0 with the fix, you can use it instead of the above workaround.

[mx-psi]

@adriencaccia Thank you! Does https://github.com/CodSpeedHQ/runner/ automatically pick up the latest version of codspeed-go?

[adriencaccia]

@adriencaccia Thank you! Does CodSpeedHQ/runner automatically pick up the latest version of codspeed-go?

@mx-psi if it is not already installed, yes! See https://github.com/CodSpeedHQ/runner/blob/2c7507f9068baed7b7d20f87c8d6551180c5ecfb/src/run/runner/helpers/introspected_golang/go.sh#L39-L44