Codspeed 2.8 hangs or kills the CI runner

[tan3-netapp]

Currently, codspeed 2.8 hangs or kills the CI runner when being run as the benchmark check of the shotover-proxy repository.
The latest working version of codspeed for this job is 2.7.2.

To reproduce the issue, please look into this change for more information.

[art049]

We just released v2.8.1 statically linked to avoid issues with further issues with Glibc.

On an unrelated note, we also saw your workflow is using the version 2 of the codspeed action, we have a temporary issue with backward compatibility that makes those workflows temporarily delayed. If you can bump the action to version 3 this will be fixed directly

[art049]

@tan3-netapp we're trying to understand the issue. Can you try with 2.8.1 and the latest runner version (v3)?

As mentioned in my previous message, there is a slight backward compatibility issue on our side that might explain the hanging you're seeing but I'm just guessing.

[rukai]

Here is an attempt with 2.8.1 shotover/shotover-proxy#1870 you can see that:

• the musl fix now allows cargo-codspeed to run at all!
• however it now hangs towards the end of the compile.
  • although it didnt happen in this run, I've seen the bug manifest in the CI runner immediately killed at the same point where it hangs.
  • I've seen this behaviour on locally compiled non-musl builds too.
  • I am most suspicious of 6a00c7f maybe we are blocking cargo's execution by not reading its stderr? maybe the logic for copying built artifacts is hitting something weird.