valgrind installation is slow (1 - 2 minutes)

[Boshen]

We always assumed that cargo codspeed run is slow for the last past two years until today I discovered that the runner runs sudo apt-get update + sudo apt-get install.

Raw logs

2025-09-25T10:23:29.4745865Z �[36;1msudo apt-get update�[0m
2025-09-25T10:23:29.4746209Z �[36;1msudo apt-get install --allow-downgrades -y /tmp/valgrind-codspeed.deb�[0m
2025-09-25T10:23:29.4778410Z shell: /usr/bin/bash -e {0}
2025-09-25T10:23:29.4778634Z env:
2025-09-25T10:23:29.4778810Z   CACHE_ON_FAILURE: false
2025-09-25T10:23:29.4779023Z   CARGO_INCREMENTAL: 0
2025-09-25T10:23:29.4779216Z ##[endgroup]
2025-09-25T10:23:29.4875677Z   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
2025-09-25T10:23:29.4876358Z                                  Dload  Upload   Total   Spent    Left  Speed
2025-09-25T10:23:29.4876683Z 
2025-09-25T10:23:29.7428603Z   0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
2025-09-25T10:23:29.7429298Z   0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
2025-09-25T10:23:29.9852746Z 
2025-09-25T10:23:30.2365342Z   2 10.8M    2  223k    0     0   449k      0  0:00:24 --:--:--  0:00:24  449k
2025-09-25T10:23:30.2366075Z 100 10.8M  100 10.8M    0     0  14.4M      0 --:--:-- --:--:-- --:--:-- 42.3M
2025-09-25T10:23:30.3815645Z Get:1 file:/etc/apt/apt-mirrors.txt Mirrorlist [144 B]
2025-09-25T10:23:30.4236231Z Hit:2 http://azure.archive.ubuntu.com/ubuntu noble InRelease
2025-09-25T10:23:30.4237790Z Hit:6 https://packages.microsoft.com/repos/azure-cli noble InRelease
2025-09-25T10:23:30.4239877Z Get:7 https://packages.microsoft.com/ubuntu/24.04/prod noble InRelease [3600 B]
2025-09-25T10:23:30.4275514Z Get:3 http://azure.archive.ubuntu.com/ubuntu noble-updates InRelease [126 kB]
2025-09-25T10:23:30.4330242Z Get:4 http://azure.archive.ubuntu.com/ubuntu noble-backports InRelease [126 kB]
2025-09-25T10:23:30.4383150Z Get:5 http://azure.archive.ubuntu.com/ubuntu noble-security InRelease [126 kB]
2025-09-25T10:23:30.6489886Z Get:8 https://packages.microsoft.com/ubuntu/24.04/prod noble/main amd64 Packages [55.6 kB]
2025-09-25T10:23:30.6716661Z Get:9 https://packages.microsoft.com/ubuntu/24.04/prod noble/main arm64 Packages [38.1 kB]
2025-09-25T10:23:30.6909606Z Get:10 http://azure.archive.ubuntu.com/ubuntu noble-updates/main amd64 Packages [1443 kB]
2025-09-25T10:23:30.6991271Z Get:11 http://azure.archive.ubuntu.com/ubuntu noble-updates/main Translation-en [282 kB]
2025-09-25T10:23:30.7018933Z Get:12 http://azure.archive.ubuntu.com/ubuntu noble-updates/main amd64 Components [175 kB]
2025-09-25T10:23:30.7042915Z Get:13 http://azure.archive.ubuntu.com/ubuntu noble-updates/main amd64 c-n-f Metadata [15.3 kB]
2025-09-25T10:23:30.7055420Z Get:14 http://azure.archive.ubuntu.com/ubuntu noble-updates/universe amd64 Packages [1485 kB]
2025-09-25T10:23:30.7143096Z Get:15 http://azure.archive.ubuntu.com/ubuntu noble-updates/universe amd64 Components [377 kB]
2025-09-25T10:23:30.7177813Z Get:16 http://azure.archive.ubuntu.com/ubuntu noble-updates/universe amd64 c-n-f Metadata [31.1 kB]
2025-09-25T10:23:30.7189063Z Get:17 http://azure.archive.ubuntu.com/ubuntu noble-updates/restricted amd64 Packages [1957 kB]
2025-09-25T10:23:30.7289128Z Get:18 http://azure.archive.ubuntu.com/ubuntu noble-updates/restricted Translation-en [441 kB]
2025-09-25T10:23:30.7319574Z Get:19 http://azure.archive.ubuntu.com/ubuntu noble-updates/restricted amd64 Components [212 B]
2025-09-25T10:23:30.7750536Z Get:20 http://azure.archive.ubuntu.com/ubuntu noble-updates/multiverse amd64 Components [940 B]
2025-09-25T10:23:30.7768409Z Get:21 http://azure.archive.ubuntu.com/ubuntu noble-backports/main amd64 Components [7072 B]
2025-09-25T10:23:30.7779194Z Get:22 http://azure.archive.ubuntu.com/ubuntu noble-backports/universe amd64 Components [19.2 kB]
2025-09-25T10:23:30.7789939Z Get:23 http://azure.archive.ubuntu.com/ubuntu noble-backports/restricted amd64 Components [212 B]
2025-09-25T10:23:30.7800206Z Get:24 http://azure.archive.ubuntu.com/ubuntu noble-backports/multiverse amd64 Components [212 B]
2025-09-25T10:23:30.8064695Z Get:25 http://azure.archive.ubuntu.com/ubuntu noble-security/main amd64 Packages [1164 kB]
2025-09-25T10:23:30.8130774Z Get:26 http://azure.archive.ubuntu.com/ubuntu noble-security/main Translation-en [197 kB]
2025-09-25T10:23:30.8149493Z Get:27 http://azure.archive.ubuntu.com/ubuntu noble-security/main amd64 Components [21.6 kB]
2025-09-25T10:23:30.8160894Z Get:28 http://azure.archive.ubuntu.com/ubuntu noble-security/main amd64 c-n-f Metadata [8744 B]
2025-09-25T10:23:30.8172542Z Get:29 http://azure.archive.ubuntu.com/ubuntu noble-security/universe amd64 Packages [880 kB]
2025-09-25T10:23:30.8225932Z Get:30 http://azure.archive.ubuntu.com/ubuntu noble-security/universe amd64 Components [52.2 kB]
2025-09-25T10:23:30.8237910Z Get:31 http://azure.archive.ubuntu.com/ubuntu noble-security/universe amd64 c-n-f Metadata [18.0 kB]
2025-09-25T10:23:30.8251966Z Get:32 http://azure.archive.ubuntu.com/ubuntu noble-security/restricted amd64 Packages [1855 kB]
2025-09-25T10:23:30.8338887Z Get:33 http://azure.archive.ubuntu.com/ubuntu noble-security/restricted Translation-en [419 kB]
2025-09-25T10:23:30.8791903Z Get:34 http://azure.archive.ubuntu.com/ubuntu noble-security/restricted amd64 Components [212 B]
2025-09-25T10:23:30.8802289Z Get:35 http://azure.archive.ubuntu.com/ubuntu noble-security/multiverse amd64 Components [212 B]
2025-09-25T10:23:41.6358792Z Fetched 11.3 MB in 1s (7782 kB/s)
2025-09-25T10:23:42.3601092Z Reading package lists...
2025-09-25T10:23:42.4153073Z Reading package lists...
2025-09-25T10:23:42.5893740Z Building dependency tree...
2025-09-25T10:23:42.5901610Z Reading state information...
2025-09-25T10:23:42.7309516Z The following additional packages will be installed:
2025-09-25T10:23:42.7312698Z   gdb libbabeltrace1 libc6-dbg libdebuginfod-common libdebuginfod1t64 libipt2
2025-09-25T10:23:42.7315212Z   libsource-highlight-common libsource-highlight4t64
2025-09-25T10:23:42.7320686Z Suggested packages:
2025-09-25T10:23:42.7321191Z   gdb-doc gdbserver valgrind-dbg valgrind-mpi kcachegrind alleyoop valkyrie
2025-09-25T10:23:42.7784268Z The following NEW packages will be installed:
2025-09-25T10:23:42.7788095Z   gdb libbabeltrace1 libc6-dbg libdebuginfod-common libdebuginfod1t64 libipt2
2025-09-25T10:23:42.7791336Z   libsource-highlight-common libsource-highlight4t64 valgrind
2025-09-25T10:23:42.8417933Z 0 upgraded, 9 newly installed, 0 to remove and 16 not upgraded.
2025-09-25T10:23:42.8418682Z Need to get 12.0 MB/23.4 MB of archives.
2025-09-25T10:23:42.8419420Z After this operation, 101 MB of additional disk space will be used.
2025-09-25T10:23:42.8420407Z Get:1 /tmp/valgrind-codspeed.deb valgrind amd64 1:3.24.0-0codspeed1 [11.4 MB]
2025-09-25T10:23:42.8421241Z Get:2 file:/etc/apt/apt-mirrors.txt Mirrorlist [144 B]
2025-09-25T10:23:42.9341302Z Get:3 http://azure.archive.ubuntu.com/ubuntu noble-updates/main amd64 libdebuginfod-common all 0.190-1.1ubuntu0.1 [14.6 kB]
2025-09-25T10:23:42.9992874Z Get:4 http://azure.archive.ubuntu.com/ubuntu noble/main amd64 libbabeltrace1 amd64 1.5.11-3build3 [164 kB]
2025-09-25T10:23:43.0649093Z Get:5 http://azure.archive.ubuntu.com/ubuntu noble-updates/main amd64 libdebuginfod1t64 amd64 0.190-1.1ubuntu0.1 [17.1 kB]
2025-09-25T10:23:43.1284887Z Get:6 http://azure.archive.ubuntu.com/ubuntu noble/main amd64 libipt2 amd64 2.0.6-1build1 [45.7 kB]
2025-09-25T10:23:43.1931933Z Get:7 http://azure.archive.ubuntu.com/ubuntu noble/main amd64 libsource-highlight-common all 3.1.9-4.3build1 [64.2 kB]
2025-09-25T10:23:43.2571316Z Get:8 http://azure.archive.ubuntu.com/ubuntu noble/main amd64 libsource-highlight4t64 amd64 3.1.9-4.3build1 [258 kB]
2025-09-25T10:23:43.3252483Z Get:9 http://azure.archive.ubuntu.com/ubuntu noble/main amd64 gdb amd64 15.0.50.20240403-0ubuntu1 [4010 kB]
2025-09-25T10:23:43.6492369Z Get:10 http://azure.archive.ubuntu.com/ubuntu noble-updates/main amd64 libc6-dbg amd64 2.39-0ubuntu8.6 [7458 kB]
2025-09-25T10:23:44.2529860Z Preconfiguring packages ...
2025-09-25T10:23:44.3033499Z Fetched 12.0 MB in 1s (10.2 MB/s)
2025-09-25T10:23:44.3615940Z Selecting previously unselected package libdebuginfod-common.
2025-09-25T10:23:44.4985715Z (Reading database ... 
2025-09-25T10:23:44.4986656Z (Reading database ... 5%
2025-09-25T10:23:44.4986981Z (Reading database ... 10%
2025-09-25T10:23:44.4987245Z (Reading database ... 15%
2025-09-25T10:23:44.4987666Z (Reading database ... 20%
2025-09-25T10:23:44.4988112Z (Reading database ... 25%
2025-09-25T10:23:44.4988566Z (Reading database ... 30%
2025-09-25T10:23:44.4989009Z (Reading database ... 35%
2025-09-25T10:23:44.4989460Z (Reading database ... 40%
2025-09-25T10:23:44.4989882Z (Reading database ... 45%
2025-09-25T10:23:44.4990305Z (Reading database ... 50%
2025-09-25T10:23:44.6300739Z (Reading database ... 55%
2025-09-25T10:23:45.2018504Z (Reading database ... 60%
2025-09-25T10:23:45.7869172Z (Reading database ... 65%
2025-09-25T10:23:46.3784861Z (Reading database ... 70%
2025-09-25T10:23:46.9349035Z (Reading database ... 75%
2025-09-25T10:23:47.4992306Z (Reading database ... 80%
2025-09-25T10:23:48.1200289Z (Reading database ... 85%
2025-09-25T10:23:48.7868539Z (Reading database ... 90%
2025-09-25T10:23:49.4534296Z (Reading database ... 95%
2025-09-25T10:23:49.4534849Z (Reading database ... 100%
2025-09-25T10:23:49.4535551Z (Reading database ... 214164 files and directories currently installed.)
2025-09-25T10:23:49.4580684Z Preparing to unpack .../0-libdebuginfod-common_0.190-1.1ubuntu0.1_all.deb ...
2025-09-25T10:23:49.4607056Z Unpacking libdebuginfod-common (0.190-1.1ubuntu0.1) ...
2025-09-25T10:23:49.5074126Z Selecting previously unselected package libbabeltrace1:amd64.
2025-09-25T10:23:49.5206781Z Preparing to unpack .../1-libbabeltrace1_1.5.11-3build3_amd64.deb ...
2025-09-25T10:23:49.5274773Z Unpacking libbabeltrace1:amd64 (1.5.11-3build3) ...
2025-09-25T10:23:49.5618781Z Selecting previously unselected package libdebuginfod1t64:amd64.
2025-09-25T10:23:49.5750475Z Preparing to unpack .../2-libdebuginfod1t64_0.190-1.1ubuntu0.1_amd64.deb ...
2025-09-25T10:23:49.5762431Z Unpacking libdebuginfod1t64:amd64 (0.190-1.1ubuntu0.1) ...
2025-09-25T10:23:49.6038809Z Selecting previously unselected package libipt2.
2025-09-25T10:23:49.6171940Z Preparing to unpack .../3-libipt2_2.0.6-1build1_amd64.deb ...
2025-09-25T10:23:49.6183398Z Unpacking libipt2 (2.0.6-1build1) ...
2025-09-25T10:23:49.7079531Z Selecting previously unselected package libsource-highlight-common.
2025-09-25T10:23:49.7211838Z Preparing to unpack .../4-libsource-highlight-common_3.1.9-4.3build1_all.deb ...
2025-09-25T10:23:49.8713296Z Unpacking libsource-highlight-common (3.1.9-4.3build1) ...
2025-09-25T10:23:51.5117218Z Selecting previously unselected package libsource-highlight4t64:amd64.
2025-09-25T10:23:51.5253074Z Preparing to unpack .../5-libsource-highlight4t64_3.1.9-4.3build1_amd64.deb ...
2025-09-25T10:23:51.5265483Z Unpacking libsource-highlight4t64:amd64 (3.1.9-4.3build1) ...
2025-09-25T10:23:51.5568670Z Selecting previously unselected package gdb.
2025-09-25T10:23:51.5699537Z Preparing to unpack .../6-gdb_15.0.50.20240403-0ubuntu1_amd64.deb ...
2025-09-25T10:23:51.5711334Z Unpacking gdb (15.0.50.20240403-0ubuntu1) ...
2025-09-25T10:23:51.7272917Z Selecting previously unselected package libc6-dbg:amd64.
2025-09-25T10:23:51.7403798Z Preparing to unpack .../7-libc6-dbg_2.39-0ubuntu8.6_amd64.deb ...
2025-09-25T10:23:51.7416151Z Unpacking libc6-dbg:amd64 (2.39-0ubuntu8.6) ...
2025-09-25T10:23:51.8479505Z Selecting previously unselected package valgrind.
2025-09-25T10:23:51.8614152Z Preparing to unpack .../8-valgrind-codspeed.deb ...
2025-09-25T10:23:51.8630906Z Unpacking valgrind (1:3.24.0-0codspeed1) ...
2025-09-25T10:23:52.2917088Z Setting up libdebuginfod-common (0.190-1.1ubuntu0.1) ...
2025-09-25T10:23:52.3899734Z Setting up libsource-highlight-common (3.1.9-4.3build1) ...
2025-09-25T10:23:52.3938329Z Setting up libc6-dbg:amd64 (2.39-0ubuntu8.6) ...
2025-09-25T10:23:52.3975625Z Setting up libipt2 (2.0.6-1build1) ...
2025-09-25T10:23:52.4011784Z Setting up libbabeltrace1:amd64 (1.5.11-3build3) ...
2025-09-25T10:23:52.4047386Z Setting up valgrind (1:3.24.0-0codspeed1) ...
2025-09-25T10:23:52.4083607Z Setting up libdebuginfod1t64:amd64 (0.190-1.1ubuntu0.1) ...
2025-09-25T10:23:52.4120888Z Setting up libsource-highlight4t64:amd64 (3.1.9-4.3build1) ...
2025-09-25T10:23:52.4157277Z Setting up gdb (15.0.50.20240403-0ubuntu1) ...
2025-09-25T10:23:52.4201976Z Processing triggers for man-db (2.12.0-4build2) ...
2025-09-25T10:25:13.3520833Z Processing triggers for libc-bin (2.39-0ubuntu8.6) ...
2025-09-25T10:25:16.6585520Z 
2025-09-25T10:25:16.6587755Z Running kernel seems to be up-to-date.
2025-09-25T10:25:16.6588117Z 
2025-09-25T10:25:16.6588250Z Restarting services...
2025-09-25T10:25:16.6657965Z 
2025-09-25T10:25:16.6658459Z Service restarts being deferred:
2025-09-25T10:25:16.6661185Z  systemctl restart hosted-compute-agent.service
2025-09-25T10:25:16.6661455Z 
2025-09-25T10:25:16.6661835Z No containers need to be restarted.
2025-09-25T10:25:16.6662322Z 
2025-09-25T10:25:16.6662521Z No user sessions are running outdated binaries.
2025-09-25T10:25:16.6662849Z 
2025-09-25T10:25:16.6663143Z No VM guests are running outdated hypervisor (qemu) binaries on this host.

10:25:16 - 10:23:29 = 1 minute and 47 seconds.

runner/src/run/runner/valgrind/setup.rs

Lines 57 to 83 in 65a2980

	pub async fn install_valgrind(system_info: &SystemInfo) -> Result<()> {
	if is_valgrind_installed() {
	info!("Valgrind is already installed with the correct version, skipping installation");
	return Ok(());
	}
	debug!("Installing valgrind");
	let valgrind_deb_url = format!(
	"https://github.com/CodSpeedHQ/valgrind-codspeed/releases/download/{}/{}",
	VALGRIND_CODSPEED_DEB_VERSION.as_str(),
	get_codspeed_valgrind_filename(system_info)?
	);
	let deb_path = env::temp_dir().join("valgrind-codspeed.deb");
	download_file(&Url::parse(valgrind_deb_url.as_str()).unwrap(), &deb_path).await?;
	run_with_sudo(&["apt-get", "update"])?;
	run_with_sudo(&[
	"apt-get",
	"install",
	"--allow-downgrades",
	"-y",
	deb_path.to_str().unwrap(),
	])?;
	info!("Valgrind installation completed successfully");
	Ok(())
	}

[GuillaumeLagrange]

Hello! Thank you for the issue.

The snippet you linked actually installs perf in order to do walltime benchmark profiling and not valgrind.
On codspeed macro runners, perf comes pre-installed, this snippet is more a failsafe.

What you mention is still an issue for instrumentation runs though, and has been slightly discussed on Discord recently, but valgrind installation happens here

runner/src/run/runner/valgrind/setup.rs

Lines 57 to 83 in 65a2980

	pub async fn install_valgrind(system_info: &SystemInfo) -> Result<()> {
	if is_valgrind_installed() {
	info!("Valgrind is already installed with the correct version, skipping installation");
	return Ok(());
	}
	debug!("Installing valgrind");
	let valgrind_deb_url = format!(
	"https://github.com/CodSpeedHQ/valgrind-codspeed/releases/download/{}/{}",
	VALGRIND_CODSPEED_DEB_VERSION.as_str(),
	get_codspeed_valgrind_filename(system_info)?
	);
	let deb_path = env::temp_dir().join("valgrind-codspeed.deb");
	download_file(&Url::parse(valgrind_deb_url.as_str()).unwrap(), &deb_path).await?;
	run_with_sudo(&["apt-get", "update"])?;
	run_with_sudo(&[
	"apt-get",
	"install",
	"--allow-downgrades",
	"-y",
	deb_path.to_str().unwrap(),
	])?;
	info!("Valgrind installation completed successfully");
	Ok(())
	}

There is definite room for improvement to cache perf and/or valgrind. But this is maybe something to do directly in https://github.com/CodSpeedHQ/action and/or as a documentation snippet on how to cache tooling.

We can keep the issue here to track potential ideas and help guide people that would like to hack at it.

[Boshen]

Oops linked the wrong code. I've updated my issue description.

---

I've figured it out, it's in Processing triggers for man-db

2025-09-25T10:23:52.4201976Z Processing triggers for man-db (2.12.0-4build2) ...
2025-09-25T10:25:13.3520833Z Processing triggers for libc-bin (2.39-0ubuntu8.6) ...

The solution is actions/runner-images#10977 (comment)

# Skip installing pacakge docs {makes the man-db trigger much faster) 
# (I disabled `/doc` and `/info` too, just in case.)
sudo tee /etc/dpkg/dpkg.cfg.d/01_nodoc > /dev/null << 'EOF'
path-exclude /usr/share/doc/*
path-exclude /usr/share/man/*
path-exclude /usr/share/info/*
EOF

[GuillaumeLagrange]

Hey there, we've just merged #129 which makes the codspeed runner cache the tools it installs.

Once it's released and CodSpeedHQ/action#140 is merged, the action will automatically use this feature from the runner.

We've decided not to directly implement the fix from @Boshen with man db triggers, as we don't want the runner to be responsible for modifying the host machine configuration, as the runner is not destined to be only run in CI, but the fix is still valid if one does not want to use codspeed runner cache or just wants to speed up installation when a cache miss occurs