Garbage collect whole `target/`

[epage]

Note: for cleaning individual build units within a build-dir, see #6229

Problem

With "cargo script", the target directory is "hidden" from the user, making it easy to leak when you delete your script.

If we move forward with rust-lang/rfcs#3371, a similar situation will happen for regular packages.

If I haven't touched a project in a long while but have run rustup update, there might be nothing of use left in target/, wasting space.

Sometimes I want to cargo clean all projects on my system (see #11305).

Proposed Solution

We should track in the GC data base a list of

• Root-manifests (ie the Cargo.toml / cargo script associated with the target directory)
• Target directory
• (maybe) The path to the Cargo.lock for future potential work like Pin cache entries still in use #13137 without having to infer the Cargo.lock (special logic needed for cargo-script, feature requests exist for even weirder situations)

Note that neither of the two fields can serve as a unique / primary key. If people use CARGO_TARGET_DIR=/tmp/cargo then multiple workspaces may point to the same target dir. Likewise, people may end up with multiple target dirs for one workspace.

We need to track the Cargo.toml / cargo script because the workspace root is ambiguous when it comes to cargo scripts.

Example entries for CARGO_TARGET_DIR=/tmp/cargo :

id	workspace-manifest	target dir	timestamp
?	/foo/Cargo.toml	/tmp/cargo	?
?	/bar/Cargo.toml	/tmp/cargo	?
?	/baz/script.rs	/tmp/cargo	?

Example entries for rust-analyzer target dir

id	workspace-manifest	target dir	timestamp
?	/foo/Cargo.toml	/foo/target	?
?	/foo/Cargo.toml	/foo/target-ra	?
?	/bar/Cargo.toml	/bar/target	?
?	/bar/Cargo.toml	/bar/target-ra	?
?	/baz/script.rs	~/.cargo/target/...	?

Forms of cleanup

• Delete target/ if unused for X time (this is in the "locally recreatable" category)
• Delete all target/ (I just upgraded Rust, maybe rustup could suggest this)
• Delete leaked target/ (workspace doesn't exist)
  • However, it might be transient (on a thumb drive). Should we make this time based?

Notes

No response

[baby230211]

@rustbot claim

[baby230211]

Three ways for cargo clean script

1. clean target dir that workspace doesn't used for specific time.
2. clean target dir that workspace doesn't used anymore.
3. clean target dir in all workspace.

[epage]

To clarify, those are use cases for why tracking of whole target/ could be useful.

I'd do a small tweak of wording in case this leads to confusion

• clean target dir if it and/or its workspace hasn't been used for specific time
• clean target dir for workspace that is no longer present
• clean all target dirs

[weihanglo]

@epage, is this garbage collection specific to target directories under ~/.cargo/target that generated by -Zscript? From the issue title I cannot tell.

It's for every target directory.

[epage]

Currently, we do the batch-save in PackageSet::get_many, so we likely want mark-workspace-used to live in a place before the get_many call

cargo/src/cargo/core/package.rs

Line 459 in c939267

pub fn get_many(&self, ids: impl IntoIterator<Item = PackageId>) -> CargoResult<Vec<&Package>> {

get_many gets called as part of PackageSet::download_accessible

cargo/src/cargo/core/package.rs

Line 540 in c939267

self.get_many(to_download.into_iter())?;

which gets called as part of resolving:

cargo/src/cargo/ops/resolve.rs

Line 221 in c939267

pkg_set.download_accessible(

which gets called as part of create_bcx:

cargo/src/cargo/ops/cargo_compile/mod.rs

Line 267 in c939267

let resolve = ops::resolve_ws_with_opts(

So our options are

• Put mark-workspace-used in resolve (before download_accessible) so every operation gets it recorded
• Put mark-workspace-used in create_bcx before the resolve so only compiles have mark-workspace-used recorded

[epage]

A code path to model off of is

cargo/src/cargo/sources/git/source.rs

Lines 147 to 157 in c939267

	fn mark_used(&self, size: Option<u64>) -> CargoResult<()> {
	self.gctx
	.deferred_global_last_use()?
	.mark_git_checkout_used(global_cache_tracker::GitCheckout {
	encoded_git_name: self.ident,
	short_name: self.short_id.expect("update before download"),
	size,
	});
	Ok(())
	}
	}