Stabilize automatic garbage collection. (#14287)

This proposes to stabilize automatic garbage collection of Cargo's
global cache data in the cargo home directory.

### What is being stabilized?

This PR stabilizes automatic garbage collection, which is triggered at
most once per day by default. This automatic gc will delete old, unused
files in cargo's home directory.

It will delete files that need to be downloaded from the network after 3
months, and files that can be generated without network access after 1
month. These thresholds are intended to balance the intent of reducing
cargo's disk usage versus deleting too often forcing cargo to do extra
work when files are missing.

Tracking of the last-use data is stored in a sqlite database in the
cargo home directory. Cargo updates timestamps in that database whenever
it accesses a file in the cache. This part is already stabilized.

This PR also stabilizes the `gc.auto.frequency` configuration option.
The primary use case for when a user may want to set that is to set it
to "never" to disable gc should the need arise to avoid it.

When gc is initiated, and there are files to delete, there will be a
progress bar while it is deleting them. The progress bar will disappear
when it finishes. If the user runs with `-v` verbose option, then cargo
will also display which files it deletes.

If there is an error while cleaning, cargo will only display a warning,
and otherwise continue.

### What is not being stabilized?

The manual garbage collection option (via `cargo clean gc`) is not
proposed to be stabilized at this time. That still needs some design
work. This is tracked in
#13060.

Additionally, there are several low-level config options currently
implemented which define the thresholds for when it will delete files. I
think these options are probably too low-level and specific. This is
tracked in #13061.

Garbage collection of build artifacts is not yet implemented, and
tracked in #13136.

### Background

This feature is tracked in
#12633 and was implemented in a
variety of PRs, primarily #12634.

The tests for this feature are located in
https://github.com/rust-lang/cargo/blob/master/tests/testsuite/global_cache_tracker.rs.

Cargo started tracking the last-use data on stable via
#13492 in 1.78 which was released
2024-05-02. This PR is proposing to stabilize automatic deletion in 1.82
which will be released in 2024-10-17.

### Risks

Users who frequently use versions of Rust older than 1.78 will not have
the last-use data tracking updated. If they infrequently use 1.78 or
newer, and use the same cache files, then the last-use tracking will
only be updated by the newer versions. If that time frame is more than 1
month (or 3 months for downloaded data), then cargo will delete files
that the older versions are still using. This means the next time they
run the older version, it will have to re-download or re-extract the
files.

The effects of deleting cache data in environments where cargo's cache
is modified by external tools is not fully known. For example, CI
caching systems may save and restore cargo's cache. Similarly, things
like Docker images that try to save the cache in a layer, or mount the
cache in a read-only filesystem may have undesirable interactions.

The once-a-day performance hit might be noticeable to some people. I've
been using this for several months, and almost never notice it. However,
slower systems, or situations where there is a lot of data to delete
might take a while (on the order of seconds hopefully).

Author: weihanglo

src/cargo/core/gc.rs

@@ -49,9 +49,6 @@ const DEFAULT_AUTO_FREQUENCY: &str = "1 day";
 /// It should be cheap to call this multiple times (subsequent calls are
 /// ignored), but try not to abuse that.
 pub fn auto_gc(gctx: &GlobalContext) {
-    if !gctx.cli_unstable().gc {
-        return;
-    }
     if !gctx.network_allowed() {
         // As a conservative choice, auto-gc is disabled when offline. If the
         // user is indefinitely offline, we don't want to delete things they
@@ -174,49 +171,74 @@ impl GcOpts {
         let config = gctx
             .get::<Option<GlobalCleanConfig>>("cache.global-clean")?
             .unwrap_or_default();
-        self.update_for_auto_gc_config(&config)
+        self.update_for_auto_gc_config(&config, gctx.cli_unstable().gc)
     }
 
-    fn update_for_auto_gc_config(&mut self, config: &GlobalCleanConfig) -> CargoResult<()> {
+    fn update_for_auto_gc_config(
+        &mut self,
+        config: &GlobalCleanConfig,
+        unstable_allowed: bool,
+    ) -> CargoResult<()> {
+        macro_rules! config_default {
+            ($config:expr, $field:ident, $default:expr, $unstable_allowed:expr) => {
+                if !unstable_allowed {
+                    // These config options require -Zgc
+                    $default
+                } else {
+                    $config.$field.as_deref().unwrap_or($default)
+                }
+            };
+        }
+
         self.max_src_age = newer_time_span_for_config(
             self.max_src_age,
-            "cache.global-clean.max-src-age",
-            config
-                .max_src_age
-                .as_deref()
-                .unwrap_or(DEFAULT_MAX_AGE_EXTRACTED),
+            "gc.auto.max-src-age",
+            config_default!(
+                config,
+                max_src_age,
+                DEFAULT_MAX_AGE_EXTRACTED,
+                unstable_allowed
+            ),
         )?;
         self.max_crate_age = newer_time_span_for_config(
             self.max_crate_age,
-            "cache.global-clean.max-crate-age",
-            config
-                .max_crate_age
-                .as_deref()
-                .unwrap_or(DEFAULT_MAX_AGE_DOWNLOADED),
+            "gc.auto.max-crate-age",
+            config_default!(
+                config,
+                max_crate_age,
+                DEFAULT_MAX_AGE_DOWNLOADED,
+                unstable_allowed
+            ),
         )?;
         self.max_index_age = newer_time_span_for_config(
             self.max_index_age,
-            "cache.global-clean.max-index-age",
-            config
-                .max_index_age
-                .as_deref()
-                .unwrap_or(DEFAULT_MAX_AGE_DOWNLOADED),
+            "gc.auto.max-index-age",
+            config_default!(
+                config,
+                max_index_age,
+                DEFAULT_MAX_AGE_DOWNLOADED,
+                unstable_allowed
+            ),
         )?;
         self.max_git_co_age = newer_time_span_for_config(
             self.max_git_co_age,
-            "cache.global-clean.max-git-co-age",
-            config
-                .max_git_co_age
-                .as_deref()
-                .unwrap_or(DEFAULT_MAX_AGE_EXTRACTED),
+            "gc.auto.max-git-co-age",
+            config_default!(
+                config,
+                max_git_co_age,
+                DEFAULT_MAX_AGE_EXTRACTED,
+                unstable_allowed
+            ),
         )?;
         self.max_git_db_age = newer_time_span_for_config(
             self.max_git_db_age,
-            "cache.global-clean.max-git-db-age",
-            config
-                .max_git_db_age
-                .as_deref()
-                .unwrap_or(DEFAULT_MAX_AGE_DOWNLOADED),
+            "gc.auto.max-git-db-age",
+            config_default!(
+                config,
+                max_git_db_age,
+                DEFAULT_MAX_AGE_DOWNLOADED,
+                unstable_allowed
+            ),
         )?;
         Ok(())
     }
@@ -255,9 +277,6 @@ impl<'a, 'gctx> Gc<'a, 'gctx> {
     /// This returns immediately without doing work if garbage collection has
     /// been performed recently (since `cache.auto-clean-frequency`).
     fn auto(&mut self, clean_ctx: &mut CleanContext<'gctx>) -> CargoResult<()> {
-        if !self.gctx.cli_unstable().gc {
-            return Ok(());
-        }
         let freq = self
             .gctx
             .get::<Option<String>>("cache.auto-clean-frequency")?;
@@ -274,7 +293,7 @@ impl<'a, 'gctx> Gc<'a, 'gctx> {
             .unwrap_or_default();
 
         let mut gc_opts = GcOpts::default();
-        gc_opts.update_for_auto_gc_config(&config)?;
+        gc_opts.update_for_auto_gc_config(&config, self.gctx.cli_unstable().gc)?;
         self.gc(clean_ctx, &gc_opts)?;
         if !clean_ctx.dry_run {
             self.global_cache_tracker.set_last_auto_gc()?;

src/doc/src/reference/config.md

@@ -95,6 +95,9 @@ ENV_VAR_NAME_3 = { value = "relative/path", relative = true }
 [future-incompat-report]
 frequency = 'always' # when to display a notification about a future incompat report
 
+[cache]
+auto-clean-frequency = "1 day"   # How often to perform automatic cache cleaning
+
 [cargo-new]
 vcs = "none"              # VCS to use ('git', 'hg', 'pijul', 'fossil', 'none')
 
@@ -664,6 +667,41 @@ Controls how often we display a notification to the terminal when a future incom
 * `always` (default): Always display a notification when a command (e.g. `cargo build`) produces a future incompat report
 * `never`: Never display a notification
 
+### `[cache]`
+
+The `[cache]` table defines settings for cargo's caches.
+
+#### Global caches
+
+When running `cargo` commands, Cargo will automatically track which files you are using within the global cache.
+Periodically, Cargo will delete files that have not been used for some period of time.
+It will delete files that have to be downloaded from the network if they have not been used in 3 months. Files that can be generated without network access will be deleted if they have not been used in 1 month.
+
+The automatic deletion of files only occurs when running commands that are already doing a significant amount of work, such as all of the build commands (`cargo build`, `cargo test`, `cargo check`, etc.), and `cargo fetch`.
+
+Automatic deletion is disabled if cargo is offline such as with `--offline` or `--frozen` to avoid deleting artifacts that may need to be used if you are offline for a long period of time.
+
+> **Note**: This tracking is currently only implemented for the global cache in Cargo's home directory.
+> This includes registry indexes and source files downloaded from registries and git dependencies.
+> Support for tracking build artifacts is not yet implemented, and tracked in [cargo#13136](https://github.com/rust-lang/cargo/issues/13136).
+>
+> Additionally, there is an unstable feature to support *manually* triggering cache cleaning, and to further customize the configuration options.
+> See the [Unstable chapter](unstable.md#gc) for more information.
+
+#### `cache.auto-clean-frequency`
+* Type: string
+* Default: `"1 day"`
+* Environment: `CARGO_CACHE_AUTO_CLEAN_FREQUENCY`
+
+This option defines how often Cargo will automatically delete unused files in the global cache.
+This does *not* define how old the files must be, those thresholds are described [above](#global-caches).
+
+It supports the following settings:
+
+* `"never"` --- Never deletes old files.
+* `"always"` --- Checks to delete old files every time Cargo runs.
+* An integer followed by "seconds", "minutes", "hours", "days", "weeks", or "months" --- Checks to delete old files at most the given time frame.
+
 ### `[http]`
 
 The `[http]` table defines settings for HTTP behavior. This includes fetching

src/doc/src/reference/environment-variables.md

@@ -99,6 +99,7 @@ In summary, the supported environment variables are:
 * `CARGO_BUILD_RUSTDOCFLAGS` --- Extra `rustdoc` flags, see [`build.rustdocflags`].
 * `CARGO_BUILD_INCREMENTAL` --- Incremental compilation, see [`build.incremental`].
 * `CARGO_BUILD_DEP_INFO_BASEDIR` --- Dep-info relative directory, see [`build.dep-info-basedir`].
+* `CARGO_CACHE_AUTO_CLEAN_FREQUENCY` --- Configures how often automatic cache cleaning runs, see [`cache.auto-clean-frequency`].
 * `CARGO_CARGO_NEW_VCS` --- The default source control system with [`cargo new`], see [`cargo-new.vcs`].
 * `CARGO_FUTURE_INCOMPAT_REPORT_FREQUENCY` --- How often we should generate a future incompat report notification, see [`future-incompat-report.frequency`].
 * `CARGO_HTTP_DEBUG` --- Enables HTTP debugging, see [`http.debug`].
@@ -163,6 +164,7 @@ In summary, the supported environment variables are:
 [`build.incremental`]: config.md#buildincremental
 [`build.dep-info-basedir`]: config.md#builddep-info-basedir
 [`doc.browser`]: config.md#docbrowser
+[`cache.auto-clean-frequency`]: config.md#cacheauto-clean-frequency
 [`cargo-new.name`]: config.md#cargo-newname
 [`cargo-new.email`]: config.md#cargo-newemail
 [`cargo-new.vcs`]: config.md#cargo-newvcs

src/doc/src/reference/unstable.md

@@ -1587,37 +1587,16 @@ This will not affect any hard-coded paths in the source code, such as in strings
 
 * Tracking Issue: [#12633](https://github.com/rust-lang/cargo/issues/12633)
 
-The `-Zgc` flag enables garbage-collection within cargo's global cache within the cargo home directory.
-This includes downloaded dependencies such as compressed `.crate` files, extracted `src` directories, registry index caches, and git dependencies.
-When `-Zgc` is present, cargo will track the last time any index and dependency was used,
-and then uses those timestamps to manually or automatically delete cache entries that have not been used for a while.
-
-```sh
-cargo build -Zgc
-```
-
-### Automatic garbage collection
-
-Automatic deletion happens on commands that are already doing a significant amount of work,
-such as all of the build commands (`cargo build`, `cargo test`, `cargo check`, etc.), and `cargo fetch`.
-The deletion happens just after resolution and packages have been downloaded.
-Automatic deletion is only done once per day (see `cache.auto-clean-frequency` to configure).
-Automatic deletion is disabled if cargo is offline such as with `--offline` or `--frozen` to avoid deleting artifacts that may need to be used if you are offline for a long period of time.
+The `-Zgc` flag is used to enable certain features related to garbage-collection of cargo's global cache within the cargo home directory.
 
 #### Automatic gc configuration
 
-The automatic gc behavior can be specified via a cargo configuration setting.
+The `-Zgc` flag will enable Cargo to read extra configuration options related to garbage collection.
 The settings available are:
 
 ```toml
 # Example config.toml file.
 
-# This table defines settings for cargo's caches.
-[cache]
-# The maximum frequency that automatic cleaning of the cache happens.
-# Can be "never" to disable, or "always" to run on every command.
-auto-clean-frequency = "1 day"
-
 # Sub-table for defining specific settings for cleaning the global cache.
 [cache.global-clean]
 # Anything older than this duration will be deleted in the source cache.
@@ -1632,9 +1611,13 @@ max-git-co-age = "1 month"
 max-git-db-age = "3 months"
 ```
 
+Note that the [`cache.auto-clean-frequency`] option was stabilized in Rust 1.88.
+
+[`cache.auto-clean-frequency`]: config.md#cacheauto-clean-frequency
+
 ### Manual garbage collection with `cargo clean`
 
-Manual deletion can be done with the `cargo clean gc` command.
+Manual deletion can be done with the `cargo clean gc -Zgc` command.
 Deletion of cache contents can be performed by passing one of the cache options:
 
 - `--max-src-age=DURATION` --- Deletes source cache files that have not been used since the given age.
@@ -1653,9 +1636,9 @@ A DURATION is specified in the form "N seconds/minutes/days/weeks/months" where
 A SIZE is specified in the form "N *suffix*" where *suffix* is B, kB, MB, GB, kiB, MiB, or GiB, and N is an integer or floating point number. If no suffix is specified, the number is the number of bytes.
 
 ```sh
-cargo clean gc
-cargo clean gc --max-download-age=1week
-cargo clean gc --max-git-size=0 --max-download-size=100MB
+cargo clean gc -Zgc
+cargo clean gc -Zgc --max-download-age=1week
+cargo clean gc -Zgc --max-git-size=0 --max-download-size=100MB
 ```
 
 ## open-namespaces
@@ -2166,3 +2149,7 @@ The 2024 edition has been stabilized in the 1.85 release.
 See the [`edition` field](manifest.md#the-edition-field) for more information on setting the edition.
 See [`cargo fix --edition`](../commands/cargo-fix.md) and [The Edition Guide](../../edition-guide/index.html) for more information on migrating existing projects.
 
+## Automatic garbage collection
+
+Support for automatically deleting old files was stabilized in Rust 1.88.
+More information can be found in the [config chapter](config.md#cache).