Support bundle: automatic deletion to maintain free dataset buffer

[smklein]

Summary

Implements automatic deletion of support bundles to maintain a buffer of free debug datasets for new allocations.

• Config: Added support_bundle_config table with target_free_percent and min_keep_percent
• DB Query: Added support_bundle_auto_delete() CTE + unit tests
• Background Task: Added auto_delete_bundles() phase before existing phases
• Schema: Added index on (state, time_created) for efficient queries
• omdb: Added nexus support-bundle-config show/set commands

How it works

The SupportBundleCollector background task now runs three phases:

1. Auto-delete (new): Marks oldest Active bundles as Destroying when free datasets < target
2. Cleanup (existing): Cleans up storage and DB for Destroying bundles
3. Collect (existing): Collects pending bundles

Configuration uses percentages of total datasets (calculated with CEIL rounding)

• target_free_percent: Target % of datasets to keep free for future bundles (default: 10%)
• min_keep_percent: Minimum % of datasets to retain (default: 10%)

Tests

• Deletion triggers only when free_datasets < target threshold
• Oldest bundles (by time_created) are selected first
• min_keep_percent constraint limits maximum deletions
• Only Active bundles are candidates; Collecting/Destroying still occupy datasets
• Failed bundles don't count toward used dataset count
• SQL validity via EXPLAIN + expectorate for change detection
• Integration test in test_support_bundle_auto_deletion()

Fixes #9660

[hawkw]

I had a bunch of very insignificant style nits, but overall, I like the approach used here.

[hawkw]

super unimportant nit: this will currently produce output formatted like:

    Support Bundle Auto-Deletion Report:
      Total debug datasets: 16
      Active bundles: 2
      Free datasets: 14
      Bundles marked for deletion: 0

meanwhile, a lot of the existing background task status commands will align all the numbers in output like this into a column. it might be nice if this output also did that, like this:

    Support Bundle Auto-Deletion Report:
      Total debug datasets:        16
      Active bundles:              2
      Free datasets:               14
      Bundles marked for deletion: 0

[hawkw]

also, I might consider changing this to nest "Free datasets" under "Total debug datasets", since that's a subset, and similarly, put "Bundles marked for deletion" under "Active bundles"?

[hawkw]

turbo nit: mayhaps this should just be

Suggested change

	//! omdb commands for support bundle auto-deletion configuration
	//! omdb commands for support bundle configuration

since it's currently only configuring auto-completion but it's entirely possible that other configs might be set through this later?

[hawkw]

i wondered if, perhaps, we ought to have some code in here that validates whether the new values are in the range 0-100, but i see that there are CHECK constraints for that in the db schema, and those will fail when we try to set the new values. do we think that the error that bubbles up from the database query will be clear enough to the user in that case, or should we be pre-validating the values here?

if we decide to just rely on the CHECK constraints for this, maybe we should leave a comment here saying that we don't need to validate the desired percentages now since they get CHECKed when we try to go set them?

[hawkw]

personally, i feel like this would read a bit better if it destructured the tuple rather than using indices, like:

Suggested change

	// Return type: (total_datasets, used_datasets, active_bundles, deleted_ids)
	let result: (i64, i64, i64, Vec<Uuid>) = query
	.get_result_async(&*conn)
	.await
	.map_err(|e| public_error_from_diesel(e, ErrorHandler::Server))?;
	let total_datasets = result.0 as usize;
	let used_datasets = result.1 as usize;
	let active_bundles = result.2 as usize;
	let deleted_ids: Vec<SupportBundleUuid> = result
	.3
	.into_iter()
	.map(SupportBundleUuid::from_untyped_uuid)
	.collect();
	Ok(AutoDeletionResult {
	deleted_ids,
	total_datasets,
	active_bundles,
	let (total_datasets, used_datasets, active_bundles, deleted_ids): (i64, i64, i64, Vec<Uuid>) = query
	.get_result_async(&*conn)
	.await
	.map_err(|e| public_error_from_diesel(e, ErrorHandler::Server))?;
	let deleted_ids: Vec<SupportBundleUuid> = deleted_ids
	.into_iter()
	.map(SupportBundleUuid::from_untyped_uuid)
	.collect();
	Ok(AutoDeletionResult {
	deleted_ids,
	total_datasets: total_datasets as usize,
	active_bundles: active_bundles as usize,

[hawkw]

won't the CHECK constraint also do this for us? or maybe it gives us a nicer looking error message...

[hawkw]

would really like it if this comment was wrapped at column 80 (sorry)

[hawkw]

another line that i think ought to be wrapped

[hawkw]

nitpick: can we wrap some of these lines? perhaps:

Suggested change

	CREATE INDEX IF NOT EXISTS lookup_bundle_by_state_and_creation ON omicron.public.support_bundle (
	state,
	time_created
	);
	/*
	* Support Bundle Config
	*
	* Configuration for automatic support bundle deletion. This table uses a
	* singleton pattern (exactly one row) to store cluster-wide configuration.
	*/
	CREATE TABLE IF NOT EXISTS omicron.public.support_bundle_config (
	-- Singleton pattern: only one row allowed
	singleton BOOL PRIMARY KEY DEFAULT TRUE CHECK (singleton = TRUE),
	-- Percentage (0-100) of total datasets to keep free for new allocations.
	-- Calculated as CEIL(total_datasets * target_free_percent / 100).
	-- Example: 10% of 100 datasets = 10 free, 10% of 5 datasets = 1 free.
	target_free_percent INT8 NOT NULL
	CHECK (target_free_percent >= 0 AND target_free_percent <= 100),
	-- Percentage (0-100) of total datasets to retain as bundles (minimum).
	-- Calculated as CEIL(total_datasets * min_keep_percent / 100).
	-- Prevents aggressive cleanup on small systems.
	min_keep_percent INT8 NOT NULL
	CHECK (min_keep_percent >= 0 AND min_keep_percent <= 100),
	time_modified TIMESTAMPTZ NOT NULL DEFAULT NOW()
	);
	-- Default: 10% free datasets, keep at least 10% worth of bundles
	INSERT INTO omicron.public.support_bundle_config (singleton, target_free_percent, min_keep_percent, time_modified)
	VALUES (TRUE, 10, 10, NOW())
	ON CONFLICT (singleton) DO NOTHING;
	CREATE INDEX IF NOT EXISTS
	lookup_bundle_by_state_and_creation
	ON omicron.public.support_bundle (
	state,
	time_created
	);
	/*
	* Support Bundle Config
	*
	* Configuration for automatic support bundle deletion. This table uses a
	* singleton pattern (exactly one row) to store cluster-wide configuration.
	*/
	CREATE TABLE IF NOT EXISTS omicron.public.support_bundle_config (
	-- Singleton pattern: only one row allowed
	singleton BOOL PRIMARY KEY DEFAULT TRUE CHECK (singleton = TRUE),
	-- Percentage (0-100) of total datasets to keep free for new allocations.
	-- Calculated as CEIL(total_datasets * target_free_percent / 100).
	-- Example: 10% of 100 datasets = 10 free, 10% of 5 datasets = 1 free.
	target_free_percent INT8 NOT NULL
	CHECK (target_free_percent >= 0 AND target_free_percent <= 100),
	-- Percentage (0-100) of total datasets to retain as bundles (minimum).
	-- Calculated as CEIL(total_datasets * min_keep_percent / 100).
	-- Prevents aggressive cleanup on small systems.
	min_keep_percent INT8 NOT NULL
	CHECK (min_keep_percent >= 0 AND min_keep_percent <= 100),
	time_modified TIMESTAMPTZ NOT NULL DEFAULT NOW()
	);
	-- Default: 10% free datasets, keep at least 10% worth of bundles
	INSERT INTO omicron.public.support_bundle_config (
	singleton,
	target_free_percent,
	min_keep_percent,
	time_modified
	)
	VALUES (TRUE, 10, 10, NOW())
	ON CONFLICT (singleton) DO NOTHING;

[hawkw]

mind line-wrapping this?

[hawkw]

...and wrapping this?