Retention policies don't work with S3/Azure storage backends

[xe-nvdk]

Summary

Retention policies are currently implemented using filesystem-only operations and do not work when S3 or Azure is configured as the primary storage backend.

Current Behavior

The deleteOldFiles() function in internal/api/retention.go uses:

• filepath.Walk() - filesystem only
• os.Stat() - filesystem only
• os.Remove() - filesystem only
• storage.GetLocalBasePath() - returns empty string for S3/Azure

When using S3 or Azure as the primary backend, retention policies silently do nothing.

Expected Behavior

Retention policies should work with all storage backends:

• Local filesystem (current - works)
• S3 (not implemented)
• Azure Blob Storage (not implemented)

Implementation Notes

Refactor deleteOldFiles() to use the storage backend interface:

1. Use h.storage.List() instead of filepath.Walk()
2. Use h.storage.Delete() instead of os.Remove()
3. For reading parquet metadata, DuckDB can read directly from S3/Azure paths

Approach

// Instead of:
err := filepath.Walk(measurementPath, ...)
os.Remove(filePath)

// Use:
files, err := h.storage.List(ctx, database+"/"+measurement+"/")
h.storage.Delete(ctx, filePath)

Considerations

• S3/Azure paths work with DuckDB read_parquet() if credentials are configured
• May need to batch deletions for large datasets
• Consider adding progress logging for long-running deletions

Files to Modify

• internal/api/retention.go - Refactor deleteOldFiles() to use storage backend interface

Related

• Retention policies should apply to cold tier storage #168 - Retention policies should apply to cold tier storage
• Part of making Arc fully cloud-native