S3 Lifecycle Checker

A utility tool that scans your AWS S3 buckets to identify versioning-enabled buckets without proper lifecycle policies for managing old object versions and incomplete multipart uploads.

Disclaimer

This Transcribe Concurrency Monitoring Solution is provided "as is" without warranties of any kind, either express or implied. Amazon Web Services (AWS) and its affiliates make no representations or warranties regarding the accuracy, reliability, or performance of this solution.

By using this solution, you acknowledge and agree that AWS shall not be liable for any direct, indirect, incidental, special, consequential, or exemplary damages, including but not limited to damages for loss of profits, goodwill, use, data, or other intangible losses resulting from the use or inability to use this solution.

The cost estimates provided are approximations only and actual costs may vary based on your specific usage patterns, AWS region, and other factors. You are solely responsible for monitoring and managing your AWS costs.

This solution is not an official AWS product and is not covered by AWS Support. For assistance with this solution, please refer to community resources or engage AWS Professional Services.

Problem Statement

S3 buckets with versioning enabled but without lifecycle policies can lead to:

• Unexpected storage costs from old object versions
• Accumulation of unnecessary data
• Orphaned incomplete multipart uploads consuming storage
• Potential compliance issues
• Increased AWS bill with no operational benefit

Features

• Automatically scans all S3 buckets in your AWS account
• Identifies buckets with versioning enabled
• Checks for the presence of lifecycle policies
• Verifies if policies include rules to delete old object versions
• Checks for rules to clean up incomplete multipart uploads
• Generates a detailed report of problematic buckets
• Provides example lifecycle policy configuration

Why This Matters

Old Object Versions

When versioning is enabled on an S3 bucket, every update to an object creates a new version. Without lifecycle policies to expire old versions, these accumulate indefinitely, leading to:

• Storage costs for data you may no longer need
• Increased complexity when listing objects
• Potential performance impacts for bucket operations

Incomplete Multipart Uploads

Multipart uploads that are initiated but never completed leave orphaned parts in your bucket that:

• Incur storage costs just like complete objects
• Are not visible in the standard object listing
• Can only be cleaned up with specific lifecycle rules

Requirements

• Python 3.13+
• AWS credentials configured (via AWS CLI, environment variables, or IAM role)
• Required Python packages:
  • boto3
  • tabulate

Installation

# Clone the repository
git clone https://github.com/yourusername/s3-lifecycle-checker.git
cd s3-lifecycle-checker

# Set up a virtual environment
python -m venv .venv
source .venv/bin/activate  # On Windows: .venv\Scripts\activate

# Install dependencies
pip install -e .

Usage

Simply run the script:

./main.py

The script will:

1. Connect to your AWS account
2. List all S3 buckets
3. Check versioning status and lifecycle policies
4. Generate a report of buckets that need attention

AWS Permissions Required

The script requires the following AWS permissions:

• s3:ListAllMyBuckets
• s3:GetBucketVersioning
• s3:GetBucketLifecycleConfiguration

Sample Output

=== S3 BUCKETS WITH LIFECYCLE ISSUES ===
Found 3 buckets with potential issues out of 10 total buckets.
+-------------------+---------------------+-----------------------------+
| Bucket Name       | Has Lifecycle Policy| Issue                       |
+===================+=====================+=============================+
| my-data-bucket    | No                  | No lifecycle policy         |
+-------------------+---------------------+-----------------------------+
| backup-bucket-2   | Yes                 | Missing policy to delete    |
|                   |                     | old versions; No policy to  |
|                   |                     | clean up incomplete         |
|                   |                     | multipart uploads           |
+-------------------+---------------------+-----------------------------+
| logs-archive      | No                  | No lifecycle policy         |
+-------------------+---------------------+-----------------------------+

RECOMMENDATION:
Consider adding lifecycle rules to manage old versions and incomplete multipart uploads for the buckets listed above.
Example policy to add:
{
    "Rules": [
        {
            "ID": "ManageObjectLifecycle",
            "Status": "Enabled",
            "Filter": {},
            "NoncurrentVersionExpiration": {
                "NoncurrentDays": 30
            },
            "AbortIncompleteMultipartUpload": {
                "DaysAfterInitiation": 7
            }
        }
    ]
}

Best Practices

• Review the recommended lifecycle policy and adjust the retention period (NoncurrentDays) based on your requirements
• Consider setting NoncurrentDays to a value that balances recovery needs with cost optimization
• The default recommendation of 30 days for old versions provides a reasonable recovery window
• For incomplete multipart uploads, 7 days is typically sufficient for most workloads
• For critical buckets, consider implementing these policies with longer retention periods
• Regularly run this tool to ensure all buckets maintain proper configurations

Cost Impact

Implementing proper lifecycle policies can significantly reduce your S3 storage costs:

• Old Versions: Each version of an object consumes storage and is billed at the same rate as the current version
• Incomplete Multipart Uploads: Parts from incomplete uploads are billed at the same rate as complete objects
• Example: A 1GB file with 10 versions and 2 incomplete uploads could cost 12x more than necessary