CloudShovel is a tool designed to search for sensitive information within public Amazon Machine Images (AMIs). It automates the process of launching instances from target AMIs, mounting their volumes, and scanning for potential secrets or sensitive data.
Table of Contents:
- Introduction
- Prerequisites
- Installation
- Usage
- How It Works
- Resources Created
- Required Permissions
- Cleaning Up
- Troubleshooting
Before using CloudShovel, ensure you have the following:
- Python 3.6 or higher
- Access to your own AWS account and IAM identity
- Python libraries (can be installed automatically)
- Boto3 library installed (
pip install boto3
) - Colorama library installed (
pip install colorama
)
- Boto3 library installed (
From terminal:
python3 -m pip install cloudshovel
-
Clone the CloudShovel repository:
git clone https://github.com/your-repo/cloudshovel.git cd cloudshovel
-
Install the required Python libraries:
pip install -r requirements.txt
To use CloudShovel, run the main.py
script with the following syntax:
cloudshovel <ami_id> --bucket <s3_bucket_name> [--profile <aws_profile> | --access-key <access_key> --secret-key <secret_key> (--session-token <session_token>)] [--region <aws_region>]
Arguments:
ami_id
: The ID of the AMI you want to scan (required)--bucket
: The name of the S3 bucket to store results (required)- Authentication with configured AWS CLI profile
--profile
: AWS CLI profile name (default is 'default')
- Authentication with access keys
--access-key
: AWS Access Key ID--secret-key
: AWS Secret Access Key--session-token
: AWS Session Token (optional)
--region
: AWS region (default is 'us-east-1')
If you don't specify an argument for authentication, the tool will try to automatically use the default
profile.
Example:
cloudshovel ami-1234567890abcdef --bucket my-cloudshovel-results --profile my-aws-profile --region us-west-2
CloudShovel operates through the following steps:
-
Initialization:
- Parses command-line arguments and creates an AWS session.
- Validates the target AMI's existence.
-
Setup:
- Creates or verifies the existence of the specified S3 bucket.
- Creates an IAM role and instance profile for the "secret searcher" EC2 instance.
- Uploads necessary scripts to the S3 bucket.
-
Secret Searcher Instance:
- Launches an EC2 instance (the "secret searcher") based on the latest Amazon Linux 202* AMI.
- Installs required tools on the secret searcher instance.
-
Target AMI Processing:
- Launches an EC2 instance from the target AMI.
- Stops the instance and detaches its volumes.
- Attaches these volumes to the secret searcher instance.
-
Scanning:
- Mounts the attached volumes on the secret searcher instance.
- Executes the
mount_and_dig.sh
script to search for potential secrets. - The script looks for specific file names and patterns that might indicate sensitive information.
-
Results:
- Uploads the scanning results to the specified S3 bucket.
-
Cleanup:
- Detaches and deletes the volumes from the target AMI.
- Terminates instances and removes created IAM resources.
CloudShovel creates the following AWS resources during its operation:
- S3 Bucket: Stores scanning scripts and results.
- IAM Role and Instance Profile: Named "minimal-ssm", used by the secret searcher instance.
- EC2 Instances:
- A "secret searcher" instance based on Amazon Linux 2023.
- A temporary instance launched from the target AMI (terminated after volume detachment).
- EBS Volumes: Temporary attachments to the secret searcher instance (deleted after scanning).
To run CloudShovel, your AWS account or IAM user needs the following permissions:
- EC2:
- Describe, run, stop, and terminate instances
- Describe, create, attach, detach, and delete volumes
- Describe and create tags
- IAM:
- Create, delete, and manage roles and instance profiles
- Attach and detach role policies
- S3:
- Create buckets
- Put, get, and delete objects
- SSM:
- Send commands to EC2 instances
- Get command invocation results
It's recommended to use the principle of least privilege and create a specific IAM user or role for running CloudShovel with only the necessary permissions.
CloudShovel attempts to clean up all created resources after completion or in case of errors. However, it's good practice to verify that all resources have been properly removed, especially:
- Check the EC2 console for any running instances tagged with "usage: CloudQuarry" or "usage: SecretSearcher".
- Verify that the IAM role and instance profile "minimal-ssm" have been deleted.
- The S3 bucket is not automatically deleted to preserve results. Delete it manually if no longer needed.
- If the script fails to mount volumes, ensure that the necessary filesystem tools (e.g., ntfs-3g for NTFS volumes) are installed on the secret searcher instance.
- For permission-related errors, verify that your AWS credentials have all the required permissions listed above.
- If experiencing issues with specific AMIs, check their requirements (e.g., virtualization type, ENA support) and adjust the instance types in the script accordingly.
Remember to use this tool responsibly and ensure you have the right to scan the AMIs you're targeting.