NOTE: This is a specialized tool to help with actions to take during an audit of AWS usage.
Please see the docs for details on the commands.
Useful for clearing large S3 buckets (many millions of objects), identifying egress EBS volumes and tracking S3 spend.
$ gw-aws-audit help
NAME:
gw-aws-audit - a collection of tools to audit AWS.
USAGE:
gw-aws-audit [global options] command [command options] [arguments...]
AUTHOR:
Derek Smith <dsmith@goodwaygroup.com>
COMMANDS:
s3 S3 related commands
rds RDS related commands
ec2 EC2 related commands
sg Security Group related commands
iam IAM related commands
cw CloudWatch related commands
install-manpage Generate and install man page
version, v Print version info
help, h Shows a list of commands or help for one command
GLOBAL OPTIONS:
--help, -h show help (default: false)
COPYRIGHT:
(c) 2020 Goodway Group
Add plugin:
$ asdf plugin-add gw-aws-audit https://github.com/GoodwayGroup/asdf-gw-aws-audit.git
Install the latest version:
$ asdf install gw-aws-audit latest
Homebrew (for macOS users)
brew tap GoodwayGroup/gw-aws-audit
brew install gw-aws-audit
$ curl https://i.jpillora.com/GoodwayGroup/gw-aws-audit! | bash
The compiled docker images are maintained on GitHub Container Registry (ghcr.io). We maintain the following tags:
edge: Image that is build from the currentHEADof the main line branch.latest: Image that is built from the latest released versionx.y.z(versions): Images that are build from the tagged versions within Github.
docker pull ghcr.io/goodwaygroup/gw-aws-audit
docker run -v "$PWD":/workdir ghcr.io/goodwaygroup/gw-aws-audit --versionTo install man page:
$ gw-aws-audit install-manpage
There is a bash helper script in the repo at audit.sh. This tool is useful in running an audit across many regions at once.
$ ./audit.sh
audit.sh helper script for gw-aws-audit
Usage:
audit.sh [gw-aws-audit commands]
Examples:
> This will run the 'gw-aws-audit sg detached' command for every region in the US (default)
$ audit.sh sg detached
> This will run the 'gw-aws-audit ec2 stopped-hosts' for ONLY the us-west-2 region
$ AWS_REGION=us-west-2 audit.sh ec2 stopped-hosts
> This will run the 'gw-aws-audit ec2 stopped-hosts' for every region in the EU
$ REGION=eu audit.sh ec2 stopped-hosts
> This will run the 'gw-aws-audit cw monitoring' using a specific version of the tool.
$ BIN_PATH=./bin/gw-aws-audit audit.sh cw monitoring
Note: REGION env values (default: US):
US: us-east-1 us-east-2 us-west-1 us-west-2
EU: eu-central-1 eu-west-1 eu-west-2 eu-west-3 eu-south-1 eu-north-1
AP: ap-east-1 ap-south-1 ap-northeast-3 ap-northeast-2 ap-southeast-1 ap-southeast-2 ap-northeast-1
CH: cn-north-1 cn-northwest-1
ROW: af-south-1 me-south-1 sa-east-2
ALL: All of the above combined
You can also set AWS_REGION and that will supersede the value of REGION
✔ Have fun!
There are commands for s3, ec2, rds, sg and cw
s3
$ gw-aws-audit s3
...
COMMANDS:
add-cost-tag Add s3-cost-name to all S3 buckets
metrics Get usage metrics
clear-bucket, exterminatus Clear all Objects within a given Bucket
ec2
$ gw-aws-audit ec2
...
COMMANDS:
enhanced-monitoring Produce report of Enhanced Monitoring enabled instances
detached-volumes List detached EBS volumes and snapshot counts
stopped-hosts List stopped EC2 hosts and associated EBS volumes
rds
$ gw-aws-audit rds
...
COMMANDS:
enhanced-monitoring Produce report of Enhanced Monitoring enabled instances
cw
$ gw-aws-audit cw
...
COMMANDS:
enhanced-monitoring Produce report of Enhanced Monitoring enabled EC2 & RDS instances
$ gw-aws-audit ec2 stopped-hosts
INSTANCE ID NAME VOLUME SIZE (GB) SNAPSHOTS MIN SIZE (GB) COSTS
i-09e42474f22039e23 dummy-box-test
vol-0d4b4a7bc95a4b8e4 8 0 0 $0.80
vol-0cc0f6cd3c99bc1cc 8 0 0 $0.80
TOTALS 1 INSTANCES 2 VOLUMES 16 GB 0 0 GB $1.60
$ gw-aws-audit ec2 detached-volumes
VOLUME SIZE (GB) SNAPSHOTS MIN SIZE (GB) COSTS
vol-0cc0f6cd3c99bc1cc 8 0 0 $0.80
TOTALS 1 VOLUMES 8 GB 0 0 GB $0.80
$ gw-aws-audit cw enhanced-monitoring
Enhanced Metrics can add a cost. See: https://aws.amazon.com/cloudwatch/pricing/
Checking for EC2 Enhanced Monitoring
NAME INSTANCE ID
master-us-east-1c.masters.us-east-1.gwdocker.com i-041h4jk12jk23sd
jumpbox12 i-02412sdfgsgdfgs
analytics-prod i-0a87d921n1rtasd
EC2 INSTANCES 3
Checking for RDS Enhanced Monitoring
DB INSTANCE ENGINE
airflow-womp-ba-prod-v2 postgres
dashboard-reporting-production-instance-1 aurora-postgresql
service-loloyol-production aurora-mysql
DB INSTANCES 3
$ gw-aws-audit s3 clear-bucket --bucket yolo
-- WARNING -- PAY ATTENTION -- FOR REALS --
This will delete ALL objects in yolo
-- THIS ACTION IS NOT REVERSIBLE --
Are you SUPER sure? [yolo]
Enter a value: yolo
Proceeding with batch delete for bucket: yolo
Pages: 198788 Listed: 198788000 Deleted: 198781000 Retries: 18373 DPS: 1921.64
$ gw-aws-audit s3 metrics > out.csv
Starting metrics pull...
Bucket metric pull complete. Buckets: 207 Processed: 207
- go v1.14+
- make
- git-chglog
- goreleaser
Run ./release.sh $VERSION
This will update docs, changelog, add the tag, push main and the tag to the repo. The goreleaser action will publish the binaries to the Github Release.
If you want to simulate the goreleaser process, run the following command:
$ curl -sL https://git.io/goreleaser | bash -s -- --rm-dist --skip-publish --snapshot
Please read CONTRIBUTING.md for details on our code of conduct, and the process for submitting pull requests to us.
- Fork the GoodwayGroup/gw-aws-audit repo
- Use
go >= 1.16 - Branch & Code
- Run linters 🧹
golangci-lint run- The project uses golangci-lint
- Commit with a Conventional Commit
- Open a PR
We employ git-chglog to manage the CHANGELOG.md. For the versions available, see the tags on this repository.
- Derek Smith - @clok
See also the list of contributors who participated in this project.
This project is licensed under the MIT License - see the LICENSE file for details
