Skip to content
This repository was archived by the owner on Jun 29, 2019. It is now read-only.

Commit fbecebb

Browse files
nefischernefischer
authored
Merge pull request #1 from UKHomeOffice/CDMP-2344
Cdmp 2344
2 parents 573740c + d6bdbc2 commit fbecebb

18 files changed

+796
-0
lines changed

‎.drone.yml

+33
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,33 @@
1+
pipeline:
2+
lint:
3+
image: python:3.7-alpine
4+
commands:
5+
- apk add --no-cache gcc musl-dev make
6+
- python3 -m venv venv
7+
- source venv/bin/activate
8+
- pip install -r requirements.txt
9+
- pip install -r requirements-dev.txt
10+
- pylint *.py tests/*.py
11+
- deactivate
12+
when:
13+
event:
14+
- push
15+
16+
unit_test:
17+
image: python:3.7-alpine
18+
commands:
19+
- apk add --no-cache make
20+
- source venv/bin/activate
21+
- python -m pytest --rootdir=tests
22+
- deactivate
23+
when:
24+
event:
25+
- push
26+
27+
inspec_check:
28+
image: chef/inspec
29+
commands:
30+
- inspec check perf-test-inspec
31+
when:
32+
event:
33+
- push

‎.gitignore

+6
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
venv/
2+
.DS_Store
3+
.idea/
4+
.vscode/
5+
attributes.yml
6+
__pycache__/

‎README.md

+46
Original file line numberDiff line numberDiff line change
@@ -1,2 +1,48 @@
11
# perf-testing-aws-resources
2+
23
CloudFormation template to create an S3 bucket for storing performance reports
4+
5+
This repo contains a python script generating a CloudFormation template that can be used to create
6+
an S3 bucket for storing performance reports.
7+
8+
An optional `PREFIX` environment variable can defined (if defined, it should end with `'-'` for naming consistency).
9+
An environment variable called `ENVIRONMENT` should also be defined. It represents the environment for which
10+
the performance testing resources are created. For example, it could be the name of a Kubernetes namespace.
11+
12+
The bucket created will be called:
13+
`${PREFIX}${ENVIRONMENT}-performance-tests`
14+
15+
Managed policies are also created:
16+
17+
* `${PREFIX}${ENVIRONMENT}-performance-test-bucket-writeonly` allowing to write to the S3 bucket and intended to be used by the
18+
performance tool
19+
* `${PREFIX}${ENVIRONMENT}-performance-tests-bucket-readonly` allowing a user to read the reports
20+
* `${PREFIX}${ENVIRONMENT}-performance-tests-bucket-admin` allowing an admin to read and delete S3 objets (but not write)
21+
22+
## Creating the resources
23+
24+
```bash
25+
# define PREFIX and ENVIRONMENT variables
26+
# PREFIX can be empty but ENVIRONMENT should not
27+
aws cloudformation create-change-set --stack-name "${PREFIX}${ENVIRONMENT}-perf-testing" \
28+
--change-set-name=perf-testing-$RANDOM \
29+
--change-set-type=CREATE \
30+
--capabilities CAPABILITY_NAMED_IAM \
31+
--template-body "$(python generate_cf_template.py)" \
32+
--parameters ParameterKey=Prefix,ParameterValue=${PREFIX} \
33+
ParameterKey=Environment,ParameterValue=${ENVIRONMENT}
34+
```
35+
36+
## Testing
37+
38+
Chef Inspec is used to test the resources.
39+
40+
Assuming credentials are available as `AWS_*` environment variables, the following will test the AWS resources:
41+
42+
```bash
43+
cat <<EOF > attributes.yml
44+
prefix: "${PREFIX}"
45+
environment: "${ENVIRONMENT}"
46+
EOF
47+
inspec exec perf-test-inspec --input-file=attributes.yml --target aws://
48+
```

‎generate_cf_template.py

+231
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,231 @@
1+
"""Script generating a CloudFormation template to create an S3 bucket destined to hold
2+
performance test reports.
3+
4+
The template also contains IAM resources to control access to the bucket.
5+
"""
6+
7+
from troposphere import Template, Parameter, Ref, Join
8+
from troposphere import iam
9+
from troposphere import s3
10+
11+
12+
class InvalidResourceName(Exception):
13+
"""Exception indicating that a resource name is invalid
14+
"""
15+
16+
17+
class PerfTestingTemplate(Template):
18+
"""Template class generating a CloudFormation template to create an S3 bucket destined to hold
19+
performance test reports.
20+
"""
21+
22+
BUCKET_SHORT_NAME = 'performance-tests'
23+
24+
def __init__(self):
25+
"""Constructor building up the CloudFormation template
26+
"""
27+
Template.__init__(self)
28+
self.set_description('Template creating an S3 bucket destined to hold performance test '
29+
'reports. The template also contains IAM resources to control access '
30+
'to the bucket.')
31+
32+
self.environmnent_param = self.add_parameter(Parameter(
33+
'Environment',
34+
Description='Name of the environment the bucket is for (e.g. a k8s namespace)',
35+
Type='String',
36+
))
37+
38+
self.prefix_param = self.add_parameter(Parameter(
39+
'Prefix',
40+
Description='A prefix to prepend all resource names with',
41+
Type='String',
42+
Default='',
43+
))
44+
45+
self.bucket = self.add_s3_bucket()
46+
self.add_iam_resources()
47+
48+
def resource_name(self, *names):
49+
"""Gets the fully qualified resource name taking into account the Prefix and Environment
50+
parameters.
51+
52+
:param names: a variadic argument (str...) specifying the suffixes to use to describe the
53+
resource
54+
:return: the fully qualified name of the resource, prepended with the prefix and environment
55+
name
56+
"""
57+
if not names:
58+
raise InvalidResourceName('Provide at least one name parameter')
59+
if len(names) == 1:
60+
name = names[0]
61+
else:
62+
name = Join('-', names)
63+
return Join('', [Ref(self.prefix_param), Ref(self.environmnent_param), '-', name])
64+
65+
def add_s3_bucket(self):
66+
"""Adds an S3 bucket to the template
67+
68+
:return: the (s3.Bucket) object that has been added
69+
"""
70+
bucket = s3.Bucket(
71+
'Bucket',
72+
BucketName=self.resource_name(PerfTestingTemplate.BUCKET_SHORT_NAME),
73+
BucketEncryption=s3.BucketEncryption(
74+
ServerSideEncryptionConfiguration=[
75+
s3.ServerSideEncryptionRule(
76+
ServerSideEncryptionByDefault=s3.ServerSideEncryptionByDefault(
77+
SSEAlgorithm='AES256', # alternative is 'aws:kms'
78+
)
79+
)
80+
]
81+
),
82+
VersioningConfiguration=s3.VersioningConfiguration(
83+
Status='Enabled',
84+
)
85+
)
86+
self.add_resource(bucket)
87+
return bucket
88+
89+
def add_iam_resources(self):
90+
"""Adds IAM resources to the template: roles and managed policies for read-only, write-only
91+
and delete roles
92+
93+
:return: None
94+
"""
95+
readonly_policy = self.add_managed_policy(
96+
title='ReadOnlyPolicy',
97+
description_verb='reading',
98+
policy_name_suffix='readonly',
99+
allowed_actions=[
100+
"s3:Get*",
101+
"s3:List*",
102+
],
103+
)
104+
self.add_role(
105+
title='ReadOnlyRole',
106+
role_name_suffix='readonly',
107+
managed_policy=readonly_policy,
108+
)
109+
110+
writeonly_policy = self.add_managed_policy(
111+
title='WriteOnlyPolicy',
112+
description_verb='writing',
113+
policy_name_suffix='writeonly',
114+
allowed_actions=[
115+
"s3:PutObject",
116+
"s3:AbortMultipartUpload",
117+
"s3:ListMultipartUploadParts",
118+
],
119+
)
120+
self.add_role(
121+
title='WriteOnlyRole',
122+
role_name_suffix='writeonly',
123+
managed_policy=writeonly_policy,
124+
)
125+
126+
delete_policy = self.add_managed_policy(
127+
title='DeletePolicy',
128+
description_verb='deleting',
129+
policy_name_suffix='delete',
130+
allowed_actions=[
131+
"s3:Get*",
132+
"s3:List*",
133+
"s3:DeleteObject*",
134+
],
135+
)
136+
self.add_role(
137+
title='DeleteRole',
138+
role_name_suffix='delete',
139+
managed_policy=delete_policy,
140+
)
141+
142+
def add_managed_policy(self, title, description_verb, policy_name_suffix, allowed_actions):
143+
"""
144+
Creates a managed policy (iam.Policy) with the permissions required to interact with the S3
145+
bucket used for performance testing data.
146+
147+
:param title: (str) the title of the resource
148+
:param description_verb: (str) the verb to use in the description, e.g. 'reading'
149+
:param policy_name_suffix: (str) the suffix to append to the policy name, e.g. 'readonly'
150+
:param allowed_actions: (str[]) the IAM actions that should be allowed on the bucket or
151+
its objects
152+
:return: an iam.ManagedPolicy object
153+
"""
154+
return self.add_resource(iam.ManagedPolicy(
155+
title,
156+
Description=Join(
157+
' ',
158+
[
159+
f"Policy allowing {description_verb} objects from the S3 bucket containing "
160+
f"performance test data for",
161+
Ref(self.prefix_param),
162+
Ref(self.environmnent_param),
163+
]
164+
),
165+
ManagedPolicyName=self.resource_name(
166+
f"{PerfTestingTemplate.BUCKET_SHORT_NAME}-bucket-{policy_name_suffix}",
167+
),
168+
PolicyDocument={
169+
"Version": "2012-10-17",
170+
"Statement": [
171+
{
172+
"Action": allowed_actions,
173+
"Effect": "Allow",
174+
"Resource": [
175+
Join('', [
176+
'arn:aws:s3:::',
177+
Ref(self.bucket),
178+
]),
179+
Join('', [
180+
'arn:aws:s3:::',
181+
Ref(self.bucket),
182+
'/*',
183+
]),
184+
]
185+
},
186+
]
187+
},
188+
))
189+
190+
def add_role(self, title, role_name_suffix, managed_policy):
191+
"""
192+
Adds a iam.Role associated with the given managed policy to the template.
193+
194+
:param title: (str) the title of the resource
195+
:param role_name_suffix: (str) the suffix to append to the role name, e.g. 'readonly'
196+
:param managed_policy: the managed policy to associate with this role (iam.ManagedPolicy)
197+
198+
:return: an iam.Role object
199+
"""
200+
return self.add_resource(iam.Role(
201+
title,
202+
RoleName=self.resource_name(
203+
f"{PerfTestingTemplate.BUCKET_SHORT_NAME}-bucket-{role_name_suffix}",
204+
),
205+
AssumeRolePolicyDocument={
206+
"Statement": [{
207+
"Effect": "Allow",
208+
"Principal": {
209+
# arn:aws:iam::AWS-account-ID:role/role-name
210+
"AWS": [
211+
Ref("AWS::AccountId"),
212+
]
213+
},
214+
"Action": ["sts:AssumeRole"]
215+
}]
216+
},
217+
ManagedPolicyArns=[
218+
Ref(managed_policy)
219+
],
220+
))
221+
222+
def get_cf_template(self):
223+
"""Returns the CloudFormation template as a string
224+
225+
:return: (str) the CloudFormation template
226+
"""
227+
return self.to_yaml()
228+
229+
230+
if __name__ == "__main__":
231+
print(PerfTestingTemplate().get_cf_template())

‎perf-test-inspec/README.md

+4
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
# InSpec Profile for testing performance testings resources in AWS
2+
3+
This InSpec profile checks the IAM resources created to manage and access the S3 bucket
4+
containing the performance data.

‎perf-test-inspec/controls/bucket_access_spec.rb

+28
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
prefix = attribute('prefix')
2+
environment = attribute('environment')
3+
bucket_name = "#{prefix}#{environment}-performance-tests"
4+
5+
bucket = aws_s3_bucket(bucket_name)
6+
7+
control 'bucket-access-controls' do
8+
impact 1.0 # The criticality, if this control fails.
9+
title 'Check access controls for the bucket' # A human-readable title
10+
desc 'Check that the bucket is not public and that all access is handled through IAM policies'
11+
12+
# http://inspec.io/docs/reference/resources/aws_s3_bucket
13+
describe bucket do
14+
it { should exist }
15+
16+
it { should_not be_public }
17+
18+
it { should have_default_encryption_enabled }
19+
20+
its(:bucket_policy) { should be_empty }
21+
end
22+
23+
# http://inspec.io/docs/reference/resources/aws_s3_bucket
24+
describe bucket.bucket_acl do
25+
# only the canonical user should be listed in the ACLs
26+
its(:length) { should be == 1 }
27+
end
28+
end

0 commit comments

Comments
 (0)