Skip to content

Commit 15a05eb

Browse files
TotalDwarf03TotalDwarf03
authored
Merge pull request #71 from ONS-Innovation/KEH-1813-concourse
KEH-1813 - Concourse
2 parents 1f3e601 + c4ad073 commit 15a05eb

File tree

7 files changed

+463
-192
lines changed

7 files changed

+463
-192
lines changed

‎.mega-linter.yml‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -38,6 +38,9 @@ DISABLE_LINTERS:
3838
# Disable due to poor configuration options
3939
- ACTION_ACTIONLINT
4040

41+
# Disable due to Prettier conflict
42+
- MARKDOWN_MARKDOWN_TABLE_FORMATTER
43+
4144
SHOW_ELAPSED_TIME: true
4245

4346
FILEIO_REPORTER: false

‎README.md‎

Lines changed: 118 additions & 84 deletions
Original file line numberDiff line numberDiff line change
@@ -2,9 +2,9 @@
22

33
This repository contains the AWS Lambda Function for updating the GitHub Copilot dashboard's organisation-wide historic data, Copilot teams, and teams history.
44

5-
The Copilot dashboard can be found on the Copilot tab within the Digital Landscape.
5+
The Copilot dashboard can be found on the GitHub Copilot tab within the Digital Landscape.
66

7-
[View the Digital Landscape's repository](https://github.com/ONS-Innovation/keh-digital-landscape).
7+
[View the Digital Landscape's repository](https://github.com/ONSdigital/keh-digital-landscape).
88

99
---
1010

@@ -14,10 +14,20 @@ The Copilot dashboard can be found on the Copilot tab within the Digital Landsca
1414
- [Table of Contents](#table-of-contents)
1515
- [Prerequisites](#prerequisites)
1616
- [Makefile](#makefile)
17-
- [AWS Lambda Scripts](#aws-lambda-scripts)
18-
- [Setup - Running in a container](#setup---running-in-a-container)
19-
- [Setup - running outside of a Container (Development only)](#running-outside-of-a-container-development-only)
20-
- [Storing the container on AWS Elastic Container Registry (ECR)](#storing-the-container-on-aws-elastic-container-registry-ecr)
17+
- [AWS Lambda Script](#aws-lambda-script)
18+
- [Running the Project](#running-the-project)
19+
- [Outside of a Container (Recommended) (Development Only)](#outside-of-a-container-recommended-development-only)
20+
- [Running in a container](#running-in-a-container)
21+
- [Deployment](#deployment)
22+
- [Deployments with Concourse](#deployments-with-concourse)
23+
- [Allowlisting your IP](#allowlisting-your-ip)
24+
- [Setting up a pipeline](#setting-up-a-pipeline)
25+
- [Prod deployment](#prod-deployment)
26+
- [Triggering a pipeline](#triggering-a-pipeline)
27+
- [Destroying a pipeline](#destroying-a-pipeline)
28+
- [Manual Deployment](#manual-deployment)
29+
- [Deployment Overview](#deployment-overview)
30+
- [Storing the container on AWS Elastic Container Registry (ECR)](#storing-the-container-on-aws-elastic-container-registry-ecr)
2131
- [Deployment to AWS](#deployment-to-aws)
2232
- [Deployment Prerequisites](#deployment-prerequisites)
2333
- [Underlying AWS Infrastructure](#underlying-aws-infrastructure)
@@ -26,10 +36,6 @@ The Copilot dashboard can be found on the Copilot tab within the Digital Landsca
2636
- [Running the Terraform](#running-the-terraform)
2737
- [Updating the running service using Terraform](#updating-the-running-service-using-terraform)
2838
- [Destroy the Main Service Resources](#destroy-the-main-service-resources)
29-
- [Deployments with Concourse](#deployments-with-concourse)
30-
- [Allowlisting your IP](#allowlisting-your-ip)
31-
- [Setting up a pipeline](#setting-up-a-pipeline)
32-
- [Triggering a pipeline](#triggering-a-pipeline)
3339
- [Documentation](#documentation)
3440
- [Testing](#testing)
3541
- [Linting](#linting)
@@ -62,7 +68,7 @@ This repository has a Makefile for executing common commands. To view all comman
6268
make all
6369
```
6470

65-
## AWS Lambda Scripts
71+
## AWS Lambda Script
6672

6773
This script:
6874

@@ -72,7 +78,42 @@ This script:
7278

7379
Further information can be found in [this project's documentation](/docs/index.md).
7480

75-
### Setup - Running in a container
81+
### Running the Project
82+
83+
### Outside of a Container (Recommended) (Development Only)
84+
85+
To run the Lambda function outside of a container, we need to execute the `handler()` function.
86+
87+
1. Uncomment the following at the bottom of `main.py`.
88+
89+
```python
90+
...
91+
# if __name__ == "__main__":
92+
# handler(None, None)
93+
...
94+
```
95+
96+
**Please Note:** If uncommenting the above in `main.py`, make sure you re-comment the code _before_ pushing back to GitHub.
97+
98+
2. Export the required environment variables:
99+
100+
```bash
101+
export AWS_ACCESS_KEY_ID=<aws_access_key_id>
102+
export AWS_SECRET_ACCESS_KEY=<aws_secret_access_key>
103+
export AWS_DEFAULT_REGION=eu-west-2
104+
export AWS_SECRET_NAME=<aws_secret_name>
105+
export GITHUB_ORG=ONSDigital
106+
export GITHUB_APP_CLIENT_ID=<github_app_client_id>
107+
export AWS_ACCOUNT_NAME=<sdp-dev/sdp-prod>
108+
```
109+
110+
3. Run the script.
111+
112+
```bash
113+
python3 src/main.py
114+
```
115+
116+
### Running in a container
76117

77118
1. Build a Docker Image
78119

@@ -89,7 +130,7 @@ Further information can be found in [this project's documentation](/docs/index.m
89130
**Example Output:**
90131

91132
| REPOSITORY | TAG | IMAGE ID | CREATED | SIZE |
92-
|-----------------------------|--------|--------------|----------------|-------|
133+
| --------------------------- | ------ | ------------ | -------------- | ----- |
93134
| copilot-usage-lambda-script | latest | 0bbe73d9256f | 11 seconds ago | 224MB |
94135

95136
3. Run the image locally mapping local host port (9000) to container port (8080) and passing in AWS credentials to download a .pem file from the AWS Secrets Manager to the running container. These credentials will also be used to upload and download `historic_usage_data.json` to and from S3.
@@ -99,7 +140,7 @@ Further information can be found in [this project's documentation](/docs/index.m
99140
```bash
100141
docker run --platform linux/amd64 -p 9000:8080 \
101142
-e AWS_ACCESS_KEY_ID=<aws_access_key_id> \
102-
-e AWS_SECRET_ACCESS_KEY=<aws_secret_access_key_id> \
143+
-e AWS_SECRET_ACCESS_KEY=<aws_secret_access_key> \
103144
-e AWS_DEFAULT_REGION=eu-west-2 \
104145
-e AWS_SECRET_NAME=<aws_secret_name> \
105146
-e GITHUB_ORG=ONSDigital \
@@ -129,7 +170,7 @@ Further information can be found in [this project's documentation](/docs/index.m
129170
**Example output:**
130171

131172
| CONTAINER ID | IMAGE | COMMAND | CREATED | STATUS | PORTS | NAMES |
132-
|--------------|-----------------------------|------------------------|----------------|---------------|-------------------------------------------|--------------|
173+
| ------------ | --------------------------- | ---------------------- | -------------- | ------------- | ----------------------------------------- | ------------ |
133174
| 3f7d64676b1a | copilot-usage-lambda-script | "/lambda-entrypoint.…" | 44 seconds ago | Up 44 seconds | 0.0.0.0:9000->8080/tcp, :::9000->8080/tcp | nice_ritchie |
134175

135176
Stop the container
@@ -138,42 +179,77 @@ Further information can be found in [this project's documentation](/docs/index.m
138179
docker stop 3f7d64676b1a
139180
```
140181

141-
### Setup
182+
## Deployment
142183

143-
Export the required environment variables:
184+
### Deployments with Concourse
144185

145-
```bash
146-
export AWS_ACCESS_KEY_ID=<aws_access_key_id>
147-
export AWS_SECRET_ACCESS_KEY=<aws_secret_access_key>
148-
export AWS_DEFAULT_REGION=eu-west-2
149-
export AWS_SECRET_NAME=<aws_secret_name>
150-
export GITHUB_ORG=ONSDigital
151-
export GITHUB_APP_CLIENT_ID=<github_app_client_id>
152-
export AWS_ACCOUNT_NAME=<sdp-dev/sdp-prod>
153-
```
186+
#### Allowlisting your IP
154187

155-
The lambda can be run outside of a container for development purposes, or inside a container image to push to AWS ECR.
188+
To setup the deployment pipeline with concourse, you must first allowlist your IP address on the Concourse
189+
server. IP addresses are flushed everyday at 00:00 so this must be done at the beginning of every working day
190+
whenever the deployment pipeline needs to be used. Follow the instructions on the Confluence page (SDP Homepage > SDP Concourse > Concourse Login) to
191+
login. All our pipelines run on sdp-pipeline-prod, whereas sdp-pipeline-dev is the account used for
192+
changes to Concourse instance itself. Make sure to export all necessary environment variables from sdp-pipeline-prod (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_SESSION_TOKEN).
156193

157-
#### Running outside of a Container (Development only)
194+
#### Setting up a pipeline
158195

159-
To run the Lambda function outside of a container, we need to execute the `handler()` function.
196+
When setting up our pipelines, we use ecs-infra-user on sdp-dev to be able to interact with our infrastructure on AWS. The credentials for this are stored on
197+
AWS Secrets Manager so you do not need to set up anything yourself.
160198

161-
1. Uncomment the following at the bottom of `main.py`.
199+
To set the pipeline, run the following script:
162200

163-
```python
164-
...
165-
# if __name__ == "__main__":
166-
# handler(None, None)
167-
...
168-
```
201+
```bash
202+
chmod u+x ./concourse/scripts/set_pipeline.sh
203+
./concourse/scripts/set_pipeline.sh
204+
```
169205

170-
**Please Note:** If uncommenting the above in `main.py`, make sure you re-comment the code _before_ pushing back to GitHub.
206+
Note that you only have to run chmod the first time running the script in order to give permissions.
207+
This script will set the branch and pipeline name to whatever branch you are currently on. It will also set the image tag on ECR to 7 characters of the current branch name if running on a branch other than main. For main, the ECR tag will be the latest release tag on the repository that has semantic versioning(vX.Y.Z).
171208

172-
2. Run the script.
209+
The pipeline name itself will usually follow a pattern as follows: `github-copilot-usage-lambda-<branch-name>` for any non-main branch and `github-copilot-usage-lambda` for the main/master branch.
173210

174-
```bash
175-
python3 src/main.py
176-
```
211+
#### Prod deployment
212+
213+
To deploy to prod, it is required that a Github Release is made on Github. The release is required to follow semantic versioning of vX.Y.Z.
214+
215+
A manual trigger is to be made on the pipeline name `github-copilot-usage-lambda > deploy-after-github-release` job through the Concourse CI UI. This will create a github-create-tag resource that is required on the `github-copilot-usage-lambda > build-and-push-prod` job. Then the prod deployment job is also through a manual trigger ensuring that prod is only deployed using the latest GitHub release tag in the form of vX.Y.Z and is manually controlled.
216+
217+
#### Triggering a pipeline
218+
219+
Once the pipeline has been set, you can manually trigger a dev build on the Concourse UI, or run the following command for non-main branch deployment:
220+
221+
```bash
222+
fly -t aws-sdp trigger-job -j copilot-usage-lambda-<branch-name>/build-and-push-dev
223+
```
224+
225+
and for main branch deployment:
226+
227+
```bash
228+
fly -t aws-sdp trigger-job -j copilot-usage-lambda/build-and-push-dev
229+
```
230+
231+
#### Destroying a pipeline
232+
233+
To destroy the pipeline, run the following command:
234+
235+
```bash
236+
fly -t aws-sdp destroy-pipeline -p copilot-usage-lambda-<branch-name>
237+
```
238+
239+
**It is unlikely that you will need to destroy a pipeline, but the command is here if needed.**
240+
241+
**Note:** This will not destroy any resources created by Terraform. You must manually destroy these resources using Terraform.
242+
243+
### Manual Deployment
244+
245+
#### Deployment Overview
246+
247+
This repository is designed to be hosted on AWS Lambda using a container image as the Lambda's definition.
248+
249+
There are 2 parts to deployment:
250+
251+
1. Updating the ECR Image.
252+
2. Updating the Lambda.
177253

178254
#### Storing the container on AWS Elastic Container Registry (ECR)
179255

@@ -298,7 +374,7 @@ If the application has been modified, the following can be performed to update t
298374

299375
The reconfigure options ensures that the backend state is reconfigured to point to the appropriate S3 bucket.
300376

301-
**_Please Note:_** This step requires an **AWS_ACCESS_KEY_ID** and **AWS_SECRET_ACCESS_KEY** to be loaded into the environment if not already in place. Please refer to [setup](#setup).
377+
**_Please Note:_** This step requires an **AWS_ACCESS_KEY_ID** and **AWS_SECRET_ACCESS_KEY** to be loaded into the environment if not already in place.
302378

303379
- Refresh the local state to ensure it is in sync with the backend
304380

@@ -338,48 +414,6 @@ terraform refresh -var-file=env/dev/dev.tfvars
338414
terraform destroy -var-file=env/dev/dev.tfvars
339415
```
340416

341-
## Deployments with Concourse
342-
343-
### Allowlisting your IP
344-
345-
To setup the deployment pipeline with concourse, you must first allowlist your IP address on the Concourse
346-
server. IP addresses are flushed everyday at 00:00 so this must be done at the beginning of every working day whenever the deployment pipeline needs to be used.
347-
348-
Follow the instructions on the Confluence page (SDP Homepage > SDP Concourse > Concourse Login) to
349-
login. All our pipelines run on `sdp-pipeline-prod`, whereas `sdp-pipeline-dev` is the account used for
350-
changes to Concourse instance itself. Make sure to export all necessary environment variables from `sdp-pipeline-prod` (**AWS_ACCESS_KEY_ID**, **AWS_SECRET_ACCESS_KEY**, **AWS_SESSION_TOKEN**).
351-
352-
### Setting up a pipeline
353-
354-
When setting up our pipelines, we use `ecs-infra-user` on `sdp-dev` to be able to interact with our infrastructure on AWS. The credentials for this are stored on AWS Secrets Manager so you do not need to set up anything yourself.
355-
356-
To set the pipeline, run the following script:
357-
358-
```bash
359-
chmod u+x ./concourse/scripts/set_pipeline.sh
360-
./concourse/scripts/set_pipeline.sh github-copilot-usage-lambda
361-
```
362-
363-
Note that you only have to run chmod the first time running the script in order to give permissions.
364-
This script will set the branch and pipeline name to whatever branch you are currently on. It will also set the image tag on ECR to the current commit hash at the time of setting the pipeline.
365-
366-
The pipeline name itself will usually follow a pattern as follows: `<repo-name>-<branch-name>`
367-
If you wish to set a pipeline for another branch without checking out, you can run the following:
368-
369-
```bash
370-
./concourse/scripts/set_pipeline.sh github-copilot-usage-lambda <branch_name>
371-
```
372-
373-
If the branch you are deploying is `main`, it will trigger a deployment to the `sdp-prod` environment. To set the ECR image tag, you must draft a GitHub release pointing to the latest release of the `main` branch that has a tag in the form of `vX.Y.Z.` Drafting up a release will automatically deploy the latest version of the `main` branch with the associated release tag, but you can also manually trigger a build through the Concourse UI or the terminal prompt.
374-
375-
### Triggering a pipeline
376-
377-
Once the pipeline has been set, you can manually trigger a build on the Concourse UI, or run the following command:
378-
379-
```bash
380-
fly -t aws-sdp trigger-job -j github-copilot-usage-lambda-<branch-name>/build-and-push
381-
```
382-
383417
## Documentation
384418

385419
This project uses MkDocs for documentation which gets deployed to GitHub Pages at a repository level.

0 commit comments

Comments
 (0)