Skip to content

Commit dbd3dac

Browse files
authored
Merge pull request #67 from ONS-Innovation/KEH-1227
KEH-1227: Rewrite Docs
2 parents 63b8dd4 + f85e8a7 commit dbd3dac

File tree

7 files changed

+109
-97
lines changed

7 files changed

+109
-97
lines changed

README.md

Lines changed: 22 additions & 24 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
# GitHub Copilot Usage Lambda
22

3-
This repository contains the AWS Lambda Function for updating the GitHub Copilot dashboard's historic information, stored within an S3 bucket.
3+
This repository contains the AWS Lambda Function for updating the GitHub Copilot dashboard's organisation-wide historic data, Copilot teams, and teams history.
44

55
The Copilot dashboard can be found on the Copilot tab within the Digital Landscape.
66

@@ -16,7 +16,7 @@ The Copilot dashboard can be found on the Copilot tab within the Digital Landsca
1616
- [Makefile](#makefile)
1717
- [AWS Lambda Scripts](#aws-lambda-scripts)
1818
- [Setup - Running in a container](#setup---running-in-a-container)
19-
- [Setup - running outside of a Container (Development only)](#setup---running-outside-of-a-container-development-only)
19+
- [Setup - running outside of a Container (Development only)](#running-outside-of-a-container-development-only)
2020
- [Storing the container on AWS Elastic Container Registry (ECR)](#storing-the-container-on-aws-elastic-container-registry-ecr)
2121
- [Deployment to AWS](#deployment-to-aws)
2222
- [Deployment Prerequisites](#deployment-prerequisites)
@@ -138,7 +138,23 @@ Further information can be found in [this project's documentation](/docs/index.m
138138
docker stop 3f7d64676b1a
139139
```
140140

141-
### Setup - running outside of a Container (Development only)
141+
### Setup
142+
143+
Export the required environment variables:
144+
145+
```bash
146+
export AWS_ACCESS_KEY_ID=<aws_access_key_id>
147+
export AWS_SECRET_ACCESS_KEY=<aws_secret_access_key>
148+
export AWS_DEFAULT_REGION=eu-west-2
149+
export AWS_SECRET_NAME=<aws_secret_name>
150+
export GITHUB_ORG=ONSDigital
151+
export GITHUB_APP_CLIENT_ID=<github_app_client_id>
152+
export AWS_ACCOUNT_NAME=<sdp-dev/sdp-prod>
153+
```
154+
155+
The lambda can be run outside of a container for development purposes, or inside a container image to push to AWS ECR.
156+
157+
#### Running outside of a Container (Development only)
142158

143159
To run the Lambda function outside of a container, we need to execute the `handler()` function.
144160

@@ -153,25 +169,13 @@ To run the Lambda function outside of a container, we need to execute the `handl
153169

154170
**Please Note:** If uncommenting the above in `main.py`, make sure you re-comment the code _before_ pushing back to GitHub.
155171

156-
2. Export the required environment variables:
157-
158-
```bash
159-
export AWS_ACCESS_KEY_ID=<aws_access_key_id>
160-
export AWS_SECRET_ACCESS_KEY=<aws_secret_access_key>
161-
export AWS_DEFAULT_REGION=eu-west-2
162-
export AWS_SECRET_NAME=<aws_secret_name>
163-
export GITHUB_ORG=ONSDigital
164-
export GITHUB_APP_CLIENT_ID=<github_app_client_id>
165-
export AWS_ACCOUNT_NAME=<sdp-dev/sdp-prod>
166-
```
167-
168-
3. Run the script.
172+
2. Run the script.
169173

170174
```bash
171175
python3 src/main.py
172176
```
173177

174-
### Storing the container on AWS Elastic Container Registry (ECR)
178+
#### Storing the container on AWS Elastic Container Registry (ECR)
175179

176180
When you make changes to the Lambda Script, a new container image must be pushed to ECR.
177181

@@ -294,13 +298,7 @@ If the application has been modified, the following can be performed to update t
294298

295299
The reconfigure options ensures that the backend state is reconfigured to point to the appropriate S3 bucket.
296300

297-
**_Please Note:_** This step requires an **AWS_ACCESS_KEY_ID** and **AWS_SECRET_ACCESS_KEY** to be loaded into the environment if not already in place.
298-
This can be done using:
299-
300-
```bash
301-
export AWS_ACCESS_KEY_ID="<aws_access_key_id>"
302-
export AWS_SECRET_ACCESS_KEY="<aws_secret_access_key>"
303-
```
301+
**_Please Note:_** This step requires an **AWS_ACCESS_KEY_ID** and **AWS_SECRET_ACCESS_KEY** to be loaded into the environment if not already in place. Please refer to [setup](#setup).
304302

305303
- Refresh the local state to ensure it is in sync with the backend
306304

docs/diagrams/architecture.drawio

Lines changed: 34 additions & 35 deletions
Original file line numberDiff line numberDiff line change
@@ -1,53 +1,52 @@
1-
<mxfile host="app.diagrams.net" modified="2024-08-06T13:01:18.866Z" agent="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36" etag="SMw1gVABOdbHb4xL4Wt8" version="24.6.5" type="device">
2-
<diagram name="Page-1" id="bDBSMM_U5Odbe-mlCb5Z">
3-
<mxGraphModel dx="754" dy="834" grid="1" gridSize="10" guides="1" tooltips="1" connect="1" arrows="1" fold="1" page="1" pageScale="1" pageWidth="827" pageHeight="1169" math="0" shadow="0">
1+
<mxfile host="app.diagrams.net" agent="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/143.0.0.0 Safari/537.36" version="29.2.9">
2+
<diagram name="Page-1" id="3vBX3EtQdH57ldrSx5Wk">
3+
<mxGraphModel dx="1395" dy="894" grid="1" gridSize="10" guides="1" tooltips="1" connect="1" arrows="1" fold="1" page="1" pageScale="1" pageWidth="827" pageHeight="1169" math="0" shadow="0">
44
<root>
55
<mxCell id="0" />
66
<mxCell id="1" parent="0" />
7-
<mxCell id="ftDg6omauAznXFmt9yRm-1" value="CoPilot Usage Dashboard" style="rounded=0;whiteSpace=wrap;html=1;align=left;verticalAlign=top;" vertex="1" parent="1">
8-
<mxGeometry x="40" y="40" width="710" height="560" as="geometry" />
9-
</mxCell>
10-
<mxCell id="ftDg6omauAznXFmt9yRm-22" value="Get Historic&lt;div&gt;Data&lt;/div&gt;" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;" edge="1" parent="1" source="ftDg6omauAznXFmt9yRm-3" target="ftDg6omauAznXFmt9yRm-11">
11-
<mxGeometry relative="1" as="geometry" />
12-
</mxCell>
13-
<mxCell id="ftDg6omauAznXFmt9yRm-33" value="Display Data to&lt;div&gt;User&lt;/div&gt;" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;" edge="1" parent="1" source="ftDg6omauAznXFmt9yRm-3" target="ftDg6omauAznXFmt9yRm-30">
7+
<mxCell id="CpdcWXR5zgmDa5A5yRl--1" parent="1" style="rounded=0;whiteSpace=wrap;html=1;align=left;verticalAlign=top;" value="Copilot Usage Lambda &amp;amp;&amp;nbsp;&lt;div&gt;Digital&amp;nbsp;&lt;span style=&quot;background-color: transparent; color: light-dark(rgb(0, 0, 0), rgb(255, 255, 255));&quot;&gt;Landscape Architecture&lt;/span&gt;&lt;/div&gt;" vertex="1">
8+
<mxGeometry height="560" width="630" x="160" y="40" as="geometry" />
9+
</mxCell>
10+
<mxCell id="CpdcWXR5zgmDa5A5yRl--2" edge="1" parent="1" source="CpdcWXR5zgmDa5A5yRl--4" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;startArrow=classic;startFill=1;" target="CpdcWXR5zgmDa5A5yRl--7" value="Get Historic &amp;amp; Team&lt;div&gt;Data&lt;/div&gt;">
11+
<mxGeometry relative="1" as="geometry">
12+
<mxPoint as="offset" />
13+
<Array as="points">
14+
<mxPoint x="540" y="240" />
15+
<mxPoint x="540" y="240" />
16+
</Array>
17+
</mxGeometry>
18+
</mxCell>
19+
<mxCell id="CpdcWXR5zgmDa5A5yRl--3" edge="1" parent="1" source="CpdcWXR5zgmDa5A5yRl--4" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;" target="CpdcWXR5zgmDa5A5yRl--15" value="Display Data to&lt;div&gt;User&lt;/div&gt;">
1420
<mxGeometry relative="1" as="geometry" />
1521
</mxCell>
16-
<mxCell id="ftDg6omauAznXFmt9yRm-3" value="Dashboard" style="rounded=1;whiteSpace=wrap;html=1;verticalAlign=top;spacingTop=25;" vertex="1" parent="1">
17-
<mxGeometry x="280" y="200" width="160" height="160" as="geometry" />
18-
</mxCell>
19-
<mxCell id="ftDg6omauAznXFmt9yRm-4" value="GitHub API Toolkit&lt;div&gt;(Imported)&lt;/div&gt;" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#f8cecc;strokeColor=#b85450;" vertex="1" parent="1">
20-
<mxGeometry x="300" y="290" width="120" height="60" as="geometry" />
22+
<mxCell id="CpdcWXR5zgmDa5A5yRl--4" parent="1" style="rounded=1;whiteSpace=wrap;html=1;verticalAlign=top;spacingTop=25;fillColor=light-dark(#FFFFFF,#333F5C);" value="Digital Landscape: Copilot Page" vertex="1">
23+
<mxGeometry height="90" width="120" x="300" y="200" as="geometry" />
2124
</mxCell>
22-
<mxCell id="ftDg6omauAznXFmt9yRm-5" value="GitHub API" style="ellipse;shape=cloud;whiteSpace=wrap;html=1;fillColor=#d5e8d4;strokeColor=#82b366;" vertex="1" parent="1">
23-
<mxGeometry x="300" y="480" width="120" height="80" as="geometry" />
25+
<mxCell id="CpdcWXR5zgmDa5A5yRl--6" parent="1" style="ellipse;shape=cloud;whiteSpace=wrap;html=1;fillColor=#d5e8d4;strokeColor=#82b366;" value="GitHub API" vertex="1">
26+
<mxGeometry height="80" width="120" x="300" y="480" as="geometry" />
2427
</mxCell>
25-
<mxCell id="ftDg6omauAznXFmt9yRm-11" value="S3&lt;div&gt;Bucket&lt;/div&gt;" style="ellipse;shape=cloud;whiteSpace=wrap;html=1;fillColor=#ffe6cc;strokeColor=#d79b00;" vertex="1" parent="1">
26-
<mxGeometry x="560" y="240" width="120" height="80" as="geometry" />
28+
<mxCell id="CpdcWXR5zgmDa5A5yRl--7" parent="1" style="ellipse;shape=cloud;whiteSpace=wrap;html=1;fillColor=#ffe6cc;strokeColor=#d79b00;" value="S3&amp;nbsp;&lt;span style=&quot;background-color: transparent; color: light-dark(rgb(0, 0, 0), rgb(255, 255, 255));&quot;&gt;Buckets&lt;/span&gt;&lt;div&gt;&lt;ul&gt;&lt;li&gt;Historic Usage Data&lt;/li&gt;&lt;li&gt;Copilot Teams Data&lt;/li&gt;&lt;/ul&gt;&lt;/div&gt;" vertex="1">
29+
<mxGeometry height="120" width="190" x="560" y="180" as="geometry" />
2730
</mxCell>
28-
<mxCell id="ftDg6omauAznXFmt9yRm-24" value="Push Update&lt;div&gt;to S3&lt;/div&gt;" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;" edge="1" parent="1" source="ftDg6omauAznXFmt9yRm-19" target="ftDg6omauAznXFmt9yRm-11">
31+
<mxCell id="CpdcWXR5zgmDa5A5yRl--8" edge="1" parent="1" source="CpdcWXR5zgmDa5A5yRl--9" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;" target="CpdcWXR5zgmDa5A5yRl--7" value="Push updated data&lt;div&gt;to S3&lt;/div&gt;">
2932
<mxGeometry relative="1" as="geometry" />
3033
</mxCell>
31-
<mxCell id="ftDg6omauAznXFmt9yRm-19" value="Lambda Function" style="rounded=1;whiteSpace=wrap;html=1;verticalAlign=top;spacingTop=25;" vertex="1" parent="1">
32-
<mxGeometry x="540" y="400" width="160" height="160" as="geometry" />
34+
<mxCell id="CpdcWXR5zgmDa5A5yRl--9" parent="1" style="rounded=1;whiteSpace=wrap;html=1;verticalAlign=top;spacingTop=25;" value="Lambda Function" vertex="1">
35+
<mxGeometry height="160" width="160" x="575" y="400" as="geometry" />
3336
</mxCell>
34-
<mxCell id="ftDg6omauAznXFmt9yRm-20" value="GitHub API Toolkit&lt;div&gt;(Imported)&lt;/div&gt;" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#f8cecc;strokeColor=#b85450;" vertex="1" parent="1">
35-
<mxGeometry x="560" y="490" width="120" height="60" as="geometry" />
37+
<mxCell id="CpdcWXR5zgmDa5A5yRl--10" parent="1" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#f8cecc;strokeColor=#b85450;" value="GitHub API Toolkit&lt;div&gt;(Imported)&lt;/div&gt;" vertex="1">
38+
<mxGeometry height="60" width="120" x="595" y="490" as="geometry" />
3639
</mxCell>
37-
<mxCell id="ftDg6omauAznXFmt9yRm-25" value="" style="shape=cylinder3;whiteSpace=wrap;html=1;boundedLbl=1;backgroundOutline=1;size=15;fillColor=#dae8fc;strokeColor=#6c8ebf;" vertex="1" parent="1">
38-
<mxGeometry x="80" y="240" width="80" height="80" as="geometry" />
39-
</mxCell>
40-
<mxCell id="ftDg6omauAznXFmt9yRm-26" value="Get API Response&lt;div&gt;to Update&lt;/div&gt;&lt;div&gt;Historic Data&lt;/div&gt;" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;exitX=0;exitY=0.5;exitDx=0;exitDy=0;entryX=0.875;entryY=0.5;entryDx=0;entryDy=0;entryPerimeter=0;" edge="1" parent="1" source="ftDg6omauAznXFmt9yRm-20" target="ftDg6omauAznXFmt9yRm-5">
41-
<mxGeometry relative="1" as="geometry" />
42-
</mxCell>
43-
<mxCell id="ftDg6omauAznXFmt9yRm-28" value="Get Live Data" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;exitX=0.5;exitY=1;exitDx=0;exitDy=0;entryX=0.5;entryY=0.075;entryDx=0;entryDy=0;entryPerimeter=0;" edge="1" parent="1" source="ftDg6omauAznXFmt9yRm-4" target="ftDg6omauAznXFmt9yRm-5">
40+
<mxCell id="CpdcWXR5zgmDa5A5yRl--12" edge="1" parent="1" source="CpdcWXR5zgmDa5A5yRl--10" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;exitX=0;exitY=0.5;exitDx=0;exitDy=0;entryX=0.875;entryY=0.5;entryDx=0;entryDy=0;entryPerimeter=0;startArrow=classic;startFill=1;" target="CpdcWXR5zgmDa5A5yRl--6" value="Get API Responses&lt;div&gt;to Update&lt;/div&gt;&lt;div&gt;Historic Data &amp;amp;&lt;/div&gt;&lt;div&gt;Copilot teams&lt;/div&gt;">
4441
<mxGeometry relative="1" as="geometry" />
4542
</mxCell>
46-
<mxCell id="ftDg6omauAznXFmt9yRm-29" value="Get Example&lt;div&gt;Dataset&lt;/div&gt;" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;exitX=0;exitY=0.5;exitDx=0;exitDy=0;entryX=1;entryY=0.5;entryDx=0;entryDy=0;entryPerimeter=0;" edge="1" parent="1" source="ftDg6omauAznXFmt9yRm-3" target="ftDg6omauAznXFmt9yRm-25">
47-
<mxGeometry relative="1" as="geometry" />
43+
<mxCell id="CpdcWXR5zgmDa5A5yRl--13" edge="1" parent="1" source="CpdcWXR5zgmDa5A5yRl--4" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;exitX=0.5;exitY=1;exitDx=0;exitDy=0;entryX=0.5;entryY=0.075;entryDx=0;entryDy=0;entryPerimeter=0;startArrow=classic;startFill=1;" target="CpdcWXR5zgmDa5A5yRl--6" value="Get Live Data">
44+
<mxGeometry relative="1" as="geometry">
45+
<mxPoint x="360" y="350" as="sourcePoint" />
46+
</mxGeometry>
4847
</mxCell>
49-
<mxCell id="ftDg6omauAznXFmt9yRm-30" value="User" style="shape=umlActor;verticalLabelPosition=top;verticalAlign=bottom;html=1;outlineConnect=0;labelPosition=center;align=center;" vertex="1" parent="1">
50-
<mxGeometry x="345" y="80" width="30" height="60" as="geometry" />
48+
<mxCell id="CpdcWXR5zgmDa5A5yRl--15" parent="1" style="shape=umlActor;verticalLabelPosition=top;verticalAlign=bottom;html=1;outlineConnect=0;labelPosition=center;align=center;" value="User" vertex="1">
49+
<mxGeometry height="60" width="30" x="345" y="80" as="geometry" />
5150
</mxCell>
5251
</root>
5352
</mxGraphModel>

docs/diagrams/architecture.png

1.92 KB
Loading

docs/index.md

Lines changed: 6 additions & 31 deletions
Original file line numberDiff line numberDiff line change
@@ -1,42 +1,17 @@
1-
# GitHub Copilot Usage Dashboard
1+
# GitHub Copilot Usage Lambda
22

33
## Overview
44

5-
This project contains an AWS Lambda Function which updates the GitHub Copilot dashboard's historic information, stored within an S3 bucket.
5+
This project contains a Python script to automatically update the GitHub Copilot dashboard's historic information for the Copilot dashboard within the Digital Landscape. This allows trend analysis into how Software Engineers are using Copilot within ONS over time.
66

77
The Copilot dashboard can be found on the Copilot tab within the Digital Landscape.
88

99
[View the Digital Landscape's repository](https://github.com/ONS-Innovation/keh-digital-landscape).
1010

11-
## Techstack Overview
12-
13-
## Architecture Overview
14-
15-
![Architecture Diagram](./diagrams/architecture.png)
16-
17-
This project uses 2 major components:
18-
19-
- The Lambda Function
20-
- The GitHub API Toolkit (**stored in another repository** - [Repository Link](https://github.com/ONS-Innovation/github-api-package))
21-
22-
### The Lambda Function
23-
24-
This component updates the dashboard's historic information, stored within an S3 bucket. The lambda imports the GitHub API Toolkit to get the API response containing the usage information. The script then adds any new data to the existing historic data within the S3 bucket.
25-
26-
### The GitHub API Toolkit
27-
28-
This component is an imported library which is shared across multiple GitHub tools. The toolkit allows applications to make authenticated requests to the GitHub API. It is imported and used by both the dashboard and lambda function.
29-
30-
## High Level Data Overview
31-
32-
### Endpoint
33-
34-
[View docs for the Copilot usage data endpoint](https://docs.github.com/en/rest/copilot/copilot-usage?apiVersion=2022-11-28#get-a-summary-of-copilot-usage-for-organization-members).
35-
36-
### Historic Data
37-
38-
This section gathers data from AWS S3. The Copilot usage endpoints have a limitation where they only return the last 28 days worth of information. To get around this, the project has an AWS Lambda function which runs weekly and stores data within an S3 bucket.
39-
4011
## Getting Started
4112

4213
To setup and use the project, please refer to the [README](https://github.com/ONS-Innovation/github-copilot-usage-lambda/blob/main/README.md).
14+
15+
## Technical Documentation
16+
17+
For information about the technical aspects of the Copilot Usage Lambda, please refer to the [Technical Documentation](./technical_documentation/overview.md).

docs/configuration.md renamed to docs/technical_documentation/configuration.md

Lines changed: 1 addition & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -15,7 +15,7 @@ The `config.json` file contains the following:
1515
}
1616
```
1717

18-
### `features` Section
18+
### `features`
1919

2020
This section contains feature flags that control which of the tool's features are enabled or disabled.
2121

@@ -29,8 +29,6 @@ When deploying to AWS, this should be set to `false` to avoid files being writte
2929

3030
If set to `true`, the tool will skip writing to the appropriate AWS S3 bucket and instead write data for copilot teams, historic usage, and teams history to `local_data`.
3131

32-
**When deploying to AWS, this must be set to `false` to ensure the tool writes to AWS.**
33-
3432
When debugging locally, you can set this to `true` to use the local configuration file. This is useful if you need to see the logs locally, without affecting the cloud deployment.
3533

3634
### Example During Local Testing
Lines changed: 44 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,44 @@
1+
# Overview
2+
3+
AWS Lambda Function which updates the GitHub Copilot dashboard's:
4+
5+
- Organisation-wide historic data
6+
- Copilot teams
7+
- Teams history
8+
9+
## Tech Stack Overview
10+
11+
This project uses:
12+
13+
- Python
14+
- AWS Lambda
15+
- AWS S3
16+
17+
## Architecture Overview
18+
19+
![Architecture Diagram](../diagrams/architecture.png)
20+
21+
This project uses 2 major components:
22+
23+
- The Lambda Function
24+
- The GitHub API Toolkit (**stored in another repository** - [Repository Link](https://github.com/ONS-Innovation/github-api-package))
25+
26+
### The Lambda Function
27+
28+
This component updates the Digital Landscape's Copilot dashboard data, stored within S3 buckets. The lambda imports the GitHub API Toolkit to get the API response containing the data, then adds any new data to the relevant S3 bucket.
29+
30+
### The GitHub API Toolkit
31+
32+
This component is an imported library which is shared across multiple GitHub tools. The toolkit allows applications to make authenticated requests to the GitHub API. It is imported and used by both the dashboard and lambda function.
33+
34+
### Endpoint
35+
36+
[View docs for the Copilot usage data endpoint](https://docs.github.com/en/rest/copilot/copilot-usage?apiVersion=2022-11-28#get-a-summary-of-copilot-usage-for-organization-members).
37+
38+
### Historic Usage Data
39+
40+
This section gathers data from AWS S3. The Copilot usage endpoints have a limitation where they only return the last 100 days worth of information. To get around this, the project has an AWS Lambda function which runs weekly and stores data within an S3 bucket.
41+
42+
### Copilot Teams Data
43+
44+
This section gathers a list of teams within the organisation with Copilot data and updates the S3 bucket accordingly. This allows all relevant teams to be displayed within the dashboard.

0 commit comments

Comments
 (0)