You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
This repository was archived by the owner on Mar 27, 2023. It is now read-only.
Here is an overview of the project architecture, including the CI/CD pipeline and the AWS infrastructure that will be automatically provisioned through the AWS Cloud Development Kit:
12
11
13
-
First, copy `.env.template` to a new file in the project's root directory called `.env`. This file will be read by `docker-compose` in the next step. Adjust any of the values in this file if needed, or add new variables for any secret information you need to pass to docker-compose (or to docker containers).
12
+
14
13
15
-
## Current Project Goals
14
+
(This diagram was created with [draw.io](https://draw.io). Here's the link to the a read-only version of the diagram on draw.io: [https://drive.google.com/file/d/1gU61zjoW80fCusUcswU1zhEE5VFB1Z5U/view?usp=sharing](https://drive.google.com/file/d/1gU61zjoW80fCusUcswU1zhEE5VFB1Z5U/view?usp=sharing)
16
15
17
-
Currently I am working on replacing CloudFormation with CDK for infrastructure and deployment.
16
+
### Legend
18
17
19
-
To work with CDK, do the following:
18
+
1 - GitLab is used to host the source code, test the source code and deploy the application to AWS.
20
19
21
-
- Make sure you are using at least version 10 of node: `nvm use 13`
22
-
- Activate the virtual environment with `source awscdk/.env/bin/activate`
23
-
-`pip install -e awscdk` to install CDK dependencies
24
-
- run `cdk synth --app awscdk/app.py --output awscdk/cdk.out` and view the resulting JSON for the nested CloudFormation stacks in `awscdk/cdk.out`
20
+
2 - Unit testing (see `.gitlab-ci.yml`)
25
21
26
-
### Social Authentication Keys
22
+
2a - Pytest
27
23
28
-
To use social sign on in development, you will need to create an application with the given provider.
24
+
2b - Jest
29
25
30
-
#### GitHub
26
+
2c - Cypress
31
27
32
-
Go to [https://github.com/settings/applications/new](https://github.com/settings/applications/new), and add the following:
28
+
3 - Deployment phase (see `/gitlab-ci/aws/cdk.yml`)
33
29
34
-
- Application Name: A name for the development application, such as `My App Dev`
35
-
- Homepage URL: `http://localhost`
36
-
- Application description: (optional)
37
-
- Authorization callback URL `http://localhost/auth/github/callback` (this route is defined in `quasar/src/router/routes.js`)
38
-
39
-
In the `.env` file, add the `Client ID` of your GitHub OAuth App as the `GITHUB_KEY` variable, and add the `Client Secret` as the `GITHUB_SECRET` variable.
30
+
3a - Quasar PWA assets are built if there are changes in the `quasar` directory
40
31
41
-
```sh
42
-
docker-compose up --build
43
-
```
44
-
45
-
Open `http://localhost` in your browser.
46
-
47
-
You can specify environment variables for docker-compose by adding an `.env` file to the root of the project based on `.env.template`.
48
-
49
-
## VuePress Documentation
50
-
51
-
This project uses VuePress for documentation. To view the documentation site locally, run the following command:
52
-
53
-
```bash
54
-
docker-compose -f compose/docs.yml up --build
55
-
```
32
+
3b - AWS Cloud Development Kit (CDK) defines all infrastructure in AWS (4a - 12)
56
33
57
-
This will make the docs available at `http://localhost:8082/docs/`. Hot-reloading through websockets is supported, so changes will show up as they are saved in your code editor.
34
+
3c - AWS CLI is used to run Fargate tasks through manual GitLab CI jobs
58
35
59
-
### Access Django Shell in Jupyter Notebook
36
+
4 - CDK Assets (ECR and S3 buckets that CDK uses internally to manage build assets and artifacts)
60
37
61
-
With all containers running, run the following commands:
38
+
4a - Elastic Container Repository is used to manage the Django docker image used in various parts of the application
62
39
63
-
```
64
-
docker exec -it backend bash
65
-
# cd notebooks/
66
-
# ../manage.py shell_plus --notebook
67
-
```
40
+
4b - S3 bucket used to store files associated with CDK and CloudFormation
68
41
69
-
or use this single command:
42
+
5 - Route53 is used to route traffic to the CloudFront distribution
7c - S3 bucket for Django assets (static files, public media and private media)
82
53
83
-
```
84
-
docker-compose exec backend pytest
85
-
```
54
+
8 - Web server and websocket servers
86
55
87
-
```
88
-
docker exec -it backend bash
89
-
root@b24c4206002e:/code# pytest
90
-
```
56
+
8a - Fargate service running uvicorn process (REST, GraphQL, Django Channels)
91
57
92
-
To run Jest tests, you can run:
58
+
8b - Autoscaling Group for Fargate Service that serves Django API
93
59
94
-
```
95
-
docker-compose exec frontend npm run test
96
-
```
60
+
9 - Celery and celery worker autoscaling
97
61
98
-
## Integration tests
62
+
9a - Fargate service that is autoscaled between 0 and `N` Fargate tasks for a given celery queue
99
63
100
-
Cypress can be used in an interactive mode with a UI, or it can be used in a headless way (such as in GitLab CI).
64
+
9b - Scheduled Event that triggers a Lambda to make a request to Django backend which collects celery queue metrics and published metrics to CloudWatch using boto3
101
65
102
-
## Running Cypress Interactively
66
+
9c - Lambda event the makes a request to `/api/celery-metrics/`
103
67
104
-
To run Cypress tests interactively, install Cypress in the root of the project:
68
+
9d - CloudWatch alarm that is used to scale the Fargate service for a celery queue
105
69
106
-
```
107
-
npm install cypress
108
-
```
70
+
9e - Autoscaling group for celery Fargate service
109
71
110
-
Then open Cypress with:
72
+
10 - Fargate tasks that run Django management commands such as `migrate` and `collectstatic`. These are triggered from manual GitLab CI jobs using the AWS CLI (3c)
111
73
112
-
```
113
-
$(npm bin)/cypress open --config baseUrl=http://localhost
114
-
```
74
+
11 - ElastiCache for Redis, used for Caching, Celery Broker, Channels Layer, etc.
115
75
116
-
Click on the button "Run all specs", and the tests will start running in a new browser window with a log of Cypress actions and test statements displayed on the left side.
76
+
12 - Aurora Postgres Serverless
117
77
118
-
## Run Cypress in headless mode
119
-
120
-
There are two ways to run Cypress in headless mode. You can run Cypress against the `docker-compose` file in `compose/test.yml`, or you can run the Cypress test using `gitlab-runner`.
121
-
122
-
### Using `compose/test.yml`
123
-
124
-
To run the test locally against the production image, run the following:
125
-
126
-
```
127
-
docker-compose -f compose/test.yml up --build
128
-
```
129
-
130
-
This will build a special environment that resembles the environment used in GitLab CI. It brings up three containers: redis, postgres and backend. The backend serves several purposes: it runs the Django webserver, it runs the ASGI server used by Django Channels, and it runs celery tasks synchronously, and it also serves the Vue application through Django templates and static files. See the Dockerfile located at `backend/scripts/prod/Dockerfile` for details on how this works.
131
-
132
-
Make sure that Cypress is properly installed with:
133
-
134
-
```
135
-
$(npm bin)/cypress verify
136
-
```
78
+
## Local Development
137
79
138
-
Then start the tests with:
80
+
First, copy `.env.template` to a new file in the project's root directory called `.env`. This file will be read by `docker-compose` in the next step. Adjust any of the values in this file if needed, or add new variables for any secret information you need to pass to docker-compose (or to docker containers).
139
81
140
-
```
141
-
$(npm bin)/cypress run --config baseUrl=http://localhost:9000
82
+
```sh
83
+
docker-compose up
142
84
```
143
85
144
-
You could also run these tests in the interactive mode using `compose/test.yml` with the following command:
86
+
Open `http://localhost` in your browser.
145
87
146
-
```
147
-
$(npm bin)/cypress open --config baseUrl=http://localhost:9000
148
-
```
88
+
You can specify environment variables for docker-compose by adding an `.env` file to the root of the project based on `.env.template`.
149
89
150
-
Note that this is similar to the previous command, but we are using the `open` command instead of the `run` command.
90
+
### Social Authentication Keys
151
91
152
-
### Using `gitlab-runner`
92
+
To use social sign on in local development, you will need to create an application with the given provider such as GitHub, Google, Facebook, etc.
153
93
154
-
It can be useful to debug your `.gitlab-ci.yml` jobs before pushing changes to GitLab. This gives us a faster feedback loop, and it doesn't use any of the CI minutes on your free (2000 minutes/month) or paid GitLab plans.
94
+
#### GitHub
155
95
156
-
There is a little bit of setup needed to run Cypress tests with `gitlab-runner`.
96
+
Go to [https://github.com/settings/applications/new](https://github.com/settings/applications/new), and add the following:
157
97
158
-
First, you need to install `gitlab-runner` on your machine.
98
+
- Application Name: A name for the development application, such as `My App Dev`
99
+
- Homepage URL: `http://localhost`
100
+
- Application description: (optional)
101
+
- Authorization callback URL `http://localhost/auth/github/callback` (this route is defined in `quasar/src/router/routes.js`)
159
102
160
-
Next, you need to start a local registry on you r computer. Run the following command ([taken from docker documentation](https://docs.docker.com/registry/deploying/)):
103
+
In the `.env` file, add the `Client ID` of your GitHub OAuth App as the `GITHUB_KEY` variable, and add the `Client Secret` as the `GITHUB_SECRET` variable.
161
104
162
-
```
163
-
docker run -d -p 5000:5000 --restart=always --name registry registry:2
164
-
```
105
+
## VuePress Documentation
165
106
166
-
Next, you need to build the production image that you will use in the test. To do this, run the following command:
107
+
This project uses VuePress for documentation. To view the documentation site locally, run the following command:
167
108
168
-
```
169
-
docker-compose -f compose/test.yml build backend
109
+
```bash
110
+
docker-compose -f compose/docs.yml up --build
170
111
```
171
112
172
-
Then tag the image with the following command:
113
+
This will make the docs available at `http://localhost:8082/docs/`. Hot-reloading through websockets is supported, so changes will show up as they are saved in your code editor.
173
114
174
-
```
175
-
docker tag compose_backend:latest localhost:5000/backend:latest
176
-
```
115
+
### Access Django Shell in Jupyter Notebook
177
116
178
-
Then push the tagged image to the local registry:
117
+
With all containers running, run the following commands:
179
118
180
119
```
181
-
docker push localhost:5000/backend:latest
120
+
docker exec -it backend bash
121
+
# cd notebooks/
122
+
# ../manage.py shell_plus --notebook
182
123
```
183
124
184
-
Now that we have pushed the production image to our local registry, we can run the `e2e-local` job with `gitlab-runner`.
185
-
186
-
To do this, we need to make sure that the `e2e-local` job defined in `.gitlab-ci.yml` is not commented. To do this, remove the `.` in front of the job name (change `.e2e-local` to `e2e-local`).
187
-
188
-
Then, commit your changes in git. Gitlab runner requires that you commit changes before running tests. Run the GitLab CI job with the following command:
0 commit comments