Skip to content

Commit

Permalink
Merge pull request fourTheorem#69 from fourTheorem/feature/share-list…
Browse files Browse the repository at this point in the history
…-#37

Merge share list fourTheorem#37
  • Loading branch information
eoinsha authored Aug 7, 2019
2 parents a6ca4b6 + 726c23d commit 8c39a3f
Show file tree
Hide file tree
Showing 149 changed files with 15,509 additions and 2,822 deletions.
3 changes: 2 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,7 @@ coverage
build/Release

**/.idea
.serverless
.serverless/
*.sw?
vim-markdown-preview.html
.vscode/
Expand All @@ -58,3 +58,4 @@ pipeline-state.env\
aws-accounts.json
slic-config.json


48 changes: 26 additions & 22 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,6 @@

<!-- /TOC -->


## How does SLIC starter help you?

1. Serverless development involves a lot of decisions around which approach to take for a multitude of issues. It aims to remove 80% of this decision making and let you focus on building valuable features.
Expand All @@ -51,11 +50,13 @@
This project is free to use by enterprise, startups, students, educators, enthusiasts and skeptics alike. We actively encourage contributions, suggestions and questions from _anyone_.

## Application Architecture

![SLIC Starter Architecture](./architecture.png)

For the CICD pipeline architecture, see [CI/CD](#cicd)

## What does it provide?

SLIC Starter is a complete, working application. By including all the aspects of a real application, SLIC Starter goes beyond a typical demo project. You are encouraged to:

- Take SLIC Starter and copy it
Expand Down Expand Up @@ -89,6 +90,7 @@ Authentication is a difficult problem with constantly-evolving security requirem
### Data Access with a RESTful API

SLIC Starter includes useful, working examples of typical CRUD (create, read, update and delete) actions with a REST API. Currently, these APIs are implemented with DynamoDB and the `DocumentClient` API. For an example of this, look at the implementation of the [Checklist](./checklist-service) service.
Upon first deploying SLIC Starter, the API Service will create Record Sets and an API Domain Name for your API. APIGateway and Route53 handles the creation of these resources for us.

### Messaging

Expand Down Expand Up @@ -130,11 +132,12 @@ All tests can be run in local development mode as well as against a fully-deploy
For details on integration (API) tests, see the [README.md in integration-tests](./integration-tests/README.md)

### Monitoring

_Work in progress_

### Logging

As a default log centralization solution, SLIC Starter publishes logs to [logz.io](https://logz.io). The [logging](./logging) module handles this using the logz.io forwarder. This is integrated into each service's Lambda function using the [serverless-log-forwarding](https://github.com/amplify-education/serverless-log-forwarding) plugin.
As a default log centralization solution, SLIC Starter publishes logs to [logz.io](https://logz.io). The [logging](./logging) module handles this using the logz.io forwarder. This is integrated into each service's Lambda function using the [serverless-log-forwarding](https://github.com/amplify-education/serverless-log-forwarding) plugin.

For further details, see the [logging README](./logging/README.md)

Expand All @@ -148,7 +151,7 @@ _Coming soon_. SLIC Starter will include support for roles and Role-Based Access

## Before you Begin!

SLIC Starter is designed to get you up in running with a real-world application as quickly as possible. The fact that we go beyond the average sample application, there is a bit more involved in getting to production. For example:
SLIC Starter is designed to get you up in running with a real-world application as quickly as possible. The fact that we go beyond the average sample application, there is a bit more involved in getting to production. For example:

1. We assume that you want to keep the CICD, staging and production accounts separate. These can be set up under one root account using [AWS Organizations](https://aws.amazon.com/organizations/).
2. SLIC Starter assumes you are using a registered domain (like `sliclists.com`) and will set up DNS entries for use in production (like `api.sliclists.com`) and staging (`stg.sliclists.com`, `api.stg.sliclists.com`).
Expand All @@ -165,15 +168,16 @@ To set up deployment to your own accounts, first run through these steps.
3. Fork the repository into your own account or organization on GitHub. If you don't use GitHub, you will have to tweak the source project in the CICD module ([source-project.ts](./cicd/lib/project/source-project.ts))
4. Enable CodeBuild to access your GitHub repo. The only way to do this is to create a temporary CodeBuild project in your CICD account and set up your GitHub repostitory as a source. Grant access to your GitHub repo. Your account now has access to the repo and the SLIC Starter CodeBuild will be able to monitor and clone your repo. The temporary CodeBuild project can already be deleted. You will need to have **admin** priveleges for the repository or **owner** permissions for the GitHub organization in order for WebHooks to be create automatically by [the CodeBuild project](./cicd/lib/project/source-project.ts).
5. (Optional - this will be required for repo tagging). Set up GitHub authentication for your repo. Create a GitHub Personal Access Token and add it as an secret with the name `GitHubPersonalAccessToken` in Secrets Manager _in the CICD account_. See [this post](https://medium.com/@eoins/securing-github-tokens-in-a-serverless-codepipeline-dc3a24ddc356) for more detail on this approach.
6. Edit the account IDs in `cicd/cross-account/serverless.yml` and `cicd/config.ts`.
7. Create a [Mailosaur](https://mailosaur.com) account. Take the server ID and API key and add them in your CICD account to the Parameter Store as `SecretString` values with the following names

* `test/mailosaur/serverId`
* `test/mailosaur/apiKey`
6. Under the parent directory, open the file `common-config.json`. If desired, you can configure the default regions and domain prefixes by account. If you have no reason to change them, leave the default values as they are.
7. Edit the account IDs in `cicd/cross-account/serverless.yml` and `cicd/config.ts`.
8. Create a [Mailosaur](https://mailosaur.com) account. Take the server ID and API key and add them in your CICD account to the Parameter Store as `SecretString` values with the following names

These are picked up by the integration and end-to-end test CodeBuild projects.
- `test/mailosaur/serverId`
- `test/mailosaur/apiKey`
These are picked up by the integration and end-to-end test CodeBuild projects.

8. Give permissions for your CICD account to deploy to staging and production accounts.
9. Create a secret string in System Manager Parameter store with a value used to sign and verify verification codes - the parameter name should be `/STAGE/sharing-service/code-secret` where STAGE is the stage you are deploying to (dev, stg or prod).
10. Give permissions for your CICD account to deploy to staging and production accounts.

```
npm install -g serverless
Expand All @@ -182,7 +186,7 @@ AWS_PROFILE=your-staging-account serverless deploy
AWS_PROFILE=your-production-account serverless deploy
```

9. Deploy the CI/CD pipeline to your CICD account.
11. Deploy the CI/CD pipeline to your CICD account.

```
cd cicd
Expand All @@ -191,15 +195,15 @@ npm run cdk -- bootstrap <<AWS_ACCOUNT_ID>>/eu-west-1
AWS_PROFILE=your-cicd-account npm run deploy
```

10. Trigger your pipeline by commiting your changes to the repository
11. Monitor your deployment by viewing the orchestrator pipeline in the AWS Console CodePipeline page.
12. Wait for your deployment to fail! _Wait, what?_ Yes, your first deployment will fail. This is expected and all part of the process. Read on to find out more!
12. Trigger your pipeline by commiting your changes to the repository
13. Monitor your deployment by viewing the orchestrator pipeline in the AWS Console CodePipeline page.
14. Wait for your deployment to fail! _Wait, what?_ Yes, your first deployment will fail. This is expected and all part of the process. Read on to find out more!

## Getting to your First Successful Deployment

The CICD process attempts to build and deploy each service in parallel. This is done so you get quick feedback and to improve the overall deployment speed. It also means that deployment can fail if there are dependencies between services. Out of the box, SLIC Starter has a `baseline` module that sets up a Route 53 Hosted Zone and some certificates. These are required by the `frontend` and `checklist-service` services, so those builds will fail if the cerificates aren't ready yet. This is just one example. There are other services that depend on common resources so it will require a few retries in both staging and production before everything is deployed.
The CICD process attempts to build and deploy each service in parallel. This is done so you get quick feedback and to improve the overall deployment speed. It also means that deployment can fail if there are dependencies between services. Out of the box, SLIC Starter has a `certs` module that sets up a Route 53 Hosted Zone and some certificates. These are required by the `frontend` and `checklist-service` services, so those builds will fail if the cerificates aren't ready yet. This is just one example. There are other services that depend on common resources so it will require a few retries in both staging and production before everything is deployed.

*Note* that deployment of some services can take quite some time! In particular, `frontend` deployment will wait until the CloudFront distribution has been created. This can take _at least_ 15 minutes.
_Note_ that deployment of some services can take quite some time! In particular, `frontend` deployment will wait until the CloudFront distribution has been created. This can take _at least_ 15 minutes.

You can inspect the failures in the Orchestrator Pipeline view in the CodePipeline console of your CICD account. You can retry the `stgDeploy` phase by clicking the _Retry_ button in the pipeline.

Expand All @@ -220,7 +224,6 @@ Because we use Route 53 for our DNS records, the verification process is quite s

By default, SES will require validation of each email address to which emails are being sent. To avoid this, you can [request a sending limit increase](https://docs.aws.amazon.com/ses/latest/DeveloperGuide/request-production-access.html), which will remove your account/region from the SES Sandbox.


## Local Development

Work on a more seamless local development environment is _in progress_. In the meantime, here's an introduction to running the `checklist-service` locally:
Expand Down Expand Up @@ -259,12 +262,13 @@ SLIC Starter is open source and contributions are welcome from everyone. It was
## Troubleshooting

1. I get this error in the CodeBuild source project:
```
87/101 | 1:05:21 PM | CREATE_FAILED | AWS::CodeBuild::Project |
sourceProject (sourceProjectBCA86C81) Failed to call CreateWebhook, reason: Repository not found or permission denied.
```

- This is because your CodeBuild configuration does not have access to your GitHub account. To grant access, create a CodeBuild project manually in the AWS Console and grant access to your repository. This project can be safely deleted afterwards once CodeBuild has been granted access.
```
87/101 | 1:05:21 PM | CREATE_FAILED | AWS::CodeBuild::Project |
sourceProject (sourceProjectBCA86C81) Failed to call CreateWebhook, reason: Repository not found or permission denied.
```

- This is because your CodeBuild configuration does not have access to your GitHub account. To grant access, create a CodeBuild project manually in the AWS Console and grant access to your repository. This project can be safely deleted afterwards once CodeBuild has been granted access.

## Contributing

Expand Down
6 changes: 6 additions & 0 deletions api-service/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
# package directories
node_modules
jspm_packages

# Serverless directories
.serverless
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ module.exports = serverless => {

const values = {}
const region = 'us-east-1'
const stackName = `baseline-${stage}`
const stackName = `certs-${stage}`
const exports = {
[`${stage}-api-cert`]: 'apiCert',
[`${stage}-public-hosted-zone`]: 'publicHostedZone'
Expand Down
47 changes: 47 additions & 0 deletions api-service/serverless.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
service: api-service

provider:
name: aws
runtime: nodejs8.10
stage: ${opt:stage, 'local'}
region: ${opt:region, '${self:custom.defaultRegions.${self:provider.stage}}'}

custom:
apiConfig: ${file(./api-config.js)}
nsDomain: ${env:SLIC_NS_DOMAIN}
domainPrefixes: ${file(../common-config.json):domainPrefixes}
defaultRegions: ${file(../common-config.json):defaultRegions}
apiDomainName: api.${self:custom.domainPrefixes.${self:provider.stage}}${self:custom.nsDomain}
# The HostedZoneId for CloudFront distributions is always this value.
# See http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-route53-aliastarget.html
# AND http://docs.aws.amazon.com/general/latest/gr/rande.html
cloudFrontHostedZoneId: Z2FDTNDATAQYW2

resources:
Resources:
apiCustomDomain:
Type: AWS::ApiGateway::DomainName
Properties:
CertificateArn: ${self:custom.apiConfig.apiCert}
DomainName: ${self:custom.apiDomainName}

apiDomainDns:
Type: AWS::Route53::RecordSetGroup
Properties:
HostedZoneId: ${self:custom.apiConfig.publicHostedZone}
RecordSets:
- Name: ${self:custom.apiDomainName}
Type: A
AliasTarget:
DNSName: !GetAtt apiCustomDomain.DistributionDomainName
HostedZoneId: ${self:custom.cloudFrontHostedZoneId}
- Name: ${self:custom.apiDomainName}
Type: AAAA
AliasTarget:
DNSName: !GetAtt apiCustomDomain.DistributionDomainName
HostedZoneId: ${self:custom.cloudFrontHostedZoneId}
Outputs:
ApiDistributionDomainName:
Value: !GetAtt apiCustomDomain.DistributionDomainName
Export:
Name: ${self:provider.stage}-api-distribution-domain-name
Binary file modified architecture.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
10 changes: 0 additions & 10 deletions baseline/custom.yml

This file was deleted.

12 changes: 8 additions & 4 deletions baseline/serverless.yml → certs/serverless.yml
Original file line number Diff line number Diff line change
@@ -1,12 +1,16 @@
service: baseline
service: certs

provider:
name: aws
runtime: ${self:custom.runtime}
stage: ${self:custom.stage}
runtime: nodejs8.10
stage: ${opt:stage, 'dev'}
region: us-east-1

custom: ${file(./custom.yml)}
custom:
nsDomain: ${env:SLIC_NS_DOMAIN}
domainPrefixes: ${file(../common-config.json):domainPrefixes}
apiDomainName: api.${self:custom.domainPrefixes.${self:provider.stage}}${self:custom.nsDomain}
siteDomainName: ${self:custom.domainPrefixes.${self:provider.stage}}${self:custom.nsDomain}

resources:
Outputs:
Expand Down
22 changes: 0 additions & 22 deletions checklist-service/custom.yml

This file was deleted.

23 changes: 0 additions & 23 deletions checklist-service/lib/event-dispatcher.js

This file was deleted.

3 changes: 0 additions & 3 deletions checklist-service/lib/log.js

This file was deleted.

7 changes: 0 additions & 7 deletions checklist-service/lib/test.js

This file was deleted.

Loading

0 comments on commit 8c39a3f

Please sign in to comment.