Merge branch 'master' into chmreid-secrets-infra

* master:
  Fix failing collections tests
  Add pagination response headers (closes #2287)
  Fix improper markdown (#2367)

Author: chmreid

CONTRIBUTING.md

@@ -1,30 +1,31 @@
 # Contributing Guidelines
+
 Contributions to HCA working group repos are subject to overall HCA
 governance and technical guidance. In addition, contributors are
-expected to abide by the following guidelines.
+expected to abide by the following guidelines:
 
 1. Rough consensus and running code: instead of formal procedures for
-  agreeing on everything, systems with working prototypes and existing
-  users are prioritized as platforms for discussion.
+   agreeing on everything, systems with working prototypes and existing
+   users are prioritized as platforms for discussion.
 
 1. Keep it simple: prioritize scalability and the ability to keep the
-  project easy to understand and scale over features. Use
-  well-supported upstream solutions where possible. Provide useful
-  defaults and don't expose unnecessary configuration options.
+   project easy to understand and scale over features. Use
+   well-supported upstream solutions where possible. Provide useful
+   defaults and don't expose unnecessary configuration options.
 
 1. Separation of concerns and layers: (TODO: elaborate)
 
 1. Don't break the build: pull requests are expected to pass all
-  automated CI checks.
+   automated CI checks.
 
 1. Keep the build simple: Automated CI checks that are fragile or don't
-  serve a clear agreed upon purpose will be removed.
+   serve a clear agreed upon purpose will be removed.
 
 1. All Pull Request comments must be addressed, even after merge.
 
 1. All code must be reviewed by at least 1 other team member.
 
-1. All Pull Requests relating to an issue must include "connected to 
-  #123", where #123 is the issue number.
+1. All pull requests relating to an issue must include "connected to 
+   \#123", where \#123 is the issue number.
 
 1. Individual commit messages should clearly express the commit's purpose.

ISSUE_TEMPLATE.md

@@ -1,24 +1,39 @@
-<!-- IF YOU ARE FILING A BUG, please uncomment/use the markdown below and apply the 'bug' label -->
+<!-- 
+**IF YOU ARE FILING A BUG**, please uncomment/use the markdown below
+and apply the 'bug' label.
+-->
+
 <!--
 ## Software version (git hash, date, branch)
 ## Expected behavior
 ## Actual behavior
 ## Steps to reproduce the behavior
 -->
 
-<!-- IF YOU ARE FILING A FEATURE REQUEST, please uncomment/use the mardown below -->
 <!--
+**IF YOU ARE FILING A FEATURE REQUEST**, please uncomment/use the markdown below
+-->
+
+<!--
+
 ## User story
+
 * As a <type of user>
 * I want to <action>
 * so that <result>
 
 ## Definition of done
+
 -->
 
 <!--
-Other best practices
-* Use the following GitHub keyword if your PR completely resolves an issue: "Fixes #123"
-* If you are on the core dev team for this repo, mention your issue in daily scrum, sprint review, or to the PM.
+
+Other best practices:
+
+* Use the GitHub keywords "fixes" or "closes" if your PR completely resolves an issue:
+  "Fixes #123" or "Closes #456"
+* If you are on the core dev team for this repo, mention your issue in daily
+  scrum, sprint review, or to the PM.
 * Also, assign the issue to someone or to yourself.
+
 -->

PULL_REQUEST_TEMPLATE.md

@@ -1,14 +1,36 @@
-<!-- Please make sure to provide a meaningful title for your PR. Do not keep the default title. -->
-<!-- Use the following GitHub keyword if your PR completely resolves an issue: "Fixes #123" -->
+<!--
+Please make sure to provide a meaningful title for your PR. 
+Do not keep the default title.
+-->
 
-<!-- Describe how you tested this PR, and how you will test the feature after it is merged. Uncomment below:
-### Test plan -->
+<!--
+Use GitHub keywords "fixes" or "closes" followed by an issue number if your PR
+completely resolves an issue:
+"Fixes #123" or "Closes #456"
+-->
 
-<!-- Describe any special instructions to the operator who will deploy your code to the integration, 
-     staging and production deployments. Uncomment below:
-### Deployment instructions & migrations -->
+<!-- 
+Describe how you tested this PR, and how you will test the feature after it is
+merged. Uncomment below:
 
-<!-- Add notes to highlight the feature when it's released/demoed. Uncomment the headings below:
-### Release notes -->
+### Test plan
+-->
 
-<!-- Do you want your PR to be merged by the reviewer using squash or rebase merging? If so, mention it here. -->
+<!--
+Describe any special instructions to the operator who will deploy your code to
+the integration, staging and production deployments. Uncomment below:
+
+### Deployment instructions & migrations
+-->
+
+<!-- 
+Add notes to highlight the feature when it's released/demoed. Uncomment the
+headings below:
+
+### Release notes
+-->
+
+<!--
+Do you want your PR to be merged by the reviewer using squash or rebase
+merging? If so, mention it here.
+-->

README.md

@@ -23,9 +23,11 @@ implements [Step Functions](https://aws.amazon.com/step-functions/) to orchestra
 as large file writes. You can find the API documentation and give it a try [here](https://dss.data.humancellatlas.org/).
 
 #### [Architectural Diagram](https://www.lucidchart.com/documents/view/b65c8898-46e3-4560-b3b2-9e85f1c0a4c7)
+
 ![DSS Sync SFN diagram](https://www.lucidchart.com/publicSegments/view/43dfe33a-47c9-466b-9cb6-6d941a406d8f/image.png)
 
 #### DSS API
+
 The DSS API uses [Swagger](http://swagger.io/) to define the [API specification](dss-api.yml) according to the
 [OpenAPI 2.0 specification](https://github.com/OAI/OpenAPI-Specification/blob/master/versions/2.0.md).
 [Connexion](https://github.com/zalando/connexion) is used to map the API specification to its implementation in Python.
@@ -71,13 +73,15 @@ to review and edit the API specification. When the API is live, the spec is also
       * [Contributing](#contributing)
 
 ## Getting Started
+
 In this section, you'll configure and deploy a local API server and your own suite of cloud services to run a
 development version of the DSS.
 
 Note that privileged access to cloud accounts (AWS, GCP, etc.) is required to deploy the data-store. IF your deployment fails
 due to access restrictions, please consult your local systems administrators.
 
 ### Install Dependencies
+
 The DSS requires Python 3.6 to run.
 
 Clone the repo and install dependencies:
@@ -91,7 +95,6 @@ Also install [terraform from Hashicorp](https://www.terraform.io/) from your fav
 
 ### Configuration
 
-
 The DSS is configured via environment variables. The required environment variables and their default values
 are defined in the file [`environment`](environment). To customize the values of these environment variables:
 
@@ -247,6 +250,7 @@ When deploying for the first time, a Google Cloud Platform service account must
 ### Setting admin emails
 
 Set admin account emails within AWS Secret Manager
+
 ```
 echo ' ' |  ./scripts/dss-ops.py secrets set --secret-name $ADMIN_USER_EMAILS_SECRETS_NAME
  ```
@@ -263,42 +267,47 @@ usual plan/review Terraform workflow, and should therefore be lightweight in nat
 added to `$DSS_HOME/infra` instead.
 
 ##### Resources
+
 Cloud resources have the potential for naming collision in both [AWS](https://docs.aws.amazon.com/general/latest/gr/aws-arns-and-namespaces.html)
  and [GCP](https://cloud.google.com/storage/docs/naming), ensure that you rename resources as needed.
 
 #### Buckets
 
 Buckets within AWS and GCP need to be available for use by the DSS. Use Terraform to setup these resources:
 
-
 ```
 make -C infra COMPONENT=buckets plan
 make -C infra COMPONENT=buckets apply
 ```
 
 #### ElasticSearch
+
 The AWS Elasticsearch Service is used for metadata indexing. Currently the DSS uses version 5.5 of ElasticSearch. For typical development deployments the
 t2.small.elasticsearch instance type is sufficient. Use the [`DSS_ES_`](./docs/environment/README.md) variables to adjust the cluster as needed. 
 
 Add allowed IPs for ElasticSearch to the secret manager, use comma separated IPs:
+
 ```
 echo ' ' | ./scripts/dss-ops.py secret set --secret-name $ES_ALLOWED_SOURCE_IP_SECRETS_NAME
-
 ```
+
 Use Terraform to deploy ES resource:
+
 ```
 make -C infra COMPONENT=elasticsearch plan
 make -C infra COMPONENT=elasticsearch apply
 ```
 
 #### Certificates
+
 A certificate matching your domain must be registered with
 [AWS Certificate Manager](https://docs.aws.amazon.com/acm/latest/userguide/acm-overview.html). Set `ACM_CERTIFICATE_IDENTIFIER`
 to the identifier of the certificate, which can be found on the AWS console.
 
 An AWS route53 zone must be available for your domain name and configured in `environment`.
 
 #### Deploying
+
 Now deploy using make:
 
     make plan-infra
@@ -314,8 +323,9 @@ And you should be able to list bundles like this:
     curl -X GET "https://<domain_name>/v1/bundles" -H  "accept: application/json"
 
 #### Monitoring
- Please see the [data-store-monitor](https://www.github.com/humancellatlas/data-store-monitor) repo for additional
- monitoring tools.
+
+Please see the [data-store-monitor](https://www.github.com/humancellatlas/data-store-monitor) repo for additional
+monitoring tools.
 
 ### CI/CD with Travis CI and GitLab
 
@@ -387,14 +397,17 @@ indexed metadata.
     '
 
 ## Running Tests
+
 1. Check that software packages required to test and deploy are available, and install them if necessary:
 
     `make --dry-run`
 
 1. Populate text fixture buckets with test fixture data _**(This command will completely empty the given buckets** before populating them with test fixture data, please ensure
-the correct bucket names are provided)_:
+the correct bucket names are provided)**_:
 
-    `tests/fixtures/populate.py --s3-bucket $DSS_S3_BUCKET_TEST_FIXTURES --gs-bucket $DSS_GS_BUCKET_TEST_FIXTURES`
+    ```
+    tests/fixtures/populate.py --s3-bucket $DSS_S3_BUCKET_TEST_FIXTURES --gs-bucket $DSS_GS_BUCKET_TEST_FIXTURES
+    ```
 
 1. Set the environment variable `DSS_TEST_ES_PATH` to the path of the `elasticsearch` binary on your machine.
 
@@ -485,12 +498,15 @@ test-deploy-test cycle after this (the test after the deploy is required to test
     ```
 
 ### Enabling Profiling
+
 AWS Xray tracing is used for profiling the performance of deployed lambdas. This can be enabled for `chalice/app.py` by
 setting the lambda environment variable `DSS_XRAY_TRACE=1`. For all other daemons you must also check
 "Enable active tracking" under "Debugging and error handling" in the AWS Lambda console.
 
 ## Security Policy
+
 See our [Security Policy](https://github.com/HumanCellAtlas/.github/blob/master/SECURITY.md).
 
 ## Contributing
+
 External contributions are welcome. Please review the [Contributing Guidelines](CONTRIBUTING.md)

daemons/dss-dlq-reaper/README.md

@@ -13,6 +13,7 @@ This daemon is a part of the DLQ-based framework for reprocessing failed Lambda
 #### Enabling DLQ-based retries for DSS daemons Lambdas
 
 In order to enable DLQ-based reprocessing for DSS daemons each daemon needs to be configured individually. 
+
 - Locate config.json file in the daemon's .chalice directory
 - Add the following entry to the `config.json` file `"dead_letter_queue_target_arn": "",`. 
 - The entry needs to be created at the top level of the json attribute hierarchy. During deployment the value would be replaced with approriate SQS queue name. 

daemons/dss-gs-event-relay/Readme.md

@@ -1,13 +1,18 @@
 #### Activating Google Cloud APIs
+
 ```
 gcloud service-management enable cloudfunctions.googleapis.com
 gcloud service-management enable runtimeconfig.googleapis.com
 ```
 
 #### Retrieving GCF logs
-`gcloud beta functions logs read dss-gs-event-relay`
+
+```
+gcloud beta functions logs read dss-gs-event-relay
+```
 
 #### Example Google Storage event
+
 ```
 {
   "timestamp": "2017-08-25T23:08:25.270Z",
@@ -35,6 +40,7 @@ gcloud service-management enable runtimeconfig.googleapis.com
 ```
 
 #### Environment variables in the GCF container
+
 ```
 {
   "WORKER_PORT": "8091",

daemons/dss-notify-v2/Readme.md

@@ -93,6 +93,7 @@ The bundle metadata document format for a new bundle or version is is
 ```
 
 For a tombstone it is
+
 ```
 {
   'event_type': "TOMBSTONE",
@@ -104,6 +105,7 @@ For a tombstone it is
 ```
 
 For a deleted bundle it is
+
 ```
 {
   'event_type': "DELETE",

daemons/dss-scalability-test/README.md

@@ -1,29 +1,29 @@
 # HCA DSS: Scalability Testing framework
 
 This is a scalability test framework for the replicated data storage system (aka the "blue box") of
-the [Human Cell Atlas](https://www.humancellatlas.org/). 
+the [Human Cell Atlas](https://www.humancellatlas.org/).
 
 #### About the scalability testing framework
 
 1. The scalability test framework is based on AWS Step Functions. Workflow definition resembles smoke test for DSS.
-2. The execution is triggered by sending SNS messages to dss-scalability-test-run-{STAGE} topic
-3. The scalability test writes results of execution of individual executions and aggregated run metrics into the 
-following DynamoDB tables: scalability_test_result, scalability_test
-4. The SFN execution is initiated and starts by entering WAIT step. Wait is configured to end at the 5 minute intervals 
-to accommodate the AWS limit on starting SFN executions and enable generation of bursts of load
-5.  Once all parallel branches of execution are done, it writes summary of the run in DynamoDB
-6. DynamoDB is configured to stream new records into Lambda which aggregates the results and writes incremental metrics
-back into DynamoDB
-7. CloudWatch dashboard has been configured to display relevant execution metrics and deployed automatically. The 
-dashboard is named as "Scalability{stage}" 
+1. The execution is triggered by sending SNS messages to `dss-scalability-test-run-{STAGE}` topic
+1. The scalability test writes results of execution of individual executions and aggregated run metrics into the
+   following DynamoDB tables: `scalability_test_result`, `scalability_test`
+1. The SFN execution is initiated and starts by entering WAIT step. Wait is configured to end at the 5 minute intervals
+   to accommodate the AWS limit on starting SFN executions and enable generation of bursts of load
+1. Once all parallel branches of execution are done, it writes summary of the run in DynamoDB
+1. DynamoDB is configured to stream new records into Lambda which aggregates the results and writes incremental metrics
+   back into DynamoDB
+1. CloudWatch dashboard has been configured to display relevant execution metrics and deployed automatically. The
+   dashboard is named as `Scalability{stage}`
 
 #### Running the scale test locally
 
 * Run with default configuration `make scaletest` in the top-level `data-store` directory.
-* Run with custom configuration './tests/scalability/scale_test_runner.py -r <rps> -d <seconds>' in the 
-top-level `data-store` directory. Where "rps" is a number of requests generated per second and "d" is a duration of the
-test in seconds
+* Run with custom configuration `./tests/scalability/scale_test_runner.py -r <rps> -d <duration_sec>` in the
+  top-level `data-store` directory, where `<rps>` is a number of requests generated per second and 
+  `<duration_sec>` is the duration of the test in seconds.
 
 #### Adding new tests
 
-New tests can easily be addeed to the existing step function definition at app.py
+New tests can easily be addeed to the existing step function definition at `app.py`.