Skip to content

Latest commit

 

History

History

backend

README.md

ATIP backend on GCP

This package is a backend to serve ATIP on App Engine. See architecture for details. It's a separate package from the ATIP frontend in the rest of this repo, and could logically be split into a separate git repo if useful.

Deployment guide

This section is a WIP right now; only use the result for testing! Don't upload any non-public data yet.

Unless other specified, each command below should only take a few seconds to run.

Note the URLs won't work; you have to manually replace $PROJECT.

Set up the GCP project

  1. Install gcloud
  2. Hook up gcloud to your GCP account
  3. Pick a project ID (which'll also be used as the GCS bucket name) and set it as a shell variable: PROJECT=atip-test-2
  4. Create it: gcloud projects create $PROJECT
  5. Manually link the billing account: https://console.cloud.google.com/billing/linkedaccount?project=$PROJECT
    • You may need to configure a billing account first for your GCP account
    • Without this, gcloud app --project=$PROJECT describe gives a permissions error
  6. Set up App Engine: gcloud app --project=$PROJECT create --region=europe-west
    • Note there's only one GAE app per project, unlike most other GCP services
    • And the region is immutable once you set it for the project!

Create the GCS bucket

Names and regions should match above.

  1. gcloud storage --project=$PROJECT buckets create gs://$PROJECT --location=EUROPE-WEST2 --uniform-bucket-level-access
  2. Generate fake scheme data and upload it: npm run generate-random-schemes && gsutil cp random_schemes.geojson gs://$PROJECT/private_layers/v1/
  3. Sync current public layers to GCS. This only works if you have S3 access (aka Dustin). Skip these otherwise, or download another way. This might take a few minutes, depending on your connection: aws s3 sync s3://atip.uk/layers layers; gsutil -m cp -r ./layers gs://$PROJECT/
  4. Also copy private files (currently only from Ordnance Survey) into gs://$PROJECT/private_layers/v1/. These have to be generated by a developer; they can't be mirrored from S3, because our S3 bucket is only appropriate for public data.
  5. Go to https://osdatahub.os.uk/projects/ATIP_scheme_browser (or enable the OS Maps API in a new project) and copy the API key. Also get a Bluesky APGB key. Create a file api_keys.json containing {"ordnance_survey": "the key", "bluesky": "the key"}, and upload it to the GCS bucket at gs://$PROJECT/private_layers/api_keys.json.

Deploy

  1. Create backend/app.yaml by copying e.g. backend/app_public_dev.yaml and setting the values according to your own project
  2. Run gcloud projects describe $PROJECT | grep projectNumber and use the result to update PROJECT_NUMBER in backend/app.yaml
  3. Set a env var for the vite resource base url, this depends if you're deploying to internal or public version. If public BASE_URL=https://{env.}plan.activetravelengland.gov.uk. If the internal version then BASE_URL=https://$PROJECT.ew.r.appspot.com.
  4. Create the files to deploy: VITE_RESOURCE_BASE="$BASE_URL/data" npm run build && cd backend && rm -rf dist && cp -R ../dist .
    • Note we could make Cloud Build do this, but we'd have to get wasm-pack and other things set up there first
    • GH Actions will eventually trigger CI deployments for our test environment, and we've already done the work of configuring that build environment
  5. gcloud app --project=$PROJECT deploy --quiet (takes a minute or two)
  6. Try the result: gcloud app browse --project=$PROJECT or https://$PROJECT.ew.r.appspot.com/browse.html

Useful debugging:

Protect with IAP

  1. gcloud services --project=$PROJECT enable iap.googleapis.com
  2. gcloud services --project=$PROJECT enable cloudresourcemanager.googleapis.com
  3. Go to https://console.cloud.google.com/apis/credentials/consent?project=$PROJECT and manually configure the oauth consent screen
    • Choose "External" (unless everyone internal can be part of our cloud org?)
    • Pick an app name, email
    • No logo, app domains, authorized domains, scopes, or test users
  4. Go to https://console.cloud.google.com/security/iap?project=$PROJECT and enable IAP (it'll have an oauth misconfigured error)

Nobody should have access by default. To get the current auth list:

gcloud iap --project=$PROJECT web get-iam-policy --resource-type=app-engine

To add someone:

gcloud iap --project=$PROJECT web add-iam-policy-binding --resource-type=app-engine --member=user:foo@bar.com --role=roles/iap.httpsResourceAccessor

After changing this, there's seemingly a propagation delay of about a minute. You might also need to Ctrl+Shift+R or clear your browser cache.

Automatically deploy from GH Actions

For the test environment, we can automatically deploy App Engine for every push to the main branch of the Github repo. One-time setup steps to create a service account with permission to deploy, and a service account JSON key:

gcloud services --project=$PROJECT enable appengine.googleapis.com

gcloud iam service-accounts --project=$PROJECT create gh-builder-service-account

# See https://github.com/google-github-actions/deploy-appengine for reference
for role in roles/appengine.appAdmin roles/storage.admin roles/cloudbuild.builds.editor roles/iam.serviceAccountUser; do
	gcloud projects add-iam-policy-binding $PROJECT --member="serviceAccount:gh-builder-v2@$PROJECT.iam.gserviceaccount.com" --role=$role;
done

gcloud iam service-accounts keys create private_key --iam-account=gh-builder-v2@$PROJECT.iam.gserviceaccount.com

You then need to set the GCP_DEV_CREDENTIALS secret in Github. Go to https://github.com/acteng/atip/settings/secrets/actions and create/edit it as needed. The contents are the private_key file created by the last command, but you need to format it as a one-liner. If you have jq, you can just do cat private_key | jq - c` and copy that.

Remove the private_key file after creating the Github secret, and make sure it's never made public

TODO

Unblocked by this: - More basemaps - Load criticals and scheme data - Regional engagement contact list

Appendix

  • For completeness, other options to serve GCS files:
    • ideally: some direct GCS URL that enforces the same IAP auth
    • some LB in front of GCS, then glued to IAP
    • make Express proxy the requests to GCS
      • add the GAE service account as a storage.objectViewer on the bucket first
    • insecure worst case: public bucket with an obfuscated URL
    • signed URLs
    • cookie based auth
  • Consider using the GAE default bucket?

References