Deployment

To deploy and maintain Workbench, get comfortable with the wonderful world of containers.

Creating Docker images

Our Google Cloud Build builder publishes images to gcr.io/workbenchdata-ci/IMAGE:SHA1, where IMAGE is:

• frontend -- serves Web requests
• fetcher -- runs "fetch" steps (the first steps of tabs, which request data from elsewhere)
• renderer -- renders all steps (producing tables)
• tusd-hooks -- helps with file uploads
• cron -- schedules runs automated updates
• The cron image also powers cron-expired-session-deleter -- deletes old workflows that weren't created by logged-in users
• The cron image also powers cron-lesson-workflow-deleter -- deletes old workflows that were created for lessons (since changes to the underlying lessons can make a bad user experience)
• The cron image also powers cron-delta-deleter -- deletes old Undo operations (limits database cruft and empowers us to gradually revamp Workbench's innards)

SHA1 is the revision of the Git commit.

You can build your own images: PROJECT_ID=workbenchdata-ci TAG=dev docker buildx bake -f docker-bake.hcl

Deploy process

Deploy with Kubernetes.

See cjworkbench-deploy for deployment scripts.

Keeping modules up-to-date

Every workbench "Step" is built on a "Module". For instance, "Filter for column A value is null" is a "Step", and "filter" is the "Module".

To refresh the generic list of modules, run cjworkbench-deploy/refresh-suggested-modules ENVIRONMENT.

Debugging

Most debugging sessions start like this. It's a good introduction to every facet of our environment:

kubectl exec -it frontend-deployment-xxxxxxxx-xxxxx app -- /opt/venv/django/bin/python ./manage.py shell

• kubectl exec -it: we use Kubernetes
• frontend-deployment-xxxxxxxx-xxxxx: when IMAGE=frontend, the Kubernetes deployment is called frontend-deployment.
• app: all our pods have a "main" container called app.
• /opt/venv/django: our Django images all have an /opt/venv/django Python virtualenv. Images that also invoke module code (renderer, fetcher and -- for now -- frontend and tusd-hooks) have another virtualenv, /opt/venv/cjwkernel, which powers module code.
• bin/python: Our Django images all use Python 3.
• ./manage.py: Django's startup script. Each image has a slightly different settings.py
• shell: Django command for a Python prompt. dbshell is another handy command, for a PostgreSQL prompt.

Viewing server logs

kubectl logs frontend-deployment-xxxxxxxx-xxxxx --follow ... or use your cloud environment's tools.