Create auto updater

README.md

@@ -4,13 +4,16 @@
 
 This repository contains:
 
- - `ditto clone`: a script to crawl an instance of PokeAPI and download all objects
- - `data/api`: a static copy of the JSON data generated with the above script
- - `ditto analyze`: a script to generate a JSON schema of the above data
- - `data/schema`: a static copy of the PokeAPI schema generated from the above data
- - `ditto transform`: a script to apply a new base url to data in `data/api` 
- - `ditto serve`: a script to serve the data in the same form as PokeAPI
-    - with full support for dynamic pagination using GET args `offset` and `limit`
+ - Ditto script:
+    - `ditto clone`: a script to crawl an instance of PokeAPI and download all objects
+    - `ditto analyze`: a script to generate a JSON schema of the above data
+    - `ditto transform`: a script to apply a new base url to data in `data/api` 
+    - `ditto serve`: a script to serve the data in the same form as PokeAPI
+       - with full support for dynamic pagination using GET args `offset` and `limit`
+ - Static data:
+    - [data/api](data/api): a static copy of the JSON data generated with the above script
+    - [data/schema](data/schema): a static copy of the PokeAPI schema generated from the above data
+    - [updater](updater): a bot that runs in docker and can update the data stored in this repo
 
 ## Docker
 
@@ -21,7 +24,7 @@ just have to run one command.
  - Replace `http://localhost:8080` with the base url of your choice
 
 ``` bash
-docker run -p 8080:80 -e DITTO_BASE_URL=http://localhost:8080 sargunv/pokeapi-ditto
+docker run -p 8080:80 -e DITTO_BASE_URL=http://localhost:8080 sargunv/pokeapi-ditto:latest
 ```
 
 ## Usage
@@ -44,64 +47,4 @@ The general idea is the same.
 
 ## Advanced
 
-You can manually update the data if necessary. If I abandon this
-project, here’s how to update it. It's a bit of an involved process.
-
-Before starting, you’ll need to install [Docker and Docker
-Compose](https://docs.docker.com/compose/install/). You'll
-also need [Poetry](https://poetry.eustace.io/) in your PATH.
-
-First clone the PokeAPI and Ditto repositories:
-
-``` bash
-cd ~
-git clone https://github.com/PokeAPI/ditto.git
-git clone https://github.com/PokeAPI/pokeapi.git
-```
-
-Apply the patch to disable rate limiting on your local PokeAPI:
-
-``` bash
-# Assuming you have the repos in ~
-cd ~/pokeapi
-git apply ~/ditto/extra/disable-rate-limit.patch
-```
-
-Run PokeAPI using docker-compose:
-
-``` bash
-docker volume create --name=redis_data
-docker volume create --name=pg_data
-docker-compose up -d
-```
-
-Build the PokeAPI database:
-
-``` bash
-docker-compose exec app python manage.py migrate
-docker-compose exec app python manage.py shell
-```
-
-``` python
-from data.v2.build import build_all
-build_all()
-```
-
-Once it’s done, you can update Ditto’s data:
-
-``` bash
-cd ~/ditto
-rm -r ./data
-poetry install
-poetry run ditto clone --src-url http://localhost/ --dest-dir ./data
-poetry run ditto analyze --api-dir ./data/api --schema-dir ./data/schema
-```
-
-This will crawl your local instance of PokeAPI, copy all the data to
-./data, and regenerate the schema.
-
-Once that's finished, you can serve the freshly updated data!
-
-``` bash
-poetry run ditto serve --port 8080 --base-url http://localhost:8080
-```
+You can manually update the data if necessary. See [the updater bot](updater). You can run the bot in docker, or read and adapt its update script yourself. 

updater/Dockerfile

@@ -0,0 +1,23 @@
+FROM docker:dind
+
+RUN apk update
+RUN apk add curl python3 git bash dos2unix openssh build-base python3-dev
+
+RUN ln -sf $(ls /usr/bin/easy_install*) /usr/bin/easy_install
+RUN easy_install pip
+RUN pip install docker-compose
+RUN pip install poetry
+
+RUN mkdir /updater
+WORKDIR /updater
+COPY . /updater/
+RUN dos2unix cmd.bash
+
+ENV COMMIT_NAME 'Updater Bot'
+ENV COMMIT_EMAIL ''
+ENV COMMIT_MESSAGE '[Updater Bot] Regenerate data'
+ENV BRANCH_NAME 'updater-bot'
+ENV REPO_POKEAPI  'git@github.com:PokeAPI/pokeapi'
+ENV REPO_DITTO  'git@github.com:PokeAPI/ditto'
+
+CMD bash cmd.bash

updater/README.md

@@ -0,0 +1,29 @@
+# Updater Bot
+
+## Usage
+
+First, make sure you can read/write the target repository over SSH.
+Launch the bot with a volume containing the SSH keys to `/root/.ssh` and an environment variable for email address.
+Since this container runs Docker within itself, it needs to run in privileged mode.
+
+```
+docker run --privileged -v ~/.ssh:/root/.ssh -e COMMIT_EMAIL=example@example.com sargunv/pokeapi-ditto:updater
+```
+
+**Note:** Due to lack of support for file permissions, this does not work on Docker for Windows.
+
+## Environment Variables
+
+### Required
+
+ - `COMMIT_EMAIL`
+
+### Optional
+
+See [the Dockerfile](updater/Dockerfile) for the defaults.
+
+ - `COMMIT_NAME`
+ - `COMMIT_MESSAGE`
+ - `BRANCH_NAME`
+ - `REPO_POKEAPI`
+ - `REPO_DITTO`

updater/cmd.bash

@@ -0,0 +1,48 @@
+#!/usr/bin/env bash
+
+[ -z "${COMMIT_NAME}" ] && { echo "Need to set COMMIT_NAME"; exit 1; }
+[ -z "${COMMIT_EMAIL}" ] && { echo "Need to set COMMIT_EMAIL"; exit 1; }
+[ -z "${COMMIT_MESSAGE}" ] && { echo "Need to set COMMIT_MESSAGE"; exit 1; }
+[ -z "${REPO_POKEAPI}" ] && { echo "Need to set REPO_POKEAPI"; exit 1; }
+[ -z "${REPO_DITTO}" ] && { echo "Need to set REPO_DITTO"; exit 1; }
+[ -z "${BRANCH_NAME}" ] && { echo "Need to set BRANCH_NAME"; exit 1; }
+
+set -e
+set -o pipefail
+set -x
+
+export COMPOSE_INTERACTIVE_NO_CLI=1
+
+dockerd --host=unix:///var/run/docker.sock --host=tcp://0.0.0.0:2375 &> /dev/null &
+
+git clone --depth=1 "$REPO_POKEAPI" pokeapi
+git clone --depth=1 "$REPO_DITTO" ditto
+
+# set up the pokeapi side
+cd pokeapi
+git apply ../disable-rate-limit.patch
+
+docker volume create --name=redis_data
+docker volume create --name=pg_data
+docker-compose up -d
+
+docker-compose exec -T app python manage.py migrate
+docker-compose exec -T app sh -c 'echo "from data.v2.build import build_all; build_all()" | python manage.py shell'
+
+# set up the ditto side
+cd ../ditto
+git branch -D "$BRANCH_NAME" || true
+git branch "$BRANCH_NAME"
+git checkout "$BRANCH_NAME"
+
+poetry install
+rm -r ./data
+poetry run ditto clone --src-url http://localhost/ --dest-dir ./data
+poetry run ditto analyze --api-dir ./data/api --schema-dir ./data/schema
+
+# commit and push
+git add data
+git config user.name "$COMMIT_NAME"
+git config user.email "$COMMIT_EMAIL"
+git commit -m "$COMMIT_MESSAGE"
+git push -fu origin "$BRANCH_NAME"