From bff9c86b2a7caf7c40839a4fd29d0e0806e19ed8 Mon Sep 17 00:00:00 2001 From: Just van den Broecke Date: Tue, 22 Jun 2021 16:48:02 +0200 Subject: [PATCH] documentation updates --- README.md | 105 ++++++++------------------------ docs/docs/howto/howto_deploy.md | 5 +- docs/docs/index.md | 38 ++---------- docs/docs/setup/ghc.md | 9 +-- docs/docs/setup/index.md | 5 +- docs/docs/setup/platform.md | 28 ++++++--- docs/docs/setup/pygeoapi.md | 8 ++- services/README.md | 9 +++ 8 files changed, 73 insertions(+), 134 deletions(-) create mode 100644 services/README.md diff --git a/README.md b/README.md index ab9ef6c..f3d94a6 100644 --- a/README.md +++ b/README.md @@ -9,12 +9,17 @@ [![Gitter](https://img.shields.io/gitter/room/Geonovum/ogc-api-testbed.svg?style=flat-square)](https://gitter.im/Geonovum/ogc-api-testbed) # OGC API Testbed -Bootstrap and continuous deployment/integration (CI) for OGC API web-service components. -See the [website](https://apitestdocs.geonovum.nl) for docmentation and details. +Bootstrap and continuous integration/deployment (CI/CD) for OGC API web-service components. +See the [website apitestdocs.geonovum.nl](https://apitestdocs.geonovum.nl) for docmentation and details. + +Want to access the (OGC) web-services? Go to: + +* Stable (production) server at [apitestbed.geonovum.nl](https://apitestbed.geonovum.nl/) +* Sandbox (experimental) server at [apisandbox.geonovum.nl](https://apisandbox.geonovum.nl/) (TODO) ## Summary -This repo contains all that is needed to bootstrap, configure and maintain (CI) a remote +This repo contains all that is needed to bootstrap, configure and maintain (CI/CD) a remote deployment of an OGC API web-service stack using modern "DevOps" tooling. The main design principles are: @@ -25,98 +30,40 @@ The main design principles are: * all credentials (passwords, SSH-keys, etc) are secured * operational stack instances for "production" (stable) and "sandbox" (playground) -The (DevOps-) components for this setup are: +The (DevOps-) components used in this setup are: * [Docker](https://www.docker.com/) *"...OS-level virtualization to deliver software in packages called containers..."* ([Wikipedia](https://en.wikipedia.org/wiki/Docker_(software))) * [Docker Compose](https://docs.docker.com/compose) *"...a tool for defining and running multi-container Docker applications..."* * [Ansible](https://www.ansible.com/) *"...an open-source software provisioning tool"* ([Wikipedia](https://en.wikipedia.org/wiki/Ansible_(software))) * [GitHub Actions/Workflows](https://docs.github.com/en/actions) *"...Automate, customize, and execute software development workflows in a GitHub repository..."* -The Docker-components are used to run the operational stack, i.e. the OGC API web-services. Ansible is used to provision both the server OS-software -and the operational stack. Ansible is executed on a local client/desktop system to invoke operations on a remote server/VM. +The Docker-components are used to run the operational stack, i.e. the OGC API web-services. +Ansible is used to provision both the server OS-software +and the operational stack. +Ansible is executed on a local client/desktop system to invoke operations on a remote server/VM. These operations are bundled in so called Ansible Playbooks, YAML files that describe a desired server state. -GitHub Actions are used to construct Workflows. These Actions will invoke these Ansible Playbooks, effectively configuring +GitHub Actions are used to construct Workflows. +These Actions invoke these Ansible Playbooks, effectively configuring and provisioning the operational stack on a remote server/VM. +GitHub Actions are triggered (selectively) on commit/push to this repo. -Security is guaranteed by the use of [Ansible-Vault](https://docs.ansible.com/ansible/latest/user_guide/vault.html) +Security is enforced by the use of [Ansible-Vault](https://docs.ansible.com/ansible/latest/user_guide/vault.html) and [GitHub Encrypted Secrets](https://docs.github.com/en/actions/reference/encrypted-secrets). -The operational stack is composed with the following components: +The operational stack has the following components: * [Traefik](https://traefik.io/) a frontend proxy/load-balancer and SSL (HTTPS) endpoint. -* [mkdocs](https://www.mkdocs.org/) for live documentation - -* both an experimental and stable stack instance is available -* [GeoHealthCheck]() is a component to monitor the availability and complience of the implementations - -# Services - * [pygeoapi](https://pygeoapi.io/) a Python server implementation of the OGC API suite of standards. * [GeoServer](http://geoserver.org/) a Java server implementation of the OGC API suite of standards. * [ldproxy](https://interactive-instruments.github.io/ldproxy/) a Java server implementation of the OGC API suite of standards. +* [QGIS Server](https://www.qgis.org/) - server component of QGIS with OGC OAFeat support. +* [PostgreSQL/PostGIS](https://postgis.net) - geospatial database -## Production and Sandbox Instances - -Two separate server/CM-instances are managed to provide stable/production and -sandbox/playground environments. As to control changes these instances are mapped to two GitHub branches: - -* `main` for the stable/production instance -* `sandbox` for the playground - -[GitHub Protected Branches](https://docs.github.com/en/github/administering-a-repository/defining-the-mergeability-of-pull-requests/about-protected-branches) are -used to provide for selective access and deployment. - - -## Selective Redeploy -When changes are pushed to this repo only the affected services are redeployed. -This is effected by a combination of GitHub Actions and Ansible Playbooks as follows: - -* each Service has a dedicated GitHub Action "deploy" file, e.g. [deploy.pygeoapi.yml](.github/workflows/deploy.pygeoapi.yml) -* the GitHub Action "deploy" file contains a trigger for a `push` with a `paths` constraint, in this example: -``` - on: - push: - paths: - - 'services/pygeoapi/**' -``` -* the GH Action then calls the Ansible Playbook [deploy.yml](ansible/deploy.yml) with a `--tags` option related to the Service, e.g. `--tags pygeoapi` -* the [deploy.yml](ansible/deploy.yml) will always update the GH repo on the server VM via the `pre_tasks` -* the Ansible task indicated by the `tags` is then executed - -TODO: this will be extended with -[GitHub Protected Branches](https://docs.github.com/en/github/administering-a-repository/defining-the-mergeability-of-pull-requests/about-protected-branches) -to map the branch pushed to a server (host) instance via Ansible Inventory settings. - -## Steps and Workflows - -These can be used to setup a running server from zero. - -### Step 0 - Obtain access to server/VM -This implies acquiring a server/VM instance from a hosting provider. -Main requirements are that server/VM runs an LTS Ubuntu (20.4 or better) and that SSL-keys are available for root access -(or an admin user account with sudo-rights). - -### Step 1 - Clone this repo - -`git clone https://github.com/Geonovum/ogc-api-testbed.git`. - -### Step 2 - Bootstrap the server/VM -"Bootstrap" here implies the complete provisioning of a remote server/VM that runs the operational service stack. -This is a one-time manual action, but can be executed at any time as Ansible actions are idempotent. -By its nature, Ansible tasks will only change the system if there is something to do. - -Startpoint is a fresh Ubuntu-server or VM with root access via SSH-keys (no passwords). -The Ansible playbook [bootstrap.yml](ansible/bootstrap.yml) installs the neccessary software, and hardens -the server security, e.g. using [fail2ban](https://www.fail2ban.org/). -In this step Docker and Docker Compose are installed and a Linux [systemd](https://en.wikipedia.org/wiki/Systemd) service is run -that automatically starts/stops the operational stack, also on reboots. -The software for the operational stack, i.e. from this repo, is cloned on the server as well. +For administration, documentation and monitoring the following components are used: -### Step 3 - Maintain the server/VM -This step is the daily operational maintenance. -The basic substeps are: +* [mkdocs](https://www.mkdocs.org/) for live documentation and landing pages +* [PGAdmin](https://www.pgadmin.org/) - visual PostgreSQL manager +* [GeoHealthCheck](https://geohealthcheck.org) to monitor the availability, compliance and QoS of OGC web services +* [Portainer](https://www.portainer.io/) visual Docker monitor and manager -* make a change, e.g. add a data Collection to an OGC API OAFeat service -* commit/push the change to GitHub -* watch the triggered GitHub Actions, check for any errors -* observe changes via website +Read more on the setup in the [documentation/website of this project](https://apitestdocs.geonovum.nl/setup). diff --git a/docs/docs/howto/howto_deploy.md b/docs/docs/howto/howto_deploy.md index cde599e..475fb07 100644 --- a/docs/docs/howto/howto_deploy.md +++ b/docs/docs/howto/howto_deploy.md @@ -4,7 +4,10 @@ title: Service deployment # Service deployment -The api testbed environment uses a configuration mechanism stored in GitHub. Whenever a commit is detected on the configuration repository, a deployment of the changed service is triggered automatically. Such an approach is known as [Continuos Deployment](https://en.wikipedia.org/wiki/Continuous_deployment). While a deployment task is running, you can follow it on [github](https://github.com/Geonovum/ogc-api-testbed/actions). +The api testbed environment uses a configuration mechanism stored in GitHub. +Whenever a commit is detected on the configuration repository, +a deployment of the changed service is triggered automatically. +Such an approach is known as [Continuous Deployment](https://en.wikipedia.org/wiki/Continuous_deployment). While a deployment task is running, you can follow it on [github](https://github.com/Geonovum/ogc-api-testbed/actions). It is possible to directly commit your changes to GitHub, but a better practice is to work from [Pull Requests](https://en.wikipedia.org/wiki/Distributed_version_control#Pull_requests). Some discussion and an approval process can happen around a pull request, before it is merged and deployed. diff --git a/docs/docs/index.md b/docs/docs/index.md index eb932e6..e8f4a35 100644 --- a/docs/docs/index.md +++ b/docs/docs/index.md @@ -7,6 +7,7 @@ title: OGC-API-Testbed Documentation This documentation describes a platform to facilitate a Bootstrap and continuous deployment/integration (CI) platform to support a testbed with OGC API web-services to be run by Geonovum. +Find a quick intro in the project [README](https://github.com/Geonovum/ogc-api-testbed#readme). The documentation is split up in three parts: @@ -20,41 +21,12 @@ The documentation is split up in three parts: [![Gitter](https://img.shields.io/gitter/room/Geonovum/ogc-api-testbed.svg?style=flat-square)](https://gitter.im/Geonovum/ogc-api-testbed) ## Services - -Below the (web-) services running on this domain. Click links to view and interact. -### OGC API Features +To access and interact with the (OGC web-) services, go to +one of the two server instances: -Access the interactive endpoints of [OGC API Features](https://ogcapi.ogc.org/features/) -Open Source products running in this instance. Links to documentation and HOWTOs included: - -| Endpoint | Author | Docs | HOWTO -| --- | --- | --- | --- -| [/pygeoapi](/pygeoapi) | [GeoPython Community](https://geopython.github.io/) | [link](https://docs.pygeoapi.io/en/latest/) | [link](howto/howto_pygeoapi.md) -| [/ldproxy](/ldproxy) | Interactive Instruments | [link](https://interactive-instruments.github.io/ldproxy/) | [link](howto/howto_ldproxy.md) -| [/geoserver](/geoserver/ogc/features) | GeoServer Community | [link](https://docs.geoserver.org/latest/en/user/community/ogc-api/index.html) | [link](howto/howto_geoserver.md) -| [/qgis*](/qgis/wfs3) | QGIS Community | [link](https://www.qgis.org/) | [link](howto/howto_qgis.md) -| [/pycsw/csw.py](/pycsw/csw.py/collections/metadata:main) | [GeoPython Community](https://geopython.github.io/) | [docs](https://docs.pycsw.org/en/latest/index.html) | [HOWTO](howto/howto_pycsw.md) - - -* *"QGIS" is the QGIS Server Application.* - -### Storage Services - -The above services serve data from local files (e.g. GeoPackage) and these storage services: - -* "PostGIS" - access via port 5432 - spatial database based on PostgreSQL. -* Manage data with the [PGAdmin web application](/pgadmin) - -### Supporting Services - -These are services for administration and maintenance. - -| Endpoint | What / Author | Docs | HOWTO -| --- | --- | --- | --- -| [PGAdmin](/pgadmin) | Manage PostgreSQL Data | PGAdmin Community | [link](https://www.pgadmin.org/) | [link](howto/howto_pgadmin.md) -| [Portainer](/portainer/) | Visual Docker Manager | Docker | [link](https://www.portainer.io/) | [link](howto/howto_portainer.md) -| [GeoHealthCheck](/ghc) | OGC Services Monitor | [GeoPython Community](https://geopython.github.io) | [link](https://geohealthcheck.org) | [link](howto/howto_ghc.md) +* Stable (production) server at [apitestbed.geonovum.nl](https://apitestbed.geonovum.nl/) +* Sandbox (experimental) server at [apisandbox.geonovum.nl](https://apisandbox.geonovum.nl/) (TODO) ## Links diff --git a/docs/docs/setup/ghc.md b/docs/docs/setup/ghc.md index b6fa9af..5958d05 100644 --- a/docs/docs/setup/ghc.md +++ b/docs/docs/setup/ghc.md @@ -17,6 +17,7 @@ OGC OAFeat standard with two basic checks (called "Probes"): - full OAS schema validation ## Deployment + GHC is part of the [Admin Stack](https://github.com/Geonovum/ogc-api-testbed/tree/main/services/admin) in the testbed. @@ -36,18 +37,12 @@ GHC needs quite some variables (around 31, though many defaults apply). These are all configured once in [ghc.env](https://github.com/Geonovum/ogc-api-testbed/blob/main/services/admin/ghc.env). Many variables represent credentials like email and -database configuration. These are bundled as `geohealthcheck_env` +database configuration. These are bundled as `etc_environment` in and forwarded from the encrypted Ansible file `vars.yml`. -The Ansible [deploy.yml](https://github.com/Geonovum/ogc-api-testbed/blob/main/ansible/deploy.yml#L52) -line applies `geohealthcheck_env` such that these are available -to the local environment and referenced in -`ghc.env` (and thus to the Docker Containers). - ``` - name: "admin" shell: "cd {{ services_home }}/admin && ./deploy.sh && docker ps" - environment: "{{ geohealthcheck_env }}" tags: admin ``` diff --git a/docs/docs/setup/index.md b/docs/docs/setup/index.md index 4c006e7..db22d98 100644 --- a/docs/docs/setup/index.md +++ b/docs/docs/setup/index.md @@ -4,11 +4,12 @@ title: Setup # Setup -This section describes the setup details of the platform. +This section describes the setup/installation details of the platform. ## Platform setup -This section introduces the setup of the platform and components used in the platform infrastructure. +This section introduces the setup of the platform and +components used in the platform infrastructure. * [Platform setup](platform.md) diff --git a/docs/docs/setup/platform.md b/docs/docs/setup/platform.md index afd3652..e1aad29 100644 --- a/docs/docs/setup/platform.md +++ b/docs/docs/setup/platform.md @@ -4,7 +4,7 @@ title: Platform setup # Platform setup -The project contains contains components to bootstrap, configure and maintain a remote +The project repository contains contains components to bootstrap, configure and maintain a remote deployment of an OGC API web-service stack using modern "DevOps" tooling. ## Design Principles @@ -37,14 +37,21 @@ and provisioning the operational stack on a remote server/VM. Security is guaranteed by the use of [Ansible-Vault](https://docs.ansible.com/ansible/latest/user_guide/vault.html) and [GitHub Encrypted Secrets](https://docs.github.com/en/actions/reference/encrypted-secrets). -The operational stack is composed with the following components: +The operational stack has the following components: * [Traefik](https://traefik.io/) a frontend proxy/load-balancer and SSL (HTTPS) endpoint. -* [mkdocs](https://www.mkdocs.org/) for live documentation and landing pages ("Home") -* [GeoHealthCheck]() monitors the availability and QoS of OGC webservices -* [portainer]() provides monitoring of and access to running Docker containers -* [PostgreSQL with PostGIS]() - geospatial RDBMS, nicknamed "PostGIS" -* PGAdmin - administration component for PostgreSQL DBs +* [pygeoapi](https://pygeoapi.io/) a Python server implementation of the OGC API suite of standards. +* [GeoServer](http://geoserver.org/) a Java server implementation of the OGC API suite of standards. +* [ldproxy](https://interactive-instruments.github.io/ldproxy/) a Java server implementation of the OGC API suite of standards. +* [QGIS Server](https://www.qgis.org/) - server component of QGIS with OGC OAFeat support. +* [PostgreSQL/PostGIS](https://postgis.net) - geospatial database + +For administration, documentation and monitoring the following components are used: + +* [mkdocs](https://www.mkdocs.org/) for live documentation and landing pages +* [PGAdmin](https://www.pgadmin.org/) - visual PostgreSQL manager +* [GeoHealthCheck](https://geohealthcheck.org) to monitor the availability, compliance and QoS of OGC web services +* [Portainer](https://www.portainer.io/) visual Docker monitor and manager ## Production and Sandbox Instance @@ -61,17 +68,20 @@ from which the **Sandbox** is cloned. *it felt that those would be less transparent and even confusing for selective access and chances of mistakes.* ## Selective Redeploy -When changes are pushed to this repo only the affected services are redeployed. + +When changes are pushed to the repo, only the affected services are redeployed. This is effected by a combination of GitHub Actions and Ansible Playbooks as follows: * each Service has a dedicated GitHub Action "deploy" file, e.g. [deploy.pygeoapi.yml](.github/workflows/deploy.pygeoapi.yml) * the GitHub Action "deploy" file contains a trigger for a `push` with a `paths` constraint, in this example: + ``` on: push: paths: - 'services/pygeoapi/**' -``` +``` + * the GH Action then calls the Ansible Playbook [deploy.yml](ansible/deploy.yml) with a `--tags` option related to the Service, e.g. `--tags pygeoapi` * the [deploy.yml](ansible/deploy.yml) will always update the GH repo on the server VM via the `pre_tasks` * the Ansible task indicated by the `tags` is then executed diff --git a/docs/docs/setup/pygeoapi.md b/docs/docs/setup/pygeoapi.md index 22625c0..8aaeada 100644 --- a/docs/docs/setup/pygeoapi.md +++ b/docs/docs/setup/pygeoapi.md @@ -8,10 +8,12 @@ pygeoapi is a python implementation of the OGC API Suite of standards. ## Installation -A docker hub image is provided by [geopython](https://hub.docker.com/r/geopython/pygeoapi). The data folder and the main config file are mounted into the container. +A docker hub image is provided by [geopython](https://hub.docker.com/r/geopython/pygeoapi). +The data folder and the main config file are mounted into the container. The data folder contains the datafiles, alternatively you can upload data to the PostGreSQL database and configure a layer on data from the database. -The configuration file contains references to the collections which are exposed via the OGC API's. Check [the documentation](https://docs.pygeoapi.io/en/latest/data-publishing/index.html#providers-overview) to know which backends are supported. +The configuration file contains references to the collections which are exposed via the OGC API's. +Check [the documentation](https://docs.pygeoapi.io/en/latest/data-publishing/index.html#providers-overview) to know which backends are supported. -You need to set up an instance of pygeoapi for each series of collections you want to serve on an endpoint. \ No newline at end of file +You need to set up an instance of pygeoapi for each series of collections you want to serve on an endpoint. diff --git a/services/README.md b/services/README.md new file mode 100644 index 0000000..820be1f --- /dev/null +++ b/services/README.md @@ -0,0 +1,9 @@ +# Services + +This is the home of the various service components. +Each directory below contains the resources for a single component. +Except the [/admin](/admin) dir, which contains most of the admin stack. + +Each service component is basically a `docker-compose.yml` file with supporting resources/files. + +Each service can be started individually with `start.sh` and `stop.sh` Bash scripts.