documentation updates

Geonovum · Jun 22, 2021 · bff9c86 · bff9c86
1 parent aede6bc
commit bff9c86
Show file tree

Hide file tree

Showing 8 changed files with 73 additions and 134 deletions.
diff --git a/README.md b/README.md
@@ -9,12 +9,17 @@
 [![Gitter](https://img.shields.io/gitter/room/Geonovum/ogc-api-testbed.svg?style=flat-square)](https://gitter.im/Geonovum/ogc-api-testbed)
 
 # OGC API Testbed
-Bootstrap and continuous deployment/integration (CI) for OGC API web-service components.
-See the [website](https://apitestdocs.geonovum.nl) for docmentation and details.
+Bootstrap and continuous integration/deployment (CI/CD) for OGC API web-service components.
+See the [website apitestdocs.geonovum.nl](https://apitestdocs.geonovum.nl) for docmentation and details.
+
+Want to access the (OGC) web-services? Go to:
+
+* Stable (production) server at [apitestbed.geonovum.nl](https://apitestbed.geonovum.nl/)
+* Sandbox (experimental) server at [apisandbox.geonovum.nl](https://apisandbox.geonovum.nl/) (TODO)
 
 ## Summary
 
-This repo contains all that is needed to bootstrap, configure and maintain (CI) a remote
+This repo contains all that is needed to bootstrap, configure and maintain (CI/CD) a remote
 deployment of an OGC API web-service stack using modern "DevOps" tooling.
 
 The main design principles are:
@@ -25,98 +30,40 @@ The main design principles are:
 * all credentials (passwords, SSH-keys, etc) are secured 
 * operational stack instances for "production" (stable) and "sandbox" (playground)
 
-The (DevOps-) components for this setup are:
+The (DevOps-) components used in this setup are:
 
 * [Docker](https://www.docker.com/) *"...OS-level virtualization to deliver software in packages called containers..."* ([Wikipedia](https://en.wikipedia.org/wiki/Docker_(software)))
 * [Docker Compose](https://docs.docker.com/compose) *"...a tool for defining and running multi-container Docker applications..."*
 * [Ansible](https://www.ansible.com/) *"...an open-source software provisioning tool"* ([Wikipedia](https://en.wikipedia.org/wiki/Ansible_(software)))
 * [GitHub Actions/Workflows](https://docs.github.com/en/actions) *"...Automate, customize, and execute software development workflows in a GitHub repository..."*
 
-The Docker-components are used to run the operational stack, i.e. the OGC API web-services. Ansible is used to provision both the server OS-software
-and the operational stack. Ansible is executed on a local client/desktop system to invoke operations on a remote server/VM.
+The Docker-components are used to run the operational stack, i.e. the OGC API web-services. 
+Ansible is used to provision both the server OS-software
+and the operational stack. 
+Ansible is executed on a local client/desktop system to invoke operations on a remote server/VM.
 These operations are bundled in so called Ansible Playbooks, YAML files that describe a desired server state.
-GitHub Actions are used to construct Workflows. These Actions will invoke these Ansible Playbooks, effectively configuring
+GitHub Actions are used to construct Workflows. 
+These Actions invoke these Ansible Playbooks, effectively configuring
 and provisioning the operational stack on a remote server/VM. 
+GitHub Actions are triggered (selectively) on commit/push to this repo.
 
-Security is guaranteed by the use of [Ansible-Vault](https://docs.ansible.com/ansible/latest/user_guide/vault.html) 
+Security is enforced by the use of [Ansible-Vault](https://docs.ansible.com/ansible/latest/user_guide/vault.html) 
 and [GitHub Encrypted Secrets](https://docs.github.com/en/actions/reference/encrypted-secrets).
 
-The operational stack is composed with the following components:
+The operational stack has the following components:
 
 * [Traefik](https://traefik.io/) a frontend proxy/load-balancer and SSL (HTTPS) endpoint.
-* [mkdocs](https://www.mkdocs.org/) for live documentation
-
-* both an experimental and stable stack instance is available
-* [GeoHealthCheck]() is a component to monitor the availability and complience of the implementations 
-
-# Services
-
 * [pygeoapi](https://pygeoapi.io/) a Python server implementation of the OGC API suite of standards.
 * [GeoServer](http://geoserver.org/) a Java server implementation of the OGC API suite of standards.
 * [ldproxy](https://interactive-instruments.github.io/ldproxy/) a Java server implementation of the OGC API suite of standards.
+* [QGIS Server](https://www.qgis.org/) - server component of QGIS with OGC OAFeat support.
+* [PostgreSQL/PostGIS](https://postgis.net) - geospatial database
 
-## Production and Sandbox Instances
-
-Two separate server/CM-instances are managed to provide stable/production and 
-sandbox/playground environments. As to control changes these instances are mapped to two GitHub branches:
-
-* `main` for the stable/production instance
-* `sandbox` for the playground
-
-[GitHub Protected Branches](https://docs.github.com/en/github/administering-a-repository/defining-the-mergeability-of-pull-requests/about-protected-branches) are
-used to provide for selective access and deployment.
-
-
-## Selective Redeploy
-When changes are pushed to this repo only the affected services are redeployed.
-This is effected by a combination of GitHub Actions and Ansible Playbooks as follows:
-
-* each Service has a dedicated GitHub Action "deploy" file, e.g. [deploy.pygeoapi.yml](.github/workflows/deploy.pygeoapi.yml)
-* the GitHub Action "deploy" file contains a trigger for a `push` with a `paths` constraint, in this example:
-```  
-    on:
-      push:
-        paths:
-          - 'services/pygeoapi/**'
-```   
-* the GH Action then calls the Ansible Playbook [deploy.yml](ansible/deploy.yml) with a `--tags` option related to the Service, e.g. `--tags pygeoapi`
-* the [deploy.yml](ansible/deploy.yml) will always update the GH repo on the server VM via the `pre_tasks`
-* the Ansible task indicated by the `tags` is then executed
-
-TODO: this will be extended with 
-[GitHub Protected Branches](https://docs.github.com/en/github/administering-a-repository/defining-the-mergeability-of-pull-requests/about-protected-branches)
-to map the branch pushed to a server (host) instance via Ansible Inventory settings.
-
-## Steps and Workflows
-
-These can be used to setup a running server from zero.
-
-### Step 0 - Obtain access to server/VM
-This implies acquiring a server/VM instance from a hosting provider.
-Main requirements are that server/VM runs an LTS Ubuntu (20.4 or better) and that SSL-keys are available for root access 
-(or an admin user account with sudo-rights).
-
-### Step 1 - Clone this repo
-
-`git clone https://github.com/Geonovum/ogc-api-testbed.git`.
-
-### Step 2 - Bootstrap the server/VM
-"Bootstrap" here implies the complete provisioning of a remote server/VM that runs the operational service stack.
-This is a one-time manual action, but can be executed at any time as Ansible actions are idempotent.
-By its nature, Ansible tasks will only change the system if there is something to do.
-
-Startpoint is a fresh Ubuntu-server or VM with root access via SSH-keys (no passwords).
-The Ansible playbook [bootstrap.yml](ansible/bootstrap.yml) installs the neccessary software, and hardens
-the server security, e.g. using [fail2ban](https://www.fail2ban.org/).
-In this step Docker and Docker Compose are installed and a Linux [systemd](https://en.wikipedia.org/wiki/Systemd) service is run
-that automatically starts/stops the operational stack, also on reboots.
-The software for the operational stack, i.e. from this repo, is cloned on the server as well.
+For administration, documentation and monitoring the following components are used:
 
-### Step 3 - Maintain the server/VM
-This step is the daily operational maintenance. 
-The basic substeps are:
+* [mkdocs](https://www.mkdocs.org/) for live documentation and landing pages
+* [PGAdmin](https://www.pgadmin.org/) - visual PostgreSQL manager  
+* [GeoHealthCheck](https://geohealthcheck.org) to monitor the availability, compliance and QoS of OGC web services
+* [Portainer](https://www.portainer.io/) visual Docker monitor and manager
 
-* make a change, e.g. add a data Collection to an OGC API OAFeat service
-* commit/push the change to GitHub
-* watch the triggered GitHub Actions, check for any errors
-* observe changes via website
+Read more on the setup in the [documentation/website of this project](https://apitestdocs.geonovum.nl/setup).
diff --git a/docs/docs/howto/howto_deploy.md b/docs/docs/howto/howto_deploy.md
@@ -4,7 +4,10 @@ title: Service deployment
 
 # Service deployment
 
-The api testbed environment uses a configuration mechanism stored in GitHub. Whenever a commit is detected on the configuration repository, a deployment of the changed service is triggered automatically. Such an approach is known as [Continuos Deployment](https://en.wikipedia.org/wiki/Continuous_deployment). While a deployment task is running, you can follow it on [github](https://github.com/Geonovum/ogc-api-testbed/actions).
+The api testbed environment uses a configuration mechanism stored in GitHub. 
+Whenever a commit is detected on the configuration repository, 
+a deployment of the changed service is triggered automatically. 
+Such an approach is known as [Continuous Deployment](https://en.wikipedia.org/wiki/Continuous_deployment). While a deployment task is running, you can follow it on [github](https://github.com/Geonovum/ogc-api-testbed/actions).
 
 It is possible to directly commit your changes to GitHub, but a better practice is to work from [Pull Requests](https://en.wikipedia.org/wiki/Distributed_version_control#Pull_requests). Some discussion and an approval process can happen around a pull request, before it is merged and deployed.
 

diff --git a/docs/docs/index.md b/docs/docs/index.md
@@ -7,6 +7,7 @@ title: OGC-API-Testbed Documentation
 This documentation describes a platform to facilitate a Bootstrap and 
 continuous deployment/integration (CI) platform to support a 
 testbed with OGC API web-services to be run by Geonovum. 
+Find a quick intro in the project [README](https://github.com/Geonovum/ogc-api-testbed#readme).
 
 The documentation is split up in three parts:
 
@@ -20,41 +21,12 @@ The documentation is split up in three parts:
 [![Gitter](https://img.shields.io/gitter/room/Geonovum/ogc-api-testbed.svg?style=flat-square)](https://gitter.im/Geonovum/ogc-api-testbed)
 
 ## Services
-
-Below the (web-) services running on this domain. Click links to view and interact.
 
-### OGC API Features
+To access and interact with the (OGC web-) services, go to 
+one of the two server instances:
 
-Access the interactive endpoints of [OGC API Features](https://ogcapi.ogc.org/features/) 
-Open Source products running in this instance. Links to documentation and HOWTOs included:
-
-| Endpoint | Author | Docs | HOWTO
-| --- | --- | --- | --- 
-| [/pygeoapi](/pygeoapi) | [GeoPython Community](https://geopython.github.io/) | [link](https://docs.pygeoapi.io/en/latest/) | [link](howto/howto_pygeoapi.md)  
-| [/ldproxy](/ldproxy) | Interactive Instruments | [link](https://interactive-instruments.github.io/ldproxy/) | [link](howto/howto_ldproxy.md)  
-| [/geoserver](/geoserver/ogc/features) | GeoServer Community | [link](https://docs.geoserver.org/latest/en/user/community/ogc-api/index.html) | [link](howto/howto_geoserver.md)  
-| [/qgis*](/qgis/wfs3) | QGIS Community | [link](https://www.qgis.org/) | [link](howto/howto_qgis.md)  
-| [/pycsw/csw.py](/pycsw/csw.py/collections/metadata:main) | [GeoPython Community](https://geopython.github.io/) | [docs](https://docs.pycsw.org/en/latest/index.html) | [HOWTO](howto/howto_pycsw.md) 
-
-
-* *"QGIS" is the QGIS Server Application.*
-
-### Storage Services
-
-The above services serve data from local files (e.g. GeoPackage) and these storage services:
-
-* "PostGIS" - access via <server-domain-name> port 5432 - spatial database based on PostgreSQL. 
-*  Manage data with the [PGAdmin web application](/pgadmin)
-
-### Supporting Services
-
-These are services for administration and maintenance.
-
-| Endpoint | What / Author | Docs | HOWTO
-| --- | --- | --- | --- 
-| [PGAdmin](/pgadmin) | Manage PostgreSQL Data | PGAdmin Community | [link](https://www.pgadmin.org/) | [link](howto/howto_pgadmin.md)  
-| [Portainer](/portainer/) | Visual Docker Manager | Docker | [link](https://www.portainer.io/) | [link](howto/howto_portainer.md)  
-| [GeoHealthCheck](/ghc) | OGC Services Monitor | [GeoPython Community](https://geopython.github.io)  | [link](https://geohealthcheck.org) | [link](howto/howto_ghc.md)  
+* Stable (production) server at [apitestbed.geonovum.nl](https://apitestbed.geonovum.nl/)
+* Sandbox (experimental) server at [apisandbox.geonovum.nl](https://apisandbox.geonovum.nl/) (TODO)
 
 ## Links
 

diff --git a/docs/docs/setup/ghc.md b/docs/docs/setup/ghc.md
@@ -17,6 +17,7 @@ OGC OAFeat standard with two basic checks (called "Probes"):
 - full OAS schema validation
 
 ## Deployment
+
 GHC is part of the
 [Admin Stack](https://github.com/Geonovum/ogc-api-testbed/tree/main/services/admin) in the testbed.
 
@@ -36,18 +37,12 @@ GHC needs quite some variables (around 31, though many defaults apply).
 These are all configured once in
 [ghc.env](https://github.com/Geonovum/ogc-api-testbed/blob/main/services/admin/ghc.env). 
 Many variables represent credentials like email and 
-database configuration. These are bundled as `geohealthcheck_env` 
+database configuration. These are bundled as `etc_environment` 
 in and forwarded from the encrypted Ansible file `vars.yml`.
 
-The Ansible [deploy.yml](https://github.com/Geonovum/ogc-api-testbed/blob/main/ansible/deploy.yml#L52) 
-line applies `geohealthcheck_env` such that these are available
-to the local environment and referenced in 
-`ghc.env` (and thus to the Docker Containers).
-
 ```
     - name: "admin"
       shell: "cd {{ services_home }}/admin && ./deploy.sh && docker ps"
-      environment: "{{ geohealthcheck_env }}"
       tags: admin
 
 ```

diff --git a/docs/docs/setup/index.md b/docs/docs/setup/index.md
@@ -4,11 +4,12 @@ title: Setup
 
 # Setup
 
-This section describes the setup details of the platform.
+This section describes the setup/installation details of the platform.
 
 ##  Platform setup
 
-This section introduces the setup of the platform and components used in the platform infrastructure.
+This section introduces the setup of the platform and 
+components used in the platform infrastructure.
 
 * [Platform setup](platform.md) 
 

diff --git a/docs/docs/setup/platform.md b/docs/docs/setup/platform.md
@@ -4,7 +4,7 @@ title: Platform setup
 
 # Platform setup
 
-The project contains contains components to bootstrap, configure and maintain a remote
+The project repository contains contains components to bootstrap, configure and maintain a remote
 deployment of an OGC API web-service stack using modern "DevOps" tooling.
 
 ## Design Principles
@@ -37,14 +37,21 @@ and provisioning the operational stack on a remote server/VM.
 Security is guaranteed by the use of [Ansible-Vault](https://docs.ansible.com/ansible/latest/user_guide/vault.html) 
 and [GitHub Encrypted Secrets](https://docs.github.com/en/actions/reference/encrypted-secrets).
 
-The operational stack is composed with the following components:
+The operational stack has the following components:
 
 * [Traefik](https://traefik.io/) a frontend proxy/load-balancer and SSL (HTTPS) endpoint.
-* [mkdocs](https://www.mkdocs.org/) for live documentation and landing pages ("Home")
-* [GeoHealthCheck]() monitors the availability and QoS of OGC webservices 
-* [portainer]() provides monitoring of and access to running Docker containers
-* [PostgreSQL with PostGIS]() - geospatial RDBMS, nicknamed "PostGIS"
-* PGAdmin - administration component for PostgreSQL DBs
+* [pygeoapi](https://pygeoapi.io/) a Python server implementation of the OGC API suite of standards.
+* [GeoServer](http://geoserver.org/) a Java server implementation of the OGC API suite of standards.
+* [ldproxy](https://interactive-instruments.github.io/ldproxy/) a Java server implementation of the OGC API suite of standards.
+* [QGIS Server](https://www.qgis.org/) - server component of QGIS with OGC OAFeat support.
+* [PostgreSQL/PostGIS](https://postgis.net) - geospatial database
+
+For administration, documentation and monitoring the following components are used:
+
+* [mkdocs](https://www.mkdocs.org/) for live documentation and landing pages
+* [PGAdmin](https://www.pgadmin.org/) - visual PostgreSQL manager  
+* [GeoHealthCheck](https://geohealthcheck.org) to monitor the availability, compliance and QoS of OGC web services
+* [Portainer](https://www.portainer.io/) visual Docker monitor and manager
 
 ## Production and Sandbox Instance
 
@@ -61,17 +68,20 @@ from which the **Sandbox** is cloned.
 *it felt that those would be less transparent and even confusing for selective access and chances of mistakes.*
 
 ## Selective Redeploy
-When changes are pushed to this repo only the affected services are redeployed.
+
+When changes are pushed to the repo, only the affected services are redeployed.
 This is effected by a combination of GitHub Actions and Ansible Playbooks as follows:
 
 * each Service has a dedicated GitHub Action "deploy" file, e.g. [deploy.pygeoapi.yml](.github/workflows/deploy.pygeoapi.yml)
 * the GitHub Action "deploy" file contains a trigger for a `push` with a `paths` constraint, in this example:
+
 ```  
     on:
       push:
         paths:
           - 'services/pygeoapi/**'
-```   
+``` 
+
 * the GH Action then calls the Ansible Playbook [deploy.yml](ansible/deploy.yml) with a `--tags` option related to the Service, e.g. `--tags pygeoapi`
 * the [deploy.yml](ansible/deploy.yml) will always update the GH repo on the server VM via the `pre_tasks`
 * the Ansible task indicated by the `tags` is then executed

diff --git a/docs/docs/setup/pygeoapi.md b/docs/docs/setup/pygeoapi.md
@@ -8,10 +8,12 @@ pygeoapi is a python implementation of the OGC API Suite of standards.
 
 ## Installation
 
-A docker hub image is provided by [geopython](https://hub.docker.com/r/geopython/pygeoapi). The data folder and the main config file are mounted into the container.
+A docker hub image is provided by [geopython](https://hub.docker.com/r/geopython/pygeoapi). 
+The data folder and the main config file are mounted into the container.
 
 The data folder contains the datafiles, alternatively you can upload data to the PostGreSQL database and configure a layer on data from the database.
 
-The configuration file contains references to the collections which are exposed via the OGC API's. Check [the documentation](https://docs.pygeoapi.io/en/latest/data-publishing/index.html#providers-overview) to know which backends are supported.
+The configuration file contains references to the collections which are exposed via the OGC API's. 
+Check [the documentation](https://docs.pygeoapi.io/en/latest/data-publishing/index.html#providers-overview) to know which backends are supported.
 
-You need to set up an instance of pygeoapi for each series of collections you want to serve on an endpoint.
+You need to set up an instance of pygeoapi for each series of collections you want to serve on an endpoint.
diff --git a/services/README.md b/services/README.md
@@ -0,0 +1,9 @@
+# Services
+
+This is the home of the various service components.
+Each directory below contains the resources for a single component.
+Except the [/admin](/admin) dir, which contains most of the admin stack.
+
+Each service component is basically a `docker-compose.yml` file with supporting resources/files.
+
+Each service can be started individually with `start.sh` and `stop.sh` Bash scripts.