Expose and restructure the documentation

README.rst

@@ -14,7 +14,8 @@ PAVICS
 
 Power Analytics and Visualization for Climate Science - Powered by Birdhouse and other ESGF software
 
-How to deploy the entire PAVICS platform, see the
-`README <birdhouse/README.md>`_ and the various extra components 
-`README for extra core components <birdhouse/components/README.rst>`_\ ,
-`README for optional components <birdhouse/optional-components/README.md>`_.
+For GitHub navigation, see the following README pages:
+
+* `README for general deployment <birdhouse/README.rst>`_
+* `README for extra core components <birdhouse/components/README.rst>`_
+* `README for optional components <birdhouse/optional-components/README.rst>`_

birdhouse/README.rst

@@ -1,7 +1,3 @@
-.. role:: raw-html-m2r(raw)
-   :format: html
-
-
 Docker instructions
 -------------------
 
@@ -23,22 +19,18 @@ Requirements:
   Install latest docker-ce and docker-compose for the chosen distro (not the
   version from the distro).
 
-To run ``docker-compose`` for PAVICS, the `\ ``pavics-compose.sh`` <pavics-compose.sh>`_ wrapper script must be used.
-This script will source the ``env.local`` file, apply the appropriate variable substitutions on all the configuration files ".template", and run ``docker-compose`` with all the command line arguments given to ``pavics-compose.sh``. See `\ ``env.local.example`` <env.local.example>`_ for more details on what can go into the ``env.local`` file.
+To run ``docker-compose`` for PAVICS, the `pavics-compose.sh <pavics-compose.sh>`_ (:download:`download </birdhouse/pavics-compose.sh>`) wrapper script must be used.
+This script will source the ``env.local`` file, apply the appropriate variable substitutions on all the configuration files
+".template", and run ``docker-compose`` with all the command line arguments given to `pavics-compose.sh <pavics-compose.sh>`_ (:download:`download </birdhouse/pavics-compose.sh>`).
+See `env.local.example <env.local.example>`_ (:download:`download </birdhouse/env.local.example>`) for more details on what can go into the ``env.local`` file.
 
-If the file ``env.local`` is somewhere else, symlink it here, next to
-``docker-compose.yml`` because many scripts assume this location.
+If the file `env.local` is somewhere else, symlink it here, next to `docker-compose.yml <docker-compose.yml>`_ (:download:`download </birdhouse/docker-compose.yml>`) because many scripts assume this location.
 
 To follow infrastructure-as-code, it is encouraged to source control the above
-``env.local`` file and any override needed to customized this PAVICS deployment
-for your organization.  For an example of possible override, see how the `emu
-service <optional-components/emu/docker-compose-extra.yml>`_
-(\ `README <optional-components/README.md>`_\ ) can be optionally added to the
-deployment via the `override
-mechanism <https://docs.docker.com/compose/extends/>`_.  Ouranos specific
-override can be found in this
-`birdhouse-deploy-ouranos <https://github.com/bird-house/birdhouse-deploy-ouranos>`_
-repo.
+`env.local` file and any override needed to customized this PAVICS deployment
+for your organization.  For an example of possible override, see how the `emu service <optional-components/emu/docker-compose-extra.yml>`_ (:download:`download </birdhouse/optional-components/emu/docker-compose-extra.yml>`)
+(`README <optional-components/README.rst#emu-wps-service-for-testing>`_) can be optionally added to the deployment via the `override mechanism <https://docs.docker.com/compose/extends/>`_.
+Ouranos specific override can be found in this `birdhouse-deploy-ouranos <https://github.com/bird-house/birdhouse-deploy-ouranos>`_ repo.
 
 Suggested deployment layout:
 
@@ -57,16 +49,14 @@ Suggested deployment layout:
 The automatic deployment is able to handle multiple repos, so will trigger if
 this repo or your private-personalized-config repo changes, giving you
 automated continuous deployment.  See the continuous deployment setup section
-below and the variable ``AUTODEPLOY_EXTRA_REPOS`` in
-`\ ``env.local.example`` <env.local.example>`_.
+below and the variable ``AUTODEPLOY_EXTRA_REPOS`` in `env.local.example <env.local.example>`_ (:download:`download </birdhouse/env.local.example>`).
 
 The automatic deployment of the PAVICS platform, of the Jupyter tutorial
 notebooks and of the automatic deployment mechanism itself can all be
-enabled by following instructions `here <components/README.rst#scheduler>`_.
+enabled by following the `scheduling instructions <components/README.rst#scheduler>`_.
 
 Resource usage monitoring (CPU, memory, ..) and alerting for the host and each
-of the containers can be enabled by following instructions
-`here <components/README.rst#monitoring>`_.
+of the containers can be enabled by following the `monitoring instructions <components/README.rst#monitoring>`_.
 
 To launch all the containers, use the following command:
 
@@ -75,7 +65,7 @@ To launch all the containers, use the following command:
    ./pavics-compose.sh up -d
 
 If you get a ``'No applicable error code, please check error log'`` error from the WPS processes, please make sure that the WPS databases exists in the
-postgres instance. See `\ ``scripts/create-wps-pgsql-databases.sh`` <scripts/create-wps-pgsql-databases.sh>`_.
+postgres instance. See `create-wps-pgsql-databases.sh <scripts/create-wps-pgsql-databases.sh>`_ (:download:`download </birdhouse/scripts/create-wps-pgsql-databases.sh>`).
 
 Manual steps post deployment
 ----------------------------
@@ -86,7 +76,7 @@ Change geoserver default admin password
 
 * 
   Go to
-  https://\ :raw-html-m2r:`<PAVICS_FQDN>`\ /geoserver/web/wicket/bookmarkable/org.geoserver.security.web.UserGroupRoleServicesPage (Security -> Users, Groups, and Roles)
+  ``https://<PAVICS_FQDN>/geoserver/web/wicket/bookmarkable/org.geoserver.security.web.UserGroupRoleServicesPage`` (Security -> Users, Groups, and Roles)
 
 * 
   Login using the default username ``admin`` and default password ``geoserver``.
@@ -106,10 +96,10 @@ Change geoserver default admin password
 Create public demo user in Magpie for JupyterHub login
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
-Use `\ ``create-magpie-users`` <scripts/create-magpie-users>`_ or follow manual
+Use `create-magpie-users <scripts/create-magpie-users>`_ (:download:`download </birdhouse/scripts/create-magpie-users>`) or follow manual
 instructions below.
 
-``config.yml`` file if using ``create-magpie-users``\ :
+``config.yml`` file if using ``create-magpie-users``:
 
 .. code-block::
 
@@ -121,19 +111,16 @@ instructions below.
 
 Manual instructions:
 
-
 * 
   Go to
-  https://\ :raw-html-m2r:`<PAVICS_FQDN>`\ /magpie/ui/login, login with the ``admin`` user,
-  password should be in ``env.local``.
+  ``https://<PAVICS_FQDN>/magpie/ui/login`` and login with the ``admin`` user. The password should be in ``env.local``.
 
 * 
-  Then go to https://\ :raw-html-m2r:`<PAVICS_FQDN>`\ /magpie/ui/users/add
+  Then go to ``https://<PAVICS_FQDN>/magpie/ui/users/add``.
 
 * 
   Fill in:
 
-
   * User name: <value of JUPYTER_DEMO_USER in ``env.local``\ >
   * Email: < anything is fine >
   * Password: < you decide >
@@ -150,11 +137,11 @@ https://github.com/Ouranosinc/PAVICS-e2e-workflow-tests with pre-configured
 Jenkins at https://github.com/Ouranosinc/jenkins-config.
 
 For that test suite to pass, run the script
-`\ ``bootstrap-instance-for-testsuite`` <scripts/bootstrap-instance-for-testsuite>`_
+`scripts/bootstrap-instance-for-testsuite <scripts/bootstrap-instance-for-testsuite>`_ (:download:`download </birdhouse/scripts/bootstrap-instance-for-testsuite>`)
 to prepare your new instance.  Further documentation inside the script.
 
 Optional component
-`all-public-access <optional-components#give-public-access-to-all-resources-for-testing-purposes>`_
+`all-public-access <./optional-components#give-public-access-to-all-resources-for-testing-purposes>`_
 also need to be enabled in ``env.local``.
 
 ESGF login is also needed for
@@ -175,17 +162,16 @@ environment for testing or to have multiple flavors of PAVICS with slightly
 different combinations of the parts all running simultaneously in their
 respective VM, allowing us to see the differences in behavior.
 
-See `\ ``vagrant_variables.yml.example`` <../vagrant_variables.yml.example>`_ for what's
+See `vagrant_variables.yml.example </vagrant_variables.yml.example>`_ (:download:`download </vagrant_variables.yml.example>`) for what's
 configurable with Vagrant.
 
-If using Centos box, follow `\ ``disk-resize`` <vagrant-utils/disk-resize>`_ after
+If using Centos box, follow `disk-resize <vagrant-utils/disk-resize>`_ (:download:`download </birdhouse/vagrant-utils/disk-resize>`) after
 first ``vagrant up`` failure due to disk full.  Then ``vagrant reload && vagrant
 provision`` to continue.  If using Ubuntu box, no manual steps required,
 everything just works.
 
-Install `VirtualBox <https://www.virtualbox.org/wiki/Downloads>`_\ , both the
-platform and the extension pack, and
-`Vagrant <https://www.vagrantup.com/downloads.html>`_.
+Install `VirtualBox <https://www.virtualbox.org/wiki/Downloads>`_, both the
+platform and the extension pack, and `Vagrant <https://www.vagrantup.com/downloads.html>`_.
 
 One time setup:
 
@@ -214,7 +200,7 @@ Starting and managing the lifecycle of the VM:
    # ex: cd /vagrant/birdhouse; ./pavics-compose.sh ps
    vagrant ssh
 
-   # poweroff VM
+   # power-off VM
    vagrant halt
 
    # delete VM
@@ -224,7 +210,7 @@ Starting and managing the lifecycle of the VM:
    vagrant reload
 
    # provision again (because all subsequent vagrant up won't provision again)
-   # useful to test all provisionning scripts or to bring a VM at unknown state,
+   # useful to test all provisioning scripts or to bring a VM at unknown state,
    # maybe because it was provisioned too long ago, to the latest state.
    # not needed normally during tight development loop
    vagrant provision
@@ -249,7 +235,7 @@ Given a version number MAJOR.MINOR.PATCH, increment the:
 #. 
    MINOR version when we add new components or update existing components
    that also require change to other existing components (ex: new Magpie that
-   also force Twitcher and/or Frondend update) or the change to the existing
+   also force Twitcher and/or Frontend update) or the change to the existing
    component is a major one (ex: major refactoring of Twitcher, big merge
    with corresponding upstream component from birdhouse project).
 

birdhouse/components/README.rst

@@ -1,4 +1,3 @@
-#################
 PAVICS Components
 #################
 
@@ -56,21 +55,21 @@ Given the unattended nature, there is no UI.  Logs are used to keep trace.
 How to Enable the Component
 ---------------------------
 
-- Edit ``env.local`` (a copy of env.local.example_)
+- Edit ``env.local`` (a copy of env.local.example_ (:download:`download <../env.local.example>`))
 
   - Add "./components/scheduler" to ``EXTRA_CONF_DIRS``.
   - Set ``AUTODEPLOY_EXTRA_REPOS``, ``AUTODEPLOY_DEPLOY_KEY_ROOT_DIR``,
     ``AUTODEPLOY_PLATFORM_FREQUENCY``, ``AUTODEPLOY_NOTEBOOK_FREQUENCY`` as desired,
     full documentation in env.local.example_.
-  - Run once fix-write-perm_, see doc in script.
+  - Run once fix-write-perm_ (:download:`download <../deployment/fix-write-perm>`), see doc in script.
 
 
 Old way to deploy the automatic deployment
 ------------------------------------------
 
-Superseeded by this new ``scheduler`` component.  Keeping for reference only.
+Superseded by this new ``scheduler`` component.  Keeping for reference only.
 
-Doing it this old way do not need the ``scheduler`` compoment but lose the
+Doing it this old way do not need the ``scheduler`` component but lose the
 ability for the autodeploy system to update itself.
 
 Configure logrotate for all following automations to prevent disk full::
@@ -83,7 +82,7 @@ To enable continuous deployment of PAVICS::
   # read the script for more options/details
 
 If you want to manually force a deployment of PAVICS (note this might not use
-latest version of deploy.sh script)::
+latest version of deploy.sh_ script (:download:`download <../deployment/deploy.sh>`)::
 
   deployment/deploy.sh .
   # read the script for more options/details
@@ -113,24 +112,27 @@ Comparison between the old and new autodeploy mechanism
 Maximum backward-compatibility has been kept with the old install scripts style:
 
 * Still log to the same existing log files under ``/var/log/PAVICS``.
-* Old single ssh deploy key is still compatible, but the new mechanism allows for different ssh deploy keys for each extra repos (again, public repos should use https clone path to avoid dealing with ssh deploy keys in the first place).
+* Old single ssh deploy key is still compatible, but the new mechanism allows for different ssh deploy keys for each extra
+  repos (again, public repos should use https clone path to avoid dealing with ssh deploy keys in the first place).
 * Old install scripts are kept and can still deploy the old way.
 
 Features missing in old install scripts or how the new mechanism improves on the old install scripts:
 
-* Autodeploy of the autodeploy itself !  This is the biggest win.  Previously, if ``triggerdeploy.sh`` or ``PAVICS-deploy-notebooks`` script changes, they have to be deployed manually.  It's very annoying.  Now they are volume-mount in so are fresh on each run.
-* ``env.local`` now drive absolutely everything, source control that file and we've got a true DevOPS pipeline.
+* Autodeploy of the autodeploy itself !  This is the biggest win.  Previously, if triggerdeploy.sh_ (:download:`download <../deployment/triggerdeploy.sh>`)
+  or the deployed ``/etc/cron.hourly/PAVICS-deploy-notebooks`` script changes, they have to be deployed manually.  It's very annoying.  Now they are volume-mount in so are fresh on each run.
+* ``env.local`` now drives absolutely everything, source control that file and we've got a true DevOPS pipeline.
 * Configurable platform and notebook autodeploy frequency.  Previously, this means manually editing the generated cron file, less ideal.
-* Do not need any support on the local host other than ``docker`` and ``docker-compose``.  ``cron/logrotate/git/ssh`` versions are all locked-down in the docker images used by the autodeploy.  Recall previously we had to deal with git version too old on some hosts.
+* Do not need any support on the local host other than ``docker`` and ``docker-compose``.  ``cron/logrotate/git/ssh``
+  versions are all locked-down in the docker images used by the autodeploy.  Recall previously we had to deal with git version too old on some hosts.
 * Each cron job run in its own docker image meaning the runtime environment is traceable and reproducible.
-* The newly introduced scheduler component is made extensible so other jobs can added into it as well (ex: backup), via ``env.local``, which should be source controlled, meaning all surrounding maintenance related tasks can also be traceable and reproducible.
+* The newly introduced scheduler component is made extensible so other jobs can added into it as well (ex: backup), via ``env.local``,
+  which should be source controlled, meaning all surrounding maintenance related tasks can also be traceable and reproducible.
 
 
 Monitoring
 ==========
 
-This component provides monitoring and alerting for the PAVICS physical host
-and containers.
+This component provides monitoring and alerting for the PAVICS physical host and containers.
 
 Prometheus stack is used:
 
@@ -152,7 +154,7 @@ Usage
 How to Enable the Component
 ---------------------------
 
-- Edit ``env.local`` (a copy of env.local.example_)
+- Edit ``env.local`` (a copy of env.local.example_ (:download:`download <../env.local.example>`))
 
   - Add "./components/monitoring" to ``EXTRA_CONF_DIRS``
   - Set ``GRAFANA_ADMIN_PASSWORD`` to login to Grafana
@@ -231,3 +233,4 @@ Customizing the Component
 .. _env.local.example: ../env.local.example
 .. _fix-write-perm: ../deployment/fix-write-perm
 .. _deploy.sh: ../deployment/deploy.sh
+.. _triggerdeploy.sh: ../deployment/triggerdeploy.sh

birdhouse/docker/geoserver/README.rst

@@ -0,0 +1,20 @@
+docker-geoserver
+================
+
+A simple docker container that runs Geoserver influenced by this docker
+recipe: https://github.com/eliotjordan/docker-geoserver/blob/master/Dockerfile
+and modified for the PAVICS Project.
+
+The actual Geoserver is 2.9.3.
+The geoserver data directory rest on the host and must be mapped on /opt/geoserver/data_dir (container)
+
+Our Dockerfile is a modified version of
+https://hub.docker.com/r/kartoza/geoserver/tags,
+https://github.com/kartoza/docker-geoserver/blob/a71a2aa79315783283a33436f101857ab7eae5a4/Dockerfile.
+
+.. code-block:: shell
+
+   docker build --build-arg -t pavics/geoserver . 
+   docker run --name "postgis" -d -t pavics/postgis:9.4-2.1
+   docker run --name "geoserver" --link postgis:postgis -p <host port>:8080 -v <host path to the geoserver datadir>:/opt/geoserver/data_dir -d -t pavics/geoserver
+   docker run --name "geoserver" --link postgis:postgis -p 8080:8080 -v /data/geoserver_data:/opt/geoserver/data_dir -d -t pavics/geoserver