From 2eb248c343d4b64fbedcc453fc770617ce5d4809 Mon Sep 17 00:00:00 2001 From: Trevor James Smith <10819524+Zeitsperre@users.noreply.github.com> Date: Mon, 25 Jan 2021 13:11:46 -0500 Subject: [PATCH 01/22] Try using a symlink to gain access to Birdhouse-deploy scripts, convert more md to rst, Fix typos and formatting. --- README.rst | 2 +- birdhouse/README.rst | 58 ++++++++++++++++++-------------------------- 2 files changed, 24 insertions(+), 36 deletions(-) diff --git a/README.rst b/README.rst index 031e4403..0f9e2ab8 100644 --- a/README.rst +++ b/README.rst @@ -15,6 +15,6 @@ PAVICS Power Analytics and Visualization for Climate Science - Powered by Birdhouse and other ESGF software How to deploy the entire PAVICS platform, see the -`README `_ and the various extra components +`README `_ and the various extra components `README for extra core components `_\ , `README for optional components `_. diff --git a/birdhouse/README.rst b/birdhouse/README.rst index b6ffd84b..ae1e2c06 100644 --- a/birdhouse/README.rst +++ b/birdhouse/README.rst @@ -1,7 +1,3 @@ -.. role:: raw-html-m2r(raw) - :format: html - - Docker instructions ------------------- @@ -23,22 +19,18 @@ Requirements: Install latest docker-ce and docker-compose for the chosen distro (not the version from the distro). -To run ``docker-compose`` for PAVICS, the `\ ``pavics-compose.sh`` `_ wrapper script must be used. -This script will source the ``env.local`` file, apply the appropriate variable substitutions on all the configuration files ".template", and run ``docker-compose`` with all the command line arguments given to ``pavics-compose.sh``. See `\ ``env.local.example`` `_ for more details on what can go into the ``env.local`` file. +To run ``docker-compose`` for PAVICS, the `pavics-compose.sh `_ wrapper script must be used. +This script will source the ``env.local`` file, apply the appropriate variable substitutions on all the configuration files +".template", and run ``docker-compose`` with all the command line arguments given to `pavics-compose.sh `_. +See `env.local.example `_ for more details on what can go into the ``env.local`` file. -If the file ``env.local`` is somewhere else, symlink it here, next to -``docker-compose.yml`` because many scripts assume this location. +If the file `env.local` is somewhere else, symlink it here, next to `docker-compose.yml `_ because many scripts assume this location. To follow infrastructure-as-code, it is encouraged to source control the above -``env.local`` file and any override needed to customized this PAVICS deployment -for your organization. For an example of possible override, see how the `emu -service `_ -(\ `README `_\ ) can be optionally added to the -deployment via the `override -mechanism `_. Ouranos specific -override can be found in this -`birdhouse-deploy-ouranos `_ -repo. +`env.local` file and any override needed to customized this PAVICS deployment +for your organization. For an example of possible override, see how the `emu service `_ +(`README `_) can be optionally added to the deployment via the `override mechanism `_. +Ouranos specific override can be found in this `birdhouse-deploy-ouranos `_ repo. Suggested deployment layout: @@ -57,16 +49,14 @@ Suggested deployment layout: The automatic deployment is able to handle multiple repos, so will trigger if this repo or your private-personalized-config repo changes, giving you automated continuous deployment. See the continuous deployment setup section -below and the variable ``AUTODEPLOY_EXTRA_REPOS`` in -`\ ``env.local.example`` `_. +below and the variable ``AUTODEPLOY_EXTRA_REPOS`` in `env.local.example `_. The automatic deployment of the PAVICS platform, of the Jupyter tutorial notebooks and of the automatic deployment mechanism itself can all be -enabled by following instructions `here `_. +enabled by following the `scheduling instructions `_. Resource usage monitoring (CPU, memory, ..) and alerting for the host and each -of the containers can be enabled by following instructions -`here `_. +of the containers can be enabled by following the `monitoring instructions `_. To launch all the containers, use the following command: @@ -75,7 +65,7 @@ To launch all the containers, use the following command: ./pavics-compose.sh up -d If you get a ``'No applicable error code, please check error log'`` error from the WPS processes, please make sure that the WPS databases exists in the -postgres instance. See `\ ``scripts/create-wps-pgsql-databases.sh`` `_. +postgres instance. See `create-wps-pgsql-databases.sh `_. Manual steps post deployment ---------------------------- @@ -86,7 +76,7 @@ Change geoserver default admin password * Go to - https://\ :raw-html-m2r:``\ /geoserver/web/wicket/bookmarkable/org.geoserver.security.web.UserGroupRoleServicesPage (Security -> Users, Groups, and Roles) + ``https:///geoserver/web/wicket/bookmarkable/org.geoserver.security.web.UserGroupRoleServicesPage`` (Security -> Users, Groups, and Roles) * Login using the default username ``admin`` and default password ``geoserver``. @@ -106,7 +96,7 @@ Change geoserver default admin password Create public demo user in Magpie for JupyterHub login ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -Use `\ ``create-magpie-users`` `_ or follow manual +Use `create-magpie-users `_ or follow manual instructions below. ``config.yml`` file if using ``create-magpie-users``\ : @@ -124,16 +114,14 @@ Manual instructions: * Go to - https://\ :raw-html-m2r:``\ /magpie/ui/login, login with the ``admin`` user, - password should be in ``env.local``. + ``https:///magpie/ui/login`` and login with the ``admin`` user. The password should be in ``env.local``. * - Then go to https://\ :raw-html-m2r:``\ /magpie/ui/users/add + Then go to ``https:///magpie/ui/users/add``. * Fill in: - * User name: * Email: < anything is fine > * Password: < you decide > @@ -150,11 +138,11 @@ https://github.com/Ouranosinc/PAVICS-e2e-workflow-tests with pre-configured Jenkins at https://github.com/Ouranosinc/jenkins-config. For that test suite to pass, run the script -`\ ``bootstrap-instance-for-testsuite`` `_ +`bootstrap-instance-for-testsuite `_ to prepare your new instance. Further documentation inside the script. Optional component -`all-public-access `_ +`all-public-access `_ also need to be enabled in ``env.local``. ESGF login is also needed for @@ -175,10 +163,10 @@ environment for testing or to have multiple flavors of PAVICS with slightly different combinations of the parts all running simultaneously in their respective VM, allowing us to see the differences in behavior. -See `\ ``vagrant_variables.yml.example`` <../vagrant_variables.yml.example>`_ for what's +See `vagrant_variables.yml.example <../vagrant_variables.yml.example>`_ for what's configurable with Vagrant. -If using Centos box, follow `\ ``disk-resize`` `_ after +If using Centos box, follow `disk-resize `_ after first ``vagrant up`` failure due to disk full. Then ``vagrant reload && vagrant provision`` to continue. If using Ubuntu box, no manual steps required, everything just works. @@ -214,7 +202,7 @@ Starting and managing the lifecycle of the VM: # ex: cd /vagrant/birdhouse; ./pavics-compose.sh ps vagrant ssh - # poweroff VM + # power-off VM vagrant halt # delete VM @@ -224,7 +212,7 @@ Starting and managing the lifecycle of the VM: vagrant reload # provision again (because all subsequent vagrant up won't provision again) - # useful to test all provisionning scripts or to bring a VM at unknown state, + # useful to test all provisioning scripts or to bring a VM at unknown state, # maybe because it was provisioned too long ago, to the latest state. # not needed normally during tight development loop vagrant provision From b10efb00dfcdfd8bfa3df9c4cd8c9f54f1e4d746 Mon Sep 17 00:00:00 2001 From: Trevor James Smith <10819524+Zeitsperre@users.noreply.github.com> Date: Mon, 25 Jan 2021 13:12:13 -0500 Subject: [PATCH 02/22] remove geoserver README.md --- birdhouse/components/README.rst | 1 - birdhouse/docker/geoserver/README.md | 23 ---- birdhouse/docker/geoserver/README.rst | 20 ++++ birdhouse/optional-components/README.rst | 140 +++++++++++++++++++++++ docs/source/deployment.rst | 2 +- docs/source/index.rst | 2 + 6 files changed, 163 insertions(+), 25 deletions(-) delete mode 100644 birdhouse/docker/geoserver/README.md create mode 100644 birdhouse/docker/geoserver/README.rst create mode 100644 birdhouse/optional-components/README.rst diff --git a/birdhouse/components/README.rst b/birdhouse/components/README.rst index e36bd0fc..2ac80a91 100644 --- a/birdhouse/components/README.rst +++ b/birdhouse/components/README.rst @@ -1,4 +1,3 @@ -################# PAVICS Components ################# diff --git a/birdhouse/docker/geoserver/README.md b/birdhouse/docker/geoserver/README.md deleted file mode 100644 index 0af13c61..00000000 --- a/birdhouse/docker/geoserver/README.md +++ /dev/null @@ -1,23 +0,0 @@ -# docker-geoserver - -A simple docker container that runs Geoserver influenced by this docker -recipe: https://github.com/eliotjordan/docker-geoserver/blob/master/Dockerfile -and modified for the PAVICS Project - -The actual Geoserver is 2.9.3. -The geoserver data directory rest on the host and must be mapped on /opt/geoserver/data_dir (container) - -Our Dockerfile is a modified version of -https://hub.docker.com/r/kartoza/geoserver/tags, -https://github.com/kartoza/docker-geoserver/blob/a71a2aa79315783283a33436f101857ab7eae5a4/Dockerfile. - - -```shell -docker build --build-arg -t pavics/geoserver . -docker run --name "postgis" -d -t pavics/postgis:9.4-2.1 - -docker run --name "geoserver" --link postgis:postgis -p :8080 -v :/opt/geoserver/data_dir -d -t pavics/geoserver - -docker run --name "geoserver" --link postgis:postgis -p 8080:8080 -v /data/geoserver_data:/opt/geoserver/data_dir -d -t pavics/geoserver -``` - diff --git a/birdhouse/docker/geoserver/README.rst b/birdhouse/docker/geoserver/README.rst new file mode 100644 index 00000000..2559dc4f --- /dev/null +++ b/birdhouse/docker/geoserver/README.rst @@ -0,0 +1,20 @@ +docker-geoserver +================ + +A simple docker container that runs Geoserver influenced by this docker +recipe: https://github.com/eliotjordan/docker-geoserver/blob/master/Dockerfile +and modified for the PAVICS Project. + +The actual Geoserver is 2.9.3. +The geoserver data directory rest on the host and must be mapped on /opt/geoserver/data_dir (container) + +Our Dockerfile is a modified version of +https://hub.docker.com/r/kartoza/geoserver/tags, +https://github.com/kartoza/docker-geoserver/blob/a71a2aa79315783283a33436f101857ab7eae5a4/Dockerfile. + +.. code-block:: shell + + docker build --build-arg -t pavics/geoserver . + docker run --name "postgis" -d -t pavics/postgis:9.4-2.1 + docker run --name "geoserver" --link postgis:postgis -p :8080 -v :/opt/geoserver/data_dir -d -t pavics/geoserver + docker run --name "geoserver" --link postgis:postgis -p 8080:8080 -v /data/geoserver_data:/opt/geoserver/data_dir -d -t pavics/geoserver diff --git a/birdhouse/optional-components/README.rst b/birdhouse/optional-components/README.rst new file mode 100644 index 00000000..b1069b82 --- /dev/null +++ b/birdhouse/optional-components/README.rst @@ -0,0 +1,140 @@ +Optional components +=================== + +Monitor all components in CANARIE node, both public and internal url +-------------------------------------------------------------------- + +So that the url ``https:///canarie/node/service/stats`` also return +what the end user really see (a component might work but is not accessible to +the end user). + +This assume all the WPS services are public. If not the case, make a copy of +this config and adjust accordingly. + +How to enable this config in ``env.local`` (a copy from +`env.local.example `_): + + +* Add ``./optional-components/canarie-api-full-monitoring`` to ``EXTRA_CONF_DIRS``. + +Emu WPS service for testing +--------------------------- + +Preconfigured for Emu but can also be used to quickly deploy any birds +temporarily without changing code. Good to preview new birds or test +alternative configuration of existing birds. + +No Postgres DB configured. If need Postgres DB, use generic_bird component +instead. + +How to enable Emu in ``env.local`` (a copy from +`env.local.example `_): + + +* Add ``./optional-components/emu`` to ``EXTRA_CONF_DIRS``. +* Optionally set ``EMU_IMAGE``, ``EMU_PORT``, + ``EMU_NAME``, ``EMU_INTERNAL_PORT``, + ``EMU_WPS_OUTPUTS_VOL`` in ``env.local`` for further customizations. + Default values are in `emu/default.env `_. + +Emu service will be available at ``http://PAVICS_FQDN:EMU_PORT/wps`` or +``https://PAVICS_FQDN_PUBLIC/TWITCHER_PROTECTED_PATH/EMU_NAME`` where +``PAVICS_FQDN``\ , ``PAVICS_FQDN_PUBLIC`` and ``TWITCHER_PROTECTED_PATH`` are defined +in your ``env.local``. + +Magpie will be automatically configured to give complete public anonymous +access for this Emu WPS service. + +CANARIE monitoring will also be automatically configured for this Emu WPS +service. + +A second THREDDs server for testing +----------------------------------- + +How to enable in ``env.local`` (a copy from `env.local.example `_): + + +* + Add ``./optional-components/testthredds`` to ``EXTRA_CONF_DIRS``. + +* + Optionally set ``TESTTHREDDS_IMAGE``\ , ``TESTTHREDDS_PORT``\ , + ``TESTTHREDDS_CONTEXT_ROOT``\ , ``TESTTHREDDS_WARFILE_NAME``\ , + ``TESTTHREDDS_INTERNAL_PORT``\ , ``TESTTHREDDS_NAME``\ , in ``env.local`` for further + customizations. Default values are in + `testthredds/default.env `_. + +Test THREDDs service will be available at +``http://PAVICS_FQDN:TESTTHREDDS_PORT/TESTTHREDDS_CONTEXT_ROOT`` or +``https://PAVICS_FQDN_PUBLIC/TESTTHREDDS_CONTEXT_ROOT`` where ``PAVICS_FQDN`` and +``PAVICS_FQDN_PUBLIC`` are defined in your ``env.local``. + +Use same docker image as regular THREDDs by default but can be customized. + +New container have new ``TestDatasets`` with volume-mount to ``/data/testdatasets`` +on the host. So your testing ``.nc`` and ``.ncml`` files should be added to +``/data/testdatasets`` on the host for them to show up on this Test THREDDs +server. + +``TestWps_Output`` dataset is for other WPS services to write to, similar to +``birdhouse/wps_outputs`` dataset in the production THREDDs. With Emu, add +``export EMU_WPS_OUTPUTS_VOL=testwps_outputs`` to ``env.local`` for Emu to write to +``TestWps_Output`` dataset. + +No Twitcher/Magpie access control, this Test THREDDs is directly behind the +Nginx proxy. + +CANARIE monitoring will also be automatically configured for this second +THREDDs server. + +A generic bird WPS service +-------------------------- + +Can be used to quickly deploy any birds temporarily without changing code. +Good to preview new birds or test alternative configuration of existing birds. + +How to enable in ``env.local`` (a copy from `env.local.example `_): + + +* + Add ``./optional-components/generic_bird`` to ``EXTRA_CONF_DIRS``. + +* + Optionally set ``GENERIC_BIRD_IMAGE``, ``GENERIC_BIRD_PORT``, + ``GENERIC_BIRD_NAME``, ``GENERIC_BIRD_INTERNAL_PORT``, and + ``GENERIC_BIRD_POSTGRES_IMAGE`` in ``env.local`` for further customizations. + Default values are in `generic_bird/default.env `_. + +The WPS service will be available at ``http://PAVICS_FQDN:GENERIC_BIRD_PORT/wps`` +or ``https://PAVICS_FQDN_PUBLIC/TWITCHER_PROTECTED_PATH/GENERIC_BIRD_NAME`` where +``PAVICS_FQDN``\ , ``PAVICS_FQDN_PUBLIC`` and ``TWITCHER_PROTECTED_PATH`` are defined +in your ``env.local``. + +Use same docker image as regular Finch by default but can be customized. + +Use a separate Postgres DB for this optional component to be completely +self-contained and to allow experimenting with different versions of Postgres +DB. + +Magpie will be automatically configured to give complete public anonymous +access for this WPS service. + +CANARIE monitoring will also be automatically configured for this WPS service. + +Give public access to all resources for testing purposes +-------------------------------------------------------- + +By enabling this component, all WPS services and data on THREDDs are completely public, please beware. +Once enabled, if you need to revert the change, you have to do it manually by logging into Magpie. +Just disabling this component will not revert the change. + +This optional component is required for the test suite at +https://github.com/Ouranosinc/PAVICS-e2e-workflow-tests. + +How to enable in ``env.local`` (a copy from +`env.local.example <../env.local.example>`_): + + +* Add ``./optional-components/all-public-access`` to ``EXTRA_CONF_DIRS``. + +The anonymous user will now have all the permissions described in `optional-components/all-public-access/all-public-access-magpie-permission.cfg `_. diff --git a/docs/source/deployment.rst b/docs/source/deployment.rst index 5b9a7544..eafa146d 100644 --- a/docs/source/deployment.rst +++ b/docs/source/deployment.rst @@ -1,4 +1,4 @@ Deployment ========== -.. include:: ../../birdhouse/README.rst \ No newline at end of file +.. include:: birdhouse/README.rst \ No newline at end of file diff --git a/docs/source/index.rst b/docs/source/index.rst index 7d4b3d61..069bbda3 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -20,6 +20,8 @@ Contents: installation deployment + birdhouse/components/README + birdhouse/docker/geoserver/README data_catalog From aa57c884f3e090372e11705607ad932d2fc5957d Mon Sep 17 00:00:00 2001 From: Trevor James Smith <10819524+Zeitsperre@users.noreply.github.com> Date: Mon, 25 Jan 2021 14:20:30 -0500 Subject: [PATCH 03/22] Remove old README.md, update paths in conf.py, add optional-components, link to birdhouse and vagrant_variables.yml.example, more locations fixes --- birdhouse/README.rst | 12 +- birdhouse/optional-components/README.md | 131 ---------------------- docs/source/birdhouse | 1 + docs/source/conf.py | 9 +- docs/source/index.rst | 1 + docs/source/vagrant_variables.yml.example | 1 + 6 files changed, 13 insertions(+), 142 deletions(-) delete mode 100644 birdhouse/optional-components/README.md create mode 120000 docs/source/birdhouse create mode 120000 docs/source/vagrant_variables.yml.example diff --git a/birdhouse/README.rst b/birdhouse/README.rst index ae1e2c06..99363573 100644 --- a/birdhouse/README.rst +++ b/birdhouse/README.rst @@ -29,7 +29,7 @@ If the file `env.local` is somewhere else, symlink it here, next to `docker-comp To follow infrastructure-as-code, it is encouraged to source control the above `env.local` file and any override needed to customized this PAVICS deployment for your organization. For an example of possible override, see how the `emu service `_ -(`README `_) can be optionally added to the deployment via the `override mechanism `_. +(`README `_) can be optionally added to the deployment via the `override mechanism `_. Ouranos specific override can be found in this `birdhouse-deploy-ouranos `_ repo. Suggested deployment layout: @@ -99,7 +99,7 @@ Create public demo user in Magpie for JupyterHub login Use `create-magpie-users `_ or follow manual instructions below. -``config.yml`` file if using ``create-magpie-users``\ : +``config.yml`` file if using ``create-magpie-users``: .. code-block:: @@ -111,7 +111,6 @@ instructions below. Manual instructions: - * Go to ``https:///magpie/ui/login`` and login with the ``admin`` user. The password should be in ``env.local``. @@ -163,7 +162,7 @@ environment for testing or to have multiple flavors of PAVICS with slightly different combinations of the parts all running simultaneously in their respective VM, allowing us to see the differences in behavior. -See `vagrant_variables.yml.example <../vagrant_variables.yml.example>`_ for what's +See `vagrant_variables.yml.example `_ for what's configurable with Vagrant. If using Centos box, follow `disk-resize `_ after @@ -171,9 +170,8 @@ first ``vagrant up`` failure due to disk full. Then ``vagrant reload && vagrant provision`` to continue. If using Ubuntu box, no manual steps required, everything just works. -Install `VirtualBox `_\ , both the -platform and the extension pack, and -`Vagrant `_. +Install `VirtualBox `_, both the +platform and the extension pack, and `Vagrant `_. One time setup: diff --git a/birdhouse/optional-components/README.md b/birdhouse/optional-components/README.md deleted file mode 100644 index 8ae79011..00000000 --- a/birdhouse/optional-components/README.md +++ /dev/null @@ -1,131 +0,0 @@ -# Optional components - -## Monitor all components in Canarie node, both public and internal url - -So that the url https:///canarie/node/service/stats also return -what the end user really see (a component might work but is not accessible to -the end user). - -This assume all the WPS services are public. If not the case, make a copy of -this config and adjust accordingly. - -How to enable this config in `env.local` (a copy from -[`env.local.example`](../env.local.example)): - -* Add `./optional-components/canarie-api-full-monitoring` to `EXTRA_CONF_DIRS`. - - -## Emu WPS service for testing - -Preconfigured for Emu but can also be used to quickly deploy any birds -temporarily without changing code. Good to preview new birds or test -alternative configuration of existing birds. - -No Postgres DB configured. If need Postgres DB, use generic_bird component -instead. - -How to enable Emu in `env.local` (a copy from -[`env.local.example`](../env.local.example)): - -* Add `./optional-components/emu` to `EXTRA_CONF_DIRS`. -* Optionally set `EMU_IMAGE`, `EMU_PORT`, - `EMU_NAME`, `EMU_INTERNAL_PORT`, - `EMU_WPS_OUTPUTS_VOL` in `env.local` for further customizations. - Default values are in [`emu/default.env`](emu/default.env). - -Emu service will be available at `http://PAVICS_FQDN:EMU_PORT/wps` or -`https://PAVICS_FQDN_PUBLIC/TWITCHER_PROTECTED_PATH/EMU_NAME` where -`PAVICS_FQDN`, `PAVICS_FQDN_PUBLIC` and `TWITCHER_PROTECTED_PATH` are defined -in your `env.local`. - -Magpie will be automatically configured to give complete public anonymous -access for this Emu WPS service. - -Canarie monitoring will also be automatically configured for this Emu WPS -service. - - -## A second Thredds server for testing - -How to enable in `env.local` (a copy from -[`env.local.example`](../env.local.example)): - -* Add `./optional-components/testthredds` to `EXTRA_CONF_DIRS`. - -* Optionally set `TESTTHREDDS_IMAGE`, `TESTTHREDDS_PORT`, - `TESTTHREDDS_CONTEXT_ROOT`, `TESTTHREDDS_WARFILE_NAME`, - `TESTTHREDDS_INTERNAL_PORT`, `TESTTHREDDS_NAME`, in `env.local` for further - customizations. Default values are in - [`testthredds/default.env`](testthredds/default.env). - -Test Thredds service will be available at -`http://PAVICS_FQDN:TESTTHREDDS_PORT/TESTTHREDDS_CONTEXT_ROOT` or -`https://PAVICS_FQDN_PUBLIC/TESTTHREDDS_CONTEXT_ROOT` where `PAVICS_FQDN` and -`PAVICS_FQDN_PUBLIC` are defined in your `env.local`. - -Use same docker image as regular Thredds by default but can be customized. - -New container have new `TestDatasets` with volume-mount to `/data/testdatasets` -on the host. So your testing `.nc` and `.ncml` files should be added to -`/data/testdatasets` on the host for them to show up on this Test Thredds -server. - -`TestWps_Output` dataset is for other WPS services to write to, similar to -`birdhouse/wps_outputs` dataset in the production Thredds. With Emu, add -`export EMU_WPS_OUTPUTS_VOL=testwps_outputs` to `env.local` for Emu to write to -`TestWps_Output` dataset. - -No Twitcher/Magpie access control, this Test Thredds is directly behind the -Nginx proxy. - -Canarie monitoring will also be automatically configured for this second -Thredds server. - - -## A generic bird WPS service - -Can be used to quickly deploy any birds temporarily without changing code. -Good to preview new birds or test alternative configuration of existing birds. - -How to enable in `env.local` (a copy from -[`env.local.example`](../env.local.example)): - -* Add `./optional-components/generic_bird` to `EXTRA_CONF_DIRS`. - -* Optionally set `GENERIC_BIRD_IMAGE`, `GENERIC_BIRD_PORT`, - `GENERIC_BIRD_NAME`, `GENERIC_BIRD_INTERNAL_PORT`, - `GENERIC_BIRD_POSTGRES_IMAGE` in `env.local` for further customizations. - Default values are in [`generic_bird/default.env`](generic_bird/default.env). - -The WPS service will be available at `http://PAVICS_FQDN:GENERIC_BIRD_PORT/wps` -or `https://PAVICS_FQDN_PUBLIC/TWITCHER_PROTECTED_PATH/GENERIC_BIRD_NAME` where -`PAVICS_FQDN`, `PAVICS_FQDN_PUBLIC` and `TWITCHER_PROTECTED_PATH` are defined -in your `env.local`. - -Use same docker image as regular Finch by default but can be customized. - -Use a separate Postgres DB for this optional component to be completely -self-contained and to allow experimenting with different versions of Postgres -DB. - -Magpie will be automatically configured to give complete public anonymous -access for this WPS service. - -Canarie monitoring will also be automatically configured for this WPS service. - - -## Give public access to all resources for testing purposes - -By enabling this component, all WPS services and data on Thredds are completely public, please beware. -Once enabled, if you need to revert the change, you have to do it manually by logging into Magpie. -Just disabling this component will not revert the change. - -This optional component is required for the test suite at -https://github.com/Ouranosinc/PAVICS-e2e-workflow-tests. - -How to enable in `env.local` (a copy from -[`env.local.example`](../env.local.example)): - -* Add `./optional-components/all-public-access` to `EXTRA_CONF_DIRS`. - -The anonymous user will now have all the permissions described in [`./optional-components/all-public-access/all-public-access-magpie-permission.cfg`](all-public-access/all-public-access-magpie-permission.cfg). diff --git a/docs/source/birdhouse b/docs/source/birdhouse new file mode 120000 index 00000000..5e32e274 --- /dev/null +++ b/docs/source/birdhouse @@ -0,0 +1 @@ +../../birdhouse \ No newline at end of file diff --git a/docs/source/conf.py b/docs/source/conf.py index 09cf3e43..da4dd1f1 100644 --- a/docs/source/conf.py +++ b/docs/source/conf.py @@ -17,9 +17,10 @@ # add these directories to sys.path here. If the directory is relative to the # documentation root, use os.path.abspath to make it absolute, like shown here. # -# import os -# import sys -# sys.path.insert(0, os.path.abspath('.')) +import os +import sys +sys.path.insert(0, os.path.abspath('.')) +sys.path.insert(0, os.path.abspath('../../birdhouse')) # -- General configuration ------------------------------------------------ @@ -48,7 +49,7 @@ # The suffix(es) of source filenames. # You can specify multiple suffix as a list of string: # -source_suffix = ['.rst', '.md'] +source_suffix = ['.rst', '.md', '.yml', '.sh', '.env', '.local'] # The encoding of source files. # diff --git a/docs/source/index.rst b/docs/source/index.rst index 069bbda3..95412756 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -22,6 +22,7 @@ Contents: deployment birdhouse/components/README birdhouse/docker/geoserver/README + birdhouse/optional-components/README data_catalog diff --git a/docs/source/vagrant_variables.yml.example b/docs/source/vagrant_variables.yml.example new file mode 120000 index 00000000..aa3e9254 --- /dev/null +++ b/docs/source/vagrant_variables.yml.example @@ -0,0 +1 @@ +../../vagrant_variables.yml.example \ No newline at end of file From e575452d3074a474bfb43eb5d09311443465aeda Mon Sep 17 00:00:00 2001 From: Trevor James Smith <10819524+Zeitsperre@users.noreply.github.com> Date: Mon, 25 Jan 2021 14:46:24 -0500 Subject: [PATCH 04/22] Fix broken relpaths, update tope-level README.rst, adjust order of pages --- README.rst | 2 +- birdhouse/optional-components/README.rst | 12 ++++++------ docs/source/conf.py | 4 ++-- docs/source/index.rst | 2 +- 4 files changed, 10 insertions(+), 10 deletions(-) diff --git a/README.rst b/README.rst index 0f9e2ab8..27117b8b 100644 --- a/README.rst +++ b/README.rst @@ -17,4 +17,4 @@ Power Analytics and Visualization for Climate Science - Powered by Birdhouse and How to deploy the entire PAVICS platform, see the `README `_ and the various extra components `README for extra core components `_\ , -`README for optional components `_. +`README for optional components `_. diff --git a/birdhouse/optional-components/README.rst b/birdhouse/optional-components/README.rst index b1069b82..431a46ca 100644 --- a/birdhouse/optional-components/README.rst +++ b/birdhouse/optional-components/README.rst @@ -28,14 +28,14 @@ No Postgres DB configured. If need Postgres DB, use generic_bird component instead. How to enable Emu in ``env.local`` (a copy from -`env.local.example `_): +`env.local.example <../env.local.example>`_): * Add ``./optional-components/emu`` to ``EXTRA_CONF_DIRS``. * Optionally set ``EMU_IMAGE``, ``EMU_PORT``, ``EMU_NAME``, ``EMU_INTERNAL_PORT``, ``EMU_WPS_OUTPUTS_VOL`` in ``env.local`` for further customizations. - Default values are in `emu/default.env `_. + Default values are in `emu/default.env `_. Emu service will be available at ``http://PAVICS_FQDN:EMU_PORT/wps`` or ``https://PAVICS_FQDN_PUBLIC/TWITCHER_PROTECTED_PATH/EMU_NAME`` where @@ -51,7 +51,7 @@ service. A second THREDDs server for testing ----------------------------------- -How to enable in ``env.local`` (a copy from `env.local.example `_): +How to enable in ``env.local`` (a copy from `env.local.example <../env.local.example>`_): * @@ -62,7 +62,7 @@ How to enable in ``env.local`` (a copy from `env.local.example `_. + `testthredds/default.env `_. Test THREDDs service will be available at ``http://PAVICS_FQDN:TESTTHREDDS_PORT/TESTTHREDDS_CONTEXT_ROOT`` or @@ -93,7 +93,7 @@ A generic bird WPS service Can be used to quickly deploy any birds temporarily without changing code. Good to preview new birds or test alternative configuration of existing birds. -How to enable in ``env.local`` (a copy from `env.local.example `_): +How to enable in ``env.local`` (a copy from `env.local.example <../env.local.example>`_): * @@ -137,4 +137,4 @@ How to enable in ``env.local`` (a copy from * Add ``./optional-components/all-public-access`` to ``EXTRA_CONF_DIRS``. -The anonymous user will now have all the permissions described in `optional-components/all-public-access/all-public-access-magpie-permission.cfg `_. +The anonymous user will now have all the permissions described in `optional-components/all-public-access/all-public-access-magpie-permission.cfg `_. diff --git a/docs/source/conf.py b/docs/source/conf.py index da4dd1f1..892330f2 100644 --- a/docs/source/conf.py +++ b/docs/source/conf.py @@ -49,7 +49,7 @@ # The suffix(es) of source filenames. # You can specify multiple suffix as a list of string: # -source_suffix = ['.rst', '.md', '.yml', '.sh', '.env', '.local'] +source_suffix = ['.rst', '.md'] # The encoding of source files. # @@ -206,7 +206,7 @@ # If true, links to the reST sources are added to the pages. # -# html_show_sourcelink = True +html_show_sourcelink = True # If true, "Created using Sphinx" is shown in the HTML footer. Default is True. # diff --git a/docs/source/index.rst b/docs/source/index.rst index 95412756..080c56b8 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -20,8 +20,8 @@ Contents: installation deployment - birdhouse/components/README birdhouse/docker/geoserver/README + birdhouse/components/README birdhouse/optional-components/README data_catalog From 89a070e1490b7e571520b0cf96304f2bc1c852da Mon Sep 17 00:00:00 2001 From: Trevor James Smith <10819524+Zeitsperre@users.noreply.github.com> Date: Mon, 25 Jan 2021 16:02:21 -0500 Subject: [PATCH 05/22] Use the download role for non-rst files in optional-components README.rst --- birdhouse/optional-components/README.rst | 14 ++++++-------- 1 file changed, 6 insertions(+), 8 deletions(-) diff --git a/birdhouse/optional-components/README.rst b/birdhouse/optional-components/README.rst index 431a46ca..ec28cccf 100644 --- a/birdhouse/optional-components/README.rst +++ b/birdhouse/optional-components/README.rst @@ -11,8 +11,7 @@ the end user). This assume all the WPS services are public. If not the case, make a copy of this config and adjust accordingly. -How to enable this config in ``env.local`` (a copy from -`env.local.example `_): +How to enable this config in ``env.local`` (a copy from :download:`env.local.example <../env.local.example>`): * Add ``./optional-components/canarie-api-full-monitoring`` to ``EXTRA_CONF_DIRS``. @@ -28,7 +27,7 @@ No Postgres DB configured. If need Postgres DB, use generic_bird component instead. How to enable Emu in ``env.local`` (a copy from -`env.local.example <../env.local.example>`_): +:download:`env.local.example <../env.local.example>`): * Add ``./optional-components/emu`` to ``EXTRA_CONF_DIRS``. @@ -51,7 +50,7 @@ service. A second THREDDs server for testing ----------------------------------- -How to enable in ``env.local`` (a copy from `env.local.example <../env.local.example>`_): +How to enable in ``env.local`` (a copy from :download:`env.local.example <../env.local.example>`): * @@ -93,7 +92,7 @@ A generic bird WPS service Can be used to quickly deploy any birds temporarily without changing code. Good to preview new birds or test alternative configuration of existing birds. -How to enable in ``env.local`` (a copy from `env.local.example <../env.local.example>`_): +How to enable in ``env.local`` (a copy from :download:`env.local.example <../env.local.example>`): * @@ -103,7 +102,7 @@ How to enable in ``env.local`` (a copy from `env.local.example <../env.local.exa Optionally set ``GENERIC_BIRD_IMAGE``, ``GENERIC_BIRD_PORT``, ``GENERIC_BIRD_NAME``, ``GENERIC_BIRD_INTERNAL_PORT``, and ``GENERIC_BIRD_POSTGRES_IMAGE`` in ``env.local`` for further customizations. - Default values are in `generic_bird/default.env `_. + Default values are in `generic_bird/default.env `_. The WPS service will be available at ``http://PAVICS_FQDN:GENERIC_BIRD_PORT/wps`` or ``https://PAVICS_FQDN_PUBLIC/TWITCHER_PROTECTED_PATH/GENERIC_BIRD_NAME`` where @@ -131,8 +130,7 @@ Just disabling this component will not revert the change. This optional component is required for the test suite at https://github.com/Ouranosinc/PAVICS-e2e-workflow-tests. -How to enable in ``env.local`` (a copy from -`env.local.example <../env.local.example>`_): +How to enable in ``env.local`` (a copy from :download:`env.local.example <../env.local.example>`): * Add ``./optional-components/all-public-access`` to ``EXTRA_CONF_DIRS``. From 533ac93a1ec61fbfc9dc19ba04f685b67b5b64ab Mon Sep 17 00:00:00 2001 From: Trevor James Smith <10819524+Zeitsperre@users.noreply.github.com> Date: Mon, 25 Jan 2021 16:03:58 -0500 Subject: [PATCH 06/22] Test with relpath --- birdhouse/optional-components/README.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/birdhouse/optional-components/README.rst b/birdhouse/optional-components/README.rst index ec28cccf..8fe9fc16 100644 --- a/birdhouse/optional-components/README.rst +++ b/birdhouse/optional-components/README.rst @@ -11,7 +11,7 @@ the end user). This assume all the WPS services are public. If not the case, make a copy of this config and adjust accordingly. -How to enable this config in ``env.local`` (a copy from :download:`env.local.example <../env.local.example>`): +How to enable this config in ``env.local`` (a copy from `env.local.example <../env.local.example>`_): * Add ``./optional-components/canarie-api-full-monitoring`` to ``EXTRA_CONF_DIRS``. From 2d5b5e93516bf25955356b886a8ee6e7c0858e72 Mon Sep 17 00:00:00 2001 From: Trevor James Smith <10819524+Zeitsperre@users.noreply.github.com> Date: Mon, 25 Jan 2021 16:19:43 -0500 Subject: [PATCH 07/22] Try more relpaths, prioritize GitHub --- birdhouse/README.rst | 32 ++++++++++++++++---------------- 1 file changed, 16 insertions(+), 16 deletions(-) diff --git a/birdhouse/README.rst b/birdhouse/README.rst index 99363573..f090885f 100644 --- a/birdhouse/README.rst +++ b/birdhouse/README.rst @@ -19,17 +19,17 @@ Requirements: Install latest docker-ce and docker-compose for the chosen distro (not the version from the distro). -To run ``docker-compose`` for PAVICS, the `pavics-compose.sh `_ wrapper script must be used. +To run ``docker-compose`` for PAVICS, the `pavics-compose.sh <./pavics-compose.sh>`_ wrapper script must be used. This script will source the ``env.local`` file, apply the appropriate variable substitutions on all the configuration files -".template", and run ``docker-compose`` with all the command line arguments given to `pavics-compose.sh `_. -See `env.local.example `_ for more details on what can go into the ``env.local`` file. +".template", and run ``docker-compose`` with all the command line arguments given to `pavics-compose.sh <./pavics-compose.sh>`_. +See `env.local.example <./env.local.example>`_ for more details on what can go into the ``env.local`` file. -If the file `env.local` is somewhere else, symlink it here, next to `docker-compose.yml `_ because many scripts assume this location. +If the file `env.local` is somewhere else, symlink it here, next to `docker-compose.yml `_ because many scripts assume this location. To follow infrastructure-as-code, it is encouraged to source control the above `env.local` file and any override needed to customized this PAVICS deployment -for your organization. For an example of possible override, see how the `emu service `_ -(`README `_) can be optionally added to the deployment via the `override mechanism `_. +for your organization. For an example of possible override, see how the `emu service <./optional-components/emu/docker-compose-extra.yml>`_ +(`README <./optional-components/README.rst>`_) can be optionally added to the deployment via the `override mechanism `_. Ouranos specific override can be found in this `birdhouse-deploy-ouranos `_ repo. Suggested deployment layout: @@ -49,14 +49,14 @@ Suggested deployment layout: The automatic deployment is able to handle multiple repos, so will trigger if this repo or your private-personalized-config repo changes, giving you automated continuous deployment. See the continuous deployment setup section -below and the variable ``AUTODEPLOY_EXTRA_REPOS`` in `env.local.example `_. +below and the variable ``AUTODEPLOY_EXTRA_REPOS`` in `env.local.example <./env.local.example>`_. The automatic deployment of the PAVICS platform, of the Jupyter tutorial notebooks and of the automatic deployment mechanism itself can all be -enabled by following the `scheduling instructions `_. +enabled by following the `scheduling instructions <./components/README.rst#scheduler>`_. Resource usage monitoring (CPU, memory, ..) and alerting for the host and each -of the containers can be enabled by following the `monitoring instructions `_. +of the containers can be enabled by following the `monitoring instructions <./components/README.rst#monitoring>`_. To launch all the containers, use the following command: @@ -65,7 +65,7 @@ To launch all the containers, use the following command: ./pavics-compose.sh up -d If you get a ``'No applicable error code, please check error log'`` error from the WPS processes, please make sure that the WPS databases exists in the -postgres instance. See `create-wps-pgsql-databases.sh `_. +postgres instance. See `create-wps-pgsql-databases.sh <./scripts/create-wps-pgsql-databases.sh>`_. Manual steps post deployment ---------------------------- @@ -96,7 +96,7 @@ Change geoserver default admin password Create public demo user in Magpie for JupyterHub login ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -Use `create-magpie-users `_ or follow manual +Use `create-magpie-users <./scripts/create-magpie-users>`_ or follow manual instructions below. ``config.yml`` file if using ``create-magpie-users``: @@ -137,11 +137,11 @@ https://github.com/Ouranosinc/PAVICS-e2e-workflow-tests with pre-configured Jenkins at https://github.com/Ouranosinc/jenkins-config. For that test suite to pass, run the script -`bootstrap-instance-for-testsuite `_ +`bootstrap-instance-for-testsuite <./scripts/bootstrap-instance-for-testsuite>`_ to prepare your new instance. Further documentation inside the script. Optional component -`all-public-access `_ +`all-public-access <./optional-components#give-public-access-to-all-resources-for-testing-purposes>`_ also need to be enabled in ``env.local``. ESGF login is also needed for @@ -162,10 +162,10 @@ environment for testing or to have multiple flavors of PAVICS with slightly different combinations of the parts all running simultaneously in their respective VM, allowing us to see the differences in behavior. -See `vagrant_variables.yml.example `_ for what's +See `vagrant_variables.yml.example <./vagrant_variables.yml.example>`_ for what's configurable with Vagrant. -If using Centos box, follow `disk-resize `_ after +If using Centos box, follow `disk-resize <./vagrant-utils/disk-resize>`_ after first ``vagrant up`` failure due to disk full. Then ``vagrant reload && vagrant provision`` to continue. If using Ubuntu box, no manual steps required, everything just works. @@ -235,7 +235,7 @@ Given a version number MAJOR.MINOR.PATCH, increment the: #. MINOR version when we add new components or update existing components that also require change to other existing components (ex: new Magpie that - also force Twitcher and/or Frondend update) or the change to the existing + also force Twitcher and/or Frontend update) or the change to the existing component is a major one (ex: major refactoring of Twitcher, big merge with corresponding upstream component from birdhouse project). From 4356cb9941187633ad6173c475e08788ad0c4b51 Mon Sep 17 00:00:00 2001 From: Trevor James Smith <10819524+Zeitsperre@users.noreply.github.com> Date: Mon, 25 Jan 2021 16:33:53 -0500 Subject: [PATCH 08/22] mroe tests --- birdhouse/optional-components/README.rst | 14 ++++++-------- docs/source/conf.py | 2 +- 2 files changed, 7 insertions(+), 9 deletions(-) diff --git a/birdhouse/optional-components/README.rst b/birdhouse/optional-components/README.rst index 8fe9fc16..4ad03386 100644 --- a/birdhouse/optional-components/README.rst +++ b/birdhouse/optional-components/README.rst @@ -11,7 +11,7 @@ the end user). This assume all the WPS services are public. If not the case, make a copy of this config and adjust accordingly. -How to enable this config in ``env.local`` (a copy from `env.local.example <../env.local.example>`_): +How to enable this config in ``env.local`` (a copy from `/birdhouse/env.local.example`_ or :download:`download it here `): * Add ``./optional-components/canarie-api-full-monitoring`` to ``EXTRA_CONF_DIRS``. @@ -26,8 +26,7 @@ alternative configuration of existing birds. No Postgres DB configured. If need Postgres DB, use generic_bird component instead. -How to enable Emu in ``env.local`` (a copy from -:download:`env.local.example <../env.local.example>`): +How to enable Emu in ``env.local`` (a copy from `/birdhouse/env.local.example` or :download:`download it here `): * Add ``./optional-components/emu`` to ``EXTRA_CONF_DIRS``. @@ -50,7 +49,7 @@ service. A second THREDDs server for testing ----------------------------------- -How to enable in ``env.local`` (a copy from :download:`env.local.example <../env.local.example>`): +How to enable in ``env.local`` (a copy from `/birdhouse/env.local.example` or :download:`download it here `.): * @@ -60,8 +59,7 @@ How to enable in ``env.local`` (a copy from :download:`env.local.example <../env Optionally set ``TESTTHREDDS_IMAGE``\ , ``TESTTHREDDS_PORT``\ , ``TESTTHREDDS_CONTEXT_ROOT``\ , ``TESTTHREDDS_WARFILE_NAME``\ , ``TESTTHREDDS_INTERNAL_PORT``\ , ``TESTTHREDDS_NAME``\ , in ``env.local`` for further - customizations. Default values are in - `testthredds/default.env `_. + customizations. Default values are in: `/birdhouse/optional-components/testthredds/default.env`. Test THREDDs service will be available at ``http://PAVICS_FQDN:TESTTHREDDS_PORT/TESTTHREDDS_CONTEXT_ROOT`` or @@ -92,7 +90,7 @@ A generic bird WPS service Can be used to quickly deploy any birds temporarily without changing code. Good to preview new birds or test alternative configuration of existing birds. -How to enable in ``env.local`` (a copy from :download:`env.local.example <../env.local.example>`): +How to enable in ``env.local`` (a copy from ` ./env.local.example`_ or :download:`download it here `.): * @@ -130,7 +128,7 @@ Just disabling this component will not revert the change. This optional component is required for the test suite at https://github.com/Ouranosinc/PAVICS-e2e-workflow-tests. -How to enable in ``env.local`` (a copy from :download:`env.local.example <../env.local.example>`): +How to enable in ``env.local`` (a copy from `../env.local.example`_ or :download:`download it here `.): * Add ``./optional-components/all-public-access`` to ``EXTRA_CONF_DIRS``. diff --git a/docs/source/conf.py b/docs/source/conf.py index 892330f2..f4662f6d 100644 --- a/docs/source/conf.py +++ b/docs/source/conf.py @@ -20,7 +20,7 @@ import os import sys sys.path.insert(0, os.path.abspath('.')) -sys.path.insert(0, os.path.abspath('../../birdhouse')) +sys.path.insert(0, os.path.abspath('../../.')) # -- General configuration ------------------------------------------------ From 88f29ec6f38058b714696efc1f4ddb27d34e31ef Mon Sep 17 00:00:00 2001 From: Trevor James Smith <10819524+Zeitsperre@users.noreply.github.com> Date: Mon, 25 Jan 2021 16:36:51 -0500 Subject: [PATCH 09/22] more tests --- birdhouse/optional-components/README.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/birdhouse/optional-components/README.rst b/birdhouse/optional-components/README.rst index 4ad03386..13a7f0d4 100644 --- a/birdhouse/optional-components/README.rst +++ b/birdhouse/optional-components/README.rst @@ -11,7 +11,7 @@ the end user). This assume all the WPS services are public. If not the case, make a copy of this config and adjust accordingly. -How to enable this config in ``env.local`` (a copy from `/birdhouse/env.local.example`_ or :download:`download it here `): +How to enable this config in ``env.local`` (a copy from `<../env.local.example>`_ or :download:`download it here `): * Add ``./optional-components/canarie-api-full-monitoring`` to ``EXTRA_CONF_DIRS``. @@ -128,7 +128,7 @@ Just disabling this component will not revert the change. This optional component is required for the test suite at https://github.com/Ouranosinc/PAVICS-e2e-workflow-tests. -How to enable in ``env.local`` (a copy from `../env.local.example`_ or :download:`download it here `.): +How to enable in ``env.local`` (a copy from `<../env.local.example>`_ or :download:`download it here `.): * Add ``./optional-components/all-public-access`` to ``EXTRA_CONF_DIRS``. From d82e1818f13bf5c18ea8507fa9160f9e4624a5af Mon Sep 17 00:00:00 2001 From: Trevor James Smith <10819524+Zeitsperre@users.noreply.github.com> Date: Mon, 25 Jan 2021 17:26:50 -0500 Subject: [PATCH 10/22] Links now fixed for GitHub navigation and for downloading of scripts on RtD --- README.rst | 9 ++++---- birdhouse/README.rst | 28 ++++++++++++------------ birdhouse/optional-components/README.rst | 18 +++++++-------- 3 files changed, 28 insertions(+), 27 deletions(-) diff --git a/README.rst b/README.rst index 27117b8b..9a815a87 100644 --- a/README.rst +++ b/README.rst @@ -14,7 +14,8 @@ PAVICS Power Analytics and Visualization for Climate Science - Powered by Birdhouse and other ESGF software -How to deploy the entire PAVICS platform, see the -`README `_ and the various extra components -`README for extra core components `_\ , -`README for optional components `_. +For GitHub navigation, see the following README pages: + +* `README for general deployment `_ +* `README for extra core components `_ +* `README for optional components `_ diff --git a/birdhouse/README.rst b/birdhouse/README.rst index f090885f..01da3648 100644 --- a/birdhouse/README.rst +++ b/birdhouse/README.rst @@ -19,17 +19,17 @@ Requirements: Install latest docker-ce and docker-compose for the chosen distro (not the version from the distro). -To run ``docker-compose`` for PAVICS, the `pavics-compose.sh <./pavics-compose.sh>`_ wrapper script must be used. +To run ``docker-compose`` for PAVICS, the `pavics-compose.sh `_ (:download:`download `) wrapper script must be used. This script will source the ``env.local`` file, apply the appropriate variable substitutions on all the configuration files -".template", and run ``docker-compose`` with all the command line arguments given to `pavics-compose.sh <./pavics-compose.sh>`_. -See `env.local.example <./env.local.example>`_ for more details on what can go into the ``env.local`` file. +".template", and run ``docker-compose`` with all the command line arguments given to `pavics-compose.sh `_ (:download:`download `). +See `env.local.example `_ (:download:`download `) for more details on what can go into the ``env.local`` file. -If the file `env.local` is somewhere else, symlink it here, next to `docker-compose.yml `_ because many scripts assume this location. +If the file `env.local` is somewhere else, symlink it here, next to `docker-compose.yml `_ (:download:`download `) because many scripts assume this location. To follow infrastructure-as-code, it is encouraged to source control the above `env.local` file and any override needed to customized this PAVICS deployment -for your organization. For an example of possible override, see how the `emu service <./optional-components/emu/docker-compose-extra.yml>`_ -(`README <./optional-components/README.rst>`_) can be optionally added to the deployment via the `override mechanism `_. +for your organization. For an example of possible override, see how the `emu service `_ (:download:`download `) +(`README `_) can be optionally added to the deployment via the `override mechanism `_. Ouranos specific override can be found in this `birdhouse-deploy-ouranos `_ repo. Suggested deployment layout: @@ -49,14 +49,14 @@ Suggested deployment layout: The automatic deployment is able to handle multiple repos, so will trigger if this repo or your private-personalized-config repo changes, giving you automated continuous deployment. See the continuous deployment setup section -below and the variable ``AUTODEPLOY_EXTRA_REPOS`` in `env.local.example <./env.local.example>`_. +below and the variable ``AUTODEPLOY_EXTRA_REPOS`` in `env.local.example `_ (:download:`download `). The automatic deployment of the PAVICS platform, of the Jupyter tutorial notebooks and of the automatic deployment mechanism itself can all be -enabled by following the `scheduling instructions <./components/README.rst#scheduler>`_. +enabled by following the `scheduling instructions `_. Resource usage monitoring (CPU, memory, ..) and alerting for the host and each -of the containers can be enabled by following the `monitoring instructions <./components/README.rst#monitoring>`_. +of the containers can be enabled by following the `monitoring instructions `_. To launch all the containers, use the following command: @@ -65,7 +65,7 @@ To launch all the containers, use the following command: ./pavics-compose.sh up -d If you get a ``'No applicable error code, please check error log'`` error from the WPS processes, please make sure that the WPS databases exists in the -postgres instance. See `create-wps-pgsql-databases.sh <./scripts/create-wps-pgsql-databases.sh>`_. +postgres instance. See `create-wps-pgsql-databases.sh `_ (:download:`download `). Manual steps post deployment ---------------------------- @@ -96,7 +96,7 @@ Change geoserver default admin password Create public demo user in Magpie for JupyterHub login ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -Use `create-magpie-users <./scripts/create-magpie-users>`_ or follow manual +Use `create-magpie-users `_ (:download:`download `) or follow manual instructions below. ``config.yml`` file if using ``create-magpie-users``: @@ -137,7 +137,7 @@ https://github.com/Ouranosinc/PAVICS-e2e-workflow-tests with pre-configured Jenkins at https://github.com/Ouranosinc/jenkins-config. For that test suite to pass, run the script -`bootstrap-instance-for-testsuite <./scripts/bootstrap-instance-for-testsuite>`_ +`scripts/bootstrap-instance-for-testsuite `_ (:download:`download `) to prepare your new instance. Further documentation inside the script. Optional component @@ -162,10 +162,10 @@ environment for testing or to have multiple flavors of PAVICS with slightly different combinations of the parts all running simultaneously in their respective VM, allowing us to see the differences in behavior. -See `vagrant_variables.yml.example <./vagrant_variables.yml.example>`_ for what's +See `vagrant_variables.yml.example `_ (:download:`download `) for what's configurable with Vagrant. -If using Centos box, follow `disk-resize <./vagrant-utils/disk-resize>`_ after +If using Centos box, follow `disk-resize `_ (:download:`download `) after first ``vagrant up`` failure due to disk full. Then ``vagrant reload && vagrant provision`` to continue. If using Ubuntu box, no manual steps required, everything just works. diff --git a/birdhouse/optional-components/README.rst b/birdhouse/optional-components/README.rst index 13a7f0d4..62af8cf8 100644 --- a/birdhouse/optional-components/README.rst +++ b/birdhouse/optional-components/README.rst @@ -11,7 +11,7 @@ the end user). This assume all the WPS services are public. If not the case, make a copy of this config and adjust accordingly. -How to enable this config in ``env.local`` (a copy from `<../env.local.example>`_ or :download:`download it here `): +How to enable this config in ``env.local`` (a copy from `env.local.example <../env.local.example>`_ (:download:`download `)): * Add ``./optional-components/canarie-api-full-monitoring`` to ``EXTRA_CONF_DIRS``. @@ -26,14 +26,14 @@ alternative configuration of existing birds. No Postgres DB configured. If need Postgres DB, use generic_bird component instead. -How to enable Emu in ``env.local`` (a copy from `/birdhouse/env.local.example` or :download:`download it here `): +How to enable Emu in ``env.local`` (a copy from `env.local.example <../env.local.example>`_ (:download:`download `)): * Add ``./optional-components/emu`` to ``EXTRA_CONF_DIRS``. * Optionally set ``EMU_IMAGE``, ``EMU_PORT``, ``EMU_NAME``, ``EMU_INTERNAL_PORT``, ``EMU_WPS_OUTPUTS_VOL`` in ``env.local`` for further customizations. - Default values are in `emu/default.env `_. + Default values are in `optional-components/emu/default.env `_ (:download:`download `). Emu service will be available at ``http://PAVICS_FQDN:EMU_PORT/wps`` or ``https://PAVICS_FQDN_PUBLIC/TWITCHER_PROTECTED_PATH/EMU_NAME`` where @@ -49,7 +49,7 @@ service. A second THREDDs server for testing ----------------------------------- -How to enable in ``env.local`` (a copy from `/birdhouse/env.local.example` or :download:`download it here `.): +How to enable in ``env.local`` (a copy from `env.local.example <../env.local.example>`_ (:download:`download `)): * @@ -59,7 +59,7 @@ How to enable in ``env.local`` (a copy from `/birdhouse/env.local.example` or :d Optionally set ``TESTTHREDDS_IMAGE``\ , ``TESTTHREDDS_PORT``\ , ``TESTTHREDDS_CONTEXT_ROOT``\ , ``TESTTHREDDS_WARFILE_NAME``\ , ``TESTTHREDDS_INTERNAL_PORT``\ , ``TESTTHREDDS_NAME``\ , in ``env.local`` for further - customizations. Default values are in: `/birdhouse/optional-components/testthredds/default.env`. + customizations. Default values are in: `optional-components/testthredds/default.env `_ (:download:`download `). Test THREDDs service will be available at ``http://PAVICS_FQDN:TESTTHREDDS_PORT/TESTTHREDDS_CONTEXT_ROOT`` or @@ -90,7 +90,7 @@ A generic bird WPS service Can be used to quickly deploy any birds temporarily without changing code. Good to preview new birds or test alternative configuration of existing birds. -How to enable in ``env.local`` (a copy from ` ./env.local.example`_ or :download:`download it here `.): +How to enable in ``env.local`` (a copy from `env.local.example <../env.local.example>`_ (:download:`download `)): * @@ -100,7 +100,7 @@ How to enable in ``env.local`` (a copy from ` ./env.local.example`_ or :download Optionally set ``GENERIC_BIRD_IMAGE``, ``GENERIC_BIRD_PORT``, ``GENERIC_BIRD_NAME``, ``GENERIC_BIRD_INTERNAL_PORT``, and ``GENERIC_BIRD_POSTGRES_IMAGE`` in ``env.local`` for further customizations. - Default values are in `generic_bird/default.env `_. + Default values are in `optional-components/generic_bird/default.env `_ (:download:`download `). The WPS service will be available at ``http://PAVICS_FQDN:GENERIC_BIRD_PORT/wps`` or ``https://PAVICS_FQDN_PUBLIC/TWITCHER_PROTECTED_PATH/GENERIC_BIRD_NAME`` where @@ -128,9 +128,9 @@ Just disabling this component will not revert the change. This optional component is required for the test suite at https://github.com/Ouranosinc/PAVICS-e2e-workflow-tests. -How to enable in ``env.local`` (a copy from `<../env.local.example>`_ or :download:`download it here `.): +How to enable in ``env.local`` (a copy from `env.local.example <../env.local.example>`_ (:download:`download `)): * Add ``./optional-components/all-public-access`` to ``EXTRA_CONF_DIRS``. -The anonymous user will now have all the permissions described in `optional-components/all-public-access/all-public-access-magpie-permission.cfg `_. +The anonymous user will now have all the permissions described in `optional-components/all-public-access/all-public-access-magpie-permission.cfg `_ (:download:`download `). From 1dc90253348eb2edd9ebf17c4c04b4070b77b519 Mon Sep 17 00:00:00 2001 From: Trevor James Smith <10819524+Zeitsperre@users.noreply.github.com> Date: Tue, 26 Jan 2021 16:23:29 -0500 Subject: [PATCH 11/22] Add links to the components README.rst --- birdhouse/components/README.rst | 29 ++++++++++++++++------------- 1 file changed, 16 insertions(+), 13 deletions(-) diff --git a/birdhouse/components/README.rst b/birdhouse/components/README.rst index 2ac80a91..b248e2a8 100644 --- a/birdhouse/components/README.rst +++ b/birdhouse/components/README.rst @@ -55,21 +55,21 @@ Given the unattended nature, there is no UI. Logs are used to keep trace. How to Enable the Component --------------------------- -- Edit ``env.local`` (a copy of env.local.example_) +- Edit ``env.local`` (a copy of env.local.example_ (:download:`download <../env.local.example>`)) - Add "./components/scheduler" to ``EXTRA_CONF_DIRS``. - Set ``AUTODEPLOY_EXTRA_REPOS``, ``AUTODEPLOY_DEPLOY_KEY_ROOT_DIR``, ``AUTODEPLOY_PLATFORM_FREQUENCY``, ``AUTODEPLOY_NOTEBOOK_FREQUENCY`` as desired, full documentation in env.local.example_. - - Run once fix-write-perm_, see doc in script. + - Run once fix-write-perm_ (:download:`download <../deployment/fix-write-perm>`), see doc in script. Old way to deploy the automatic deployment ------------------------------------------ -Superseeded by this new ``scheduler`` component. Keeping for reference only. +Superseded by this new ``scheduler`` component. Keeping for reference only. -Doing it this old way do not need the ``scheduler`` compoment but lose the +Doing it this old way do not need the ``scheduler`` component but lose the ability for the autodeploy system to update itself. Configure logrotate for all following automations to prevent disk full:: @@ -82,7 +82,7 @@ To enable continuous deployment of PAVICS:: # read the script for more options/details If you want to manually force a deployment of PAVICS (note this might not use -latest version of deploy.sh script):: +latest version of deploy.sh script (:download:`download <../deployment/deploy.sh>`):: deployment/deploy.sh . # read the script for more options/details @@ -112,24 +112,27 @@ Comparison between the old and new autodeploy mechanism Maximum backward-compatibility has been kept with the old install scripts style: * Still log to the same existing log files under ``/var/log/PAVICS``. -* Old single ssh deploy key is still compatible, but the new mechanism allows for different ssh deploy keys for each extra repos (again, public repos should use https clone path to avoid dealing with ssh deploy keys in the first place). +* Old single ssh deploy key is still compatible, but the new mechanism allows for different ssh deploy keys for each extra + repos (again, public repos should use https clone path to avoid dealing with ssh deploy keys in the first place). * Old install scripts are kept and can still deploy the old way. Features missing in old install scripts or how the new mechanism improves on the old install scripts: -* Autodeploy of the autodeploy itself ! This is the biggest win. Previously, if ``triggerdeploy.sh`` or ``PAVICS-deploy-notebooks`` script changes, they have to be deployed manually. It's very annoying. Now they are volume-mount in so are fresh on each run. -* ``env.local`` now drive absolutely everything, source control that file and we've got a true DevOPS pipeline. +* Autodeploy of the autodeploy itself ! This is the biggest win. Previously, if ``triggerdeploy.sh`` (:download:`download <../deployment/triggerdeploy.sh>`) + or ``PAVICS-deploy-notebooks`` (TODO: @tlvu Where is this file?) script changes, they have to be deployed manually. It's very annoying. Now they are volume-mount in so are fresh on each run. +* ``env.local`` now drives absolutely everything, source control that file and we've got a true DevOPS pipeline. * Configurable platform and notebook autodeploy frequency. Previously, this means manually editing the generated cron file, less ideal. -* Do not need any support on the local host other than ``docker`` and ``docker-compose``. ``cron/logrotate/git/ssh`` versions are all locked-down in the docker images used by the autodeploy. Recall previously we had to deal with git version too old on some hosts. +* Do not need any support on the local host other than ``docker`` and ``docker-compose``. ``cron/logrotate/git/ssh`` + versions are all locked-down in the docker images used by the autodeploy. Recall previously we had to deal with git version too old on some hosts. * Each cron job run in its own docker image meaning the runtime environment is traceable and reproducible. -* The newly introduced scheduler component is made extensible so other jobs can added into it as well (ex: backup), via ``env.local``, which should be source controlled, meaning all surrounding maintenance related tasks can also be traceable and reproducible. +* The newly introduced scheduler component is made extensible so other jobs can added into it as well (ex: backup), via ``env.local``, + which should be source controlled, meaning all surrounding maintenance related tasks can also be traceable and reproducible. Monitoring ========== -This component provides monitoring and alerting for the PAVICS physical host -and containers. +This component provides monitoring and alerting for the PAVICS physical host and containers. Prometheus stack is used: @@ -151,7 +154,7 @@ Usage How to Enable the Component --------------------------- -- Edit ``env.local`` (a copy of env.local.example_) +- Edit ``env.local`` (a copy of env.local.example_ (:download:`download <../env.local.example>`)) - Add "./components/monitoring" to ``EXTRA_CONF_DIRS`` - Set ``GRAFANA_ADMIN_PASSWORD`` to login to Grafana From 62e5ccaa367c523bbaeb2c0e721659241cdfcfd4 Mon Sep 17 00:00:00 2001 From: Long Vu Date: Tue, 26 Jan 2021 18:00:22 -0500 Subject: [PATCH 12/22] components/README: remove TODO note --- birdhouse/components/README.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/birdhouse/components/README.rst b/birdhouse/components/README.rst index b248e2a8..a501b4c1 100644 --- a/birdhouse/components/README.rst +++ b/birdhouse/components/README.rst @@ -119,7 +119,7 @@ Maximum backward-compatibility has been kept with the old install scripts style: Features missing in old install scripts or how the new mechanism improves on the old install scripts: * Autodeploy of the autodeploy itself ! This is the biggest win. Previously, if ``triggerdeploy.sh`` (:download:`download <../deployment/triggerdeploy.sh>`) - or ``PAVICS-deploy-notebooks`` (TODO: @tlvu Where is this file?) script changes, they have to be deployed manually. It's very annoying. Now they are volume-mount in so are fresh on each run. + or the deployed ``/etc/cron.hourly/PAVICS-deploy-notebooks`` script changes, they have to be deployed manually. It's very annoying. Now they are volume-mount in so are fresh on each run. * ``env.local`` now drives absolutely everything, source control that file and we've got a true DevOPS pipeline. * Configurable platform and notebook autodeploy frequency. Previously, this means manually editing the generated cron file, less ideal. * Do not need any support on the local host other than ``docker`` and ``docker-compose``. ``cron/logrotate/git/ssh`` From cbdb44216bd2e2565667e280e287c97e8e76eca2 Mon Sep 17 00:00:00 2001 From: Long Vu Date: Tue, 26 Jan 2021 18:02:47 -0500 Subject: [PATCH 13/22] optional-components/README: add automated TOC generation --- birdhouse/optional-components/README.rst | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/birdhouse/optional-components/README.rst b/birdhouse/optional-components/README.rst index 62af8cf8..0f730ecd 100644 --- a/birdhouse/optional-components/README.rst +++ b/birdhouse/optional-components/README.rst @@ -1,6 +1,10 @@ Optional components =================== + +.. contents:: + + Monitor all components in CANARIE node, both public and internal url -------------------------------------------------------------------- From 44e05c6845184f8897b03c80e42b156bc06fa1cc Mon Sep 17 00:00:00 2001 From: Long Vu Date: Tue, 26 Jan 2021 18:13:38 -0500 Subject: [PATCH 14/22] components/README: add more links to mentionned scripts in this repo --- birdhouse/components/README.rst | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/birdhouse/components/README.rst b/birdhouse/components/README.rst index a501b4c1..34331aff 100644 --- a/birdhouse/components/README.rst +++ b/birdhouse/components/README.rst @@ -82,7 +82,7 @@ To enable continuous deployment of PAVICS:: # read the script for more options/details If you want to manually force a deployment of PAVICS (note this might not use -latest version of deploy.sh script (:download:`download <../deployment/deploy.sh>`):: +latest version of deploy.sh_ script (:download:`download <../deployment/deploy.sh>`):: deployment/deploy.sh . # read the script for more options/details @@ -118,7 +118,7 @@ Maximum backward-compatibility has been kept with the old install scripts style: Features missing in old install scripts or how the new mechanism improves on the old install scripts: -* Autodeploy of the autodeploy itself ! This is the biggest win. Previously, if ``triggerdeploy.sh`` (:download:`download <../deployment/triggerdeploy.sh>`) +* Autodeploy of the autodeploy itself ! This is the biggest win. Previously, if triggerdeploy.sh_ (:download:`download <../deployment/triggerdeploy.sh>`) or the deployed ``/etc/cron.hourly/PAVICS-deploy-notebooks`` script changes, they have to be deployed manually. It's very annoying. Now they are volume-mount in so are fresh on each run. * ``env.local`` now drives absolutely everything, source control that file and we've got a true DevOPS pipeline. * Configurable platform and notebook autodeploy frequency. Previously, this means manually editing the generated cron file, less ideal. @@ -233,3 +233,4 @@ Customizing the Component .. _env.local.example: ../env.local.example .. _fix-write-perm: ../deployment/fix-write-perm .. _deploy.sh: ../deployment/deploy.sh +.. _triggerdeploy.sh: ../deployment/triggerdeploy.sh From fe68cfe22819b5e8c97df77e7d3f8b00d6ed2a38 Mon Sep 17 00:00:00 2001 From: Long Vu Date: Tue, 26 Jan 2021 18:21:18 -0500 Subject: [PATCH 15/22] optional-components/README: avoid dupe definition of env.local.example link dest and remove some useless empty lines --- birdhouse/optional-components/README.rst | 35 ++++++++++++------------ 1 file changed, 17 insertions(+), 18 deletions(-) diff --git a/birdhouse/optional-components/README.rst b/birdhouse/optional-components/README.rst index 0f730ecd..b4b06c8f 100644 --- a/birdhouse/optional-components/README.rst +++ b/birdhouse/optional-components/README.rst @@ -15,11 +15,11 @@ the end user). This assume all the WPS services are public. If not the case, make a copy of this config and adjust accordingly. -How to enable this config in ``env.local`` (a copy from `env.local.example <../env.local.example>`_ (:download:`download `)): - +How to enable this config in ``env.local`` (a copy from env.local.example_ (:download:`download `)): * Add ``./optional-components/canarie-api-full-monitoring`` to ``EXTRA_CONF_DIRS``. + Emu WPS service for testing --------------------------- @@ -30,8 +30,7 @@ alternative configuration of existing birds. No Postgres DB configured. If need Postgres DB, use generic_bird component instead. -How to enable Emu in ``env.local`` (a copy from `env.local.example <../env.local.example>`_ (:download:`download `)): - +How to enable Emu in ``env.local`` (a copy from env.local.example_ (:download:`download `)): * Add ``./optional-components/emu`` to ``EXTRA_CONF_DIRS``. * Optionally set ``EMU_IMAGE``, ``EMU_PORT``, @@ -50,17 +49,15 @@ access for this Emu WPS service. CANARIE monitoring will also be automatically configured for this Emu WPS service. + A second THREDDs server for testing ----------------------------------- -How to enable in ``env.local`` (a copy from `env.local.example <../env.local.example>`_ (:download:`download `)): - +How to enable in ``env.local`` (a copy from env.local.example_ (:download:`download `)): -* - Add ``./optional-components/testthredds`` to ``EXTRA_CONF_DIRS``. +* Add ``./optional-components/testthredds`` to ``EXTRA_CONF_DIRS``. -* - Optionally set ``TESTTHREDDS_IMAGE``\ , ``TESTTHREDDS_PORT``\ , +* Optionally set ``TESTTHREDDS_IMAGE``\ , ``TESTTHREDDS_PORT``\ , ``TESTTHREDDS_CONTEXT_ROOT``\ , ``TESTTHREDDS_WARFILE_NAME``\ , ``TESTTHREDDS_INTERNAL_PORT``\ , ``TESTTHREDDS_NAME``\ , in ``env.local`` for further customizations. Default values are in: `optional-components/testthredds/default.env `_ (:download:`download `). @@ -88,20 +85,18 @@ Nginx proxy. CANARIE monitoring will also be automatically configured for this second THREDDs server. + A generic bird WPS service -------------------------- Can be used to quickly deploy any birds temporarily without changing code. Good to preview new birds or test alternative configuration of existing birds. -How to enable in ``env.local`` (a copy from `env.local.example <../env.local.example>`_ (:download:`download `)): +How to enable in ``env.local`` (a copy from env.local.example_ (:download:`download `)): +* Add ``./optional-components/generic_bird`` to ``EXTRA_CONF_DIRS``. -* - Add ``./optional-components/generic_bird`` to ``EXTRA_CONF_DIRS``. - -* - Optionally set ``GENERIC_BIRD_IMAGE``, ``GENERIC_BIRD_PORT``, +* Optionally set ``GENERIC_BIRD_IMAGE``, ``GENERIC_BIRD_PORT``, ``GENERIC_BIRD_NAME``, ``GENERIC_BIRD_INTERNAL_PORT``, and ``GENERIC_BIRD_POSTGRES_IMAGE`` in ``env.local`` for further customizations. Default values are in `optional-components/generic_bird/default.env `_ (:download:`download `). @@ -122,6 +117,7 @@ access for this WPS service. CANARIE monitoring will also be automatically configured for this WPS service. + Give public access to all resources for testing purposes -------------------------------------------------------- @@ -132,9 +128,12 @@ Just disabling this component will not revert the change. This optional component is required for the test suite at https://github.com/Ouranosinc/PAVICS-e2e-workflow-tests. -How to enable in ``env.local`` (a copy from `env.local.example <../env.local.example>`_ (:download:`download `)): - +How to enable in ``env.local`` (a copy from env.local.example_ (:download:`download `)): * Add ``./optional-components/all-public-access`` to ``EXTRA_CONF_DIRS``. The anonymous user will now have all the permissions described in `optional-components/all-public-access/all-public-access-magpie-permission.cfg `_ (:download:`download `). + + + +.. _env.local.example: ../env.local.example From cc61639016e7ef121c5947e957c0b9fcc127c112 Mon Sep 17 00:00:00 2001 From: Long Vu Date: Tue, 26 Jan 2021 18:28:31 -0500 Subject: [PATCH 16/22] docs: do not reference geoserver docker build instructions We want to enventually drop our custom build of geoserver or if we would keep it, it should be moved out to a separate repo. --- docs/source/index.rst | 1 - 1 file changed, 1 deletion(-) diff --git a/docs/source/index.rst b/docs/source/index.rst index 080c56b8..f0a53450 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -20,7 +20,6 @@ Contents: installation deployment - birdhouse/docker/geoserver/README birdhouse/components/README birdhouse/optional-components/README data_catalog From 9650df7188645641af6a7cfeaba8d0143d28e949 Mon Sep 17 00:00:00 2001 From: Long Vu Date: Tue, 26 Jan 2021 18:47:02 -0500 Subject: [PATCH 17/22] docs: comment out index and module index since they do not work on RtD --- docs/source/index.rst | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/docs/source/index.rst b/docs/source/index.rst index f0a53450..7e29bc50 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -28,7 +28,9 @@ Contents: Indices and tables ================== -* :ref:`genindex` -* :ref:`modindex` +.. + * :ref:`genindex` + * :ref:`modindex` + * :ref:`search` From f4cac342157e0c0737c151ac5591900a2e9d7eed Mon Sep 17 00:00:00 2001 From: Long Vu Date: Tue, 26 Jan 2021 18:59:25 -0500 Subject: [PATCH 18/22] docs: add script to locally test build on RtD --- docs/source/.gitignore | 1 + docs/source/RtD-build | 6 ++++++ 2 files changed, 7 insertions(+) create mode 100644 docs/source/.gitignore create mode 100755 docs/source/RtD-build diff --git a/docs/source/.gitignore b/docs/source/.gitignore new file mode 100644 index 00000000..e35d8850 --- /dev/null +++ b/docs/source/.gitignore @@ -0,0 +1 @@ +_build diff --git a/docs/source/RtD-build b/docs/source/RtD-build new file mode 100755 index 00000000..946cda05 --- /dev/null +++ b/docs/source/RtD-build @@ -0,0 +1,6 @@ +#!/bin/sh -x + +# conda activate < some env that has sphinx installed, like finch-docs for example > + +# This line copied from RtD build log. +python -m sphinx -T -E -b html -d _build/doctrees -D language=en . _build/html From 9520ace426fb122a9fa548972cb1f5021f5d4cee Mon Sep 17 00:00:00 2001 From: Long Vu Date: Tue, 26 Jan 2021 19:10:25 -0500 Subject: [PATCH 19/22] docs: delete empty installation.rst file --- docs/source/index.rst | 1 - docs/source/installation.rst | 2 -- 2 files changed, 3 deletions(-) delete mode 100644 docs/source/installation.rst diff --git a/docs/source/index.rst b/docs/source/index.rst index 7e29bc50..1e9b0497 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -18,7 +18,6 @@ Contents: .. toctree:: :maxdepth: 2 - installation deployment birdhouse/components/README birdhouse/optional-components/README diff --git a/docs/source/installation.rst b/docs/source/installation.rst deleted file mode 100644 index 11e44375..00000000 --- a/docs/source/installation.rst +++ /dev/null @@ -1,2 +0,0 @@ -Installation -============ From 222e84dcda9ae8d959e0c0d68d62c9ffb0fb6a68 Mon Sep 17 00:00:00 2001 From: Long Vu Date: Tue, 26 Jan 2021 19:14:55 -0500 Subject: [PATCH 20/22] RtD-build: delete previous build output to ensure fresh build --- docs/source/RtD-build | 3 +++ 1 file changed, 3 insertions(+) diff --git a/docs/source/RtD-build b/docs/source/RtD-build index 946cda05..62c80878 100755 --- a/docs/source/RtD-build +++ b/docs/source/RtD-build @@ -2,5 +2,8 @@ # conda activate < some env that has sphinx installed, like finch-docs for example > +# Delete output from previous build. +rm -rf _build + # This line copied from RtD build log. python -m sphinx -T -E -b html -d _build/doctrees -D language=en . _build/html From db4d5aab1bf24a8d2cd9972b2e0a638d35533e5f Mon Sep 17 00:00:00 2001 From: Long Vu Date: Tue, 26 Jan 2021 19:44:17 -0500 Subject: [PATCH 21/22] docs: make relative links for files at root of RtD work --- docs/source/conf.py | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/docs/source/conf.py b/docs/source/conf.py index f4662f6d..3cae4f15 100644 --- a/docs/source/conf.py +++ b/docs/source/conf.py @@ -171,6 +171,12 @@ # directly to the root of the documentation. # # html_extra_path = [] +html_extra_path = [ + 'birdhouse/README.rst', + 'birdhouse/env.local.example', + 'birdhouse/pavics-compose.sh', + 'birdhouse/docker-compose.yml', +] # If not None, a 'Last updated on:' timestamp is inserted at every page # bottom, using the given strftime format. From 7f5e0ef3c624d96eda2db677c6c9625b43f124e2 Mon Sep 17 00:00:00 2001 From: Long Vu Date: Tue, 26 Jan 2021 19:49:42 -0500 Subject: [PATCH 22/22] components/README: add missing deploy.sh download link --- birdhouse/components/README.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/birdhouse/components/README.rst b/birdhouse/components/README.rst index 34331aff..49b51e45 100644 --- a/birdhouse/components/README.rst +++ b/birdhouse/components/README.rst @@ -36,7 +36,7 @@ private repo containing the source controlled ``env.local`` file and any other docker-compose override for true infrastructure-as-code. Note: there are still cases where a human intervention is needed. See note in -script deploy.sh_. +script deploy.sh_ (:download:`download <../deployment/deploy.sh>`). Usage