Magpie thredds

birdhouse/README.md

@@ -14,19 +14,20 @@ Requirements:
   version from the distro).
 
 To run `docker-compose` for PAVICS, the [`pavics-compose.sh`](pavics-compose.sh) wrapper script must be used.
-This script will source the `env.local` file, apply the appropriate variable substitutions on all the configuration files ".template", and run `docker-compose` with all the command line arguments given to `pavics-compose.sh`. See [`env.local.example`](env.local.example) for more details on what can go into the `env.local` file.
+This script will source the `env.local` file, apply the appropriate variable substitutions on all the configuration
+files ".template", and run `docker-compose` with all the command line arguments given to `pavics-compose.sh`.
+See [`env.local.example`](env.local.example) for more details on what can go into the `env.local` file.
 
 If the file `env.local` is somewhere else, symlink it here, next to
 `docker-compose.yml` because many scripts assume this location.
 
 To follow infrastructure-as-code, it is encouraged to source control the above
 `env.local` file and any override needed to customized this PAVICS deployment
-for your organization.  For an example of possible override, see how the [emu
-service](optional-components/emu/docker-compose-extra.yml)
+for your organization.  For an example of possible override, see how the
+[emu service](optional-components/emu/docker-compose-extra.yml)
 ([README](optional-components/README.md)) can be optionally added to the
-deployment via the [override
-mechanism](https://docs.docker.com/compose/extends/).  Ouranos specific
-override can be found in this
+deployment via the [override mechanism](https://docs.docker.com/compose/extends/).
+Ouranos specific override can be found in this
 [birdhouse-deploy-ouranos](https://github.com/bird-house/birdhouse-deploy-ouranos)
 repo.
 
@@ -62,16 +63,18 @@ To launch all the containers, use the following command:
 ./pavics-compose.sh up -d
 ```
 
-If you get a `'No applicable error code, please check error log'` error from the WPS processes, please make sure that the WPS databases exists in the
-postgres instance. See [`scripts/create-wps-pgsql-databases.sh`](scripts/create-wps-pgsql-databases.sh).
+If you get a `'No applicable error code, please check error log'` error from the WPS processes,
+please make sure that the WPS databases exists in the postgres instance.
+See [`scripts/create-wps-pgsql-databases.sh`](scripts/create-wps-pgsql-databases.sh).
 
 
 ## Manual steps post deployment
 
 ### Change geoserver default admin password
 
 * Go to
-  https://<PAVICS_FQDN>/geoserver/web/wicket/bookmarkable/org.geoserver.security.web.UserGroupRoleServicesPage (Security -> Users, Groups, and Roles)
+  https://<PAVICS_FQDN>/geoserver/web/wicket/bookmarkable/org.geoserver.security.web.UserGroupRoleServicesPage
+  (Security -> Users, Groups, and Roles)
 
 * Login using the default username `admin` and default password `geoserver`.
 
@@ -129,9 +132,13 @@ Optional component
 [all-public-access](optional-components#give-public-access-to-all-resources-for-testing-purposes)
 also need to be enabled in `env.local`.
 
+To test secured access to THREDDS files and directories, optional component
+[secure-thredds](optional-components#control-secured-access-to-resources-example)
+also need to be enabled in `env.local`.
+
 ESGF login is also needed for
 https://github.com/Ouranosinc/pavics-sdi/blob/master/docs/source/notebooks/esgf-dap.ipynb
-part of test suite.  ESGF credentails can be given to Jenkins via
+part of test suite.  ESGF credentials can be given to Jenkins via
 https://github.com/Ouranosinc/jenkins-config/blob/aafaf6c33ea60faede2a32850604c07c901189e8/env.local.example#L11-L13
 
 The canarie monitoring link
@@ -216,7 +223,7 @@ Given a version number MAJOR.MINOR.PATCH, increment the:
 
   1. MINOR version when we add new components or update existing components
      that also require change to other existing components (ex: new Magpie that
-     also force Twitcher and/or Frondend update) or the change to the existing
+     also force Twitcher and/or Frontend update) or the change to the existing
      component is a major one (ex: major refactoring of Twitcher, big merge
      with corresponding upstream component from birdhouse project).
 

birdhouse/components/README.rst

@@ -68,9 +68,9 @@ How to Enable the Component
 Old way to deploy the automatic deployment
 ------------------------------------------
 
-Superseeded by this new ``scheduler`` component.  Keeping for reference only.
+Superseded by this new ``scheduler`` component.  Keeping for reference only.
 
-Doing it this old way do not need the ``scheduler`` compoment but lose the
+Doing it this old way do not need the ``scheduler`` component but lose the
 ability for the autodeploy system to update itself.
 
 Configure logrotate for all following automations to prevent disk full::

birdhouse/config/magpie/providers.cfg.template

@@ -62,6 +62,31 @@ providers:
     c4i: false
     type: thredds
     sync_type: thredds
+    # below is a custom config to indicate how magpie should convert thredds path elements into resources/permissions
+    # see: https://pavics-magpie.readthedocs.io/en/latest/services.html#servicethredds
+    configuration:
+      skip_prefix: "thredds"  # prefix to ignore, below prefixes will be matched against whatever comes after in path
+      file_patterns:
+        # note: make sure to employ quotes and double escapes to avoid parsing YAML error
+        - ".+\\.nc"
+      metadata_type:
+        prefixes:
+          - null  # note: special YAML value evaluated as `no-prefix`, use quotes if literal value is needed
+          - "\\w+\\.gif"  # threddsIcon, folder icon, etc.
+          - "\\w+\\.ico"  # favicon
+          - "\\w+\\.txt"  # licence
+          - "\\w+\\.css"  # tds.css
+          - "catalog\\.\\w+"  # note: special case for `THREDDS` top-level directory (root) accessed for `BROWSE`
+          - catalog
+          - ncml
+          - uddc
+          - iso
+      data_type:
+        prefixes:
+          - fileServer
+          - dodsC
+          - wcs
+          - wms
 
   ncWMS2:
     url: http://${PAVICS_FQDN}:8080/ncWMS2

birdhouse/docker-compose.yml

@@ -281,7 +281,7 @@ services:
     restart: always
 
   magpie:
-    image: pavics/magpie:1.7.3
+    image: pavics/magpie:3.3.0
     container_name: magpie
     ports:
       - "2001:2001"
@@ -307,7 +307,7 @@ services:
     restart: always
 
   twitcher:
-    image: pavics/twitcher:magpie-1.7.3
+    image: pavics/twitcher:magpie-3.3.0
     container_name: twitcher
     ports:
       - "8000:8000"

birdhouse/env.local.example

@@ -69,14 +69,17 @@ export POSTGRES_MAGPIE_PASSWORD=postgres-qwerty
 # Format: space separated list of dirs
 #
 #export EXTRA_CONF_DIRS="/path/to/dir1 ./path/to/dir2 dir3 dir4"
-#export EXTRA_CONF_DIRS="./optional-components/canarie-api-full-monitoring
+#export EXTRA_CONF_DIRS="
+#    ./optional-components/canarie-api-full-monitoring
 #    ./optional-components/emu
 #    ./optional-components/testthredds
 #    ./optional-components/generic_bird
 #    ./optional-components/all-public-access
+#    ./optional-components/secure-thredds
 #    ./components/scheduler
 #    ./components/monitoring
-#    /path/to/private-config-repo"
+#    /path/to/private-config-repo
+#"
 
 # Extra repos, than the current repo, the autodeploy should keep up-to-date.
 # Any changes to these extra repos will also trigger autodeploy.

birdhouse/optional-components/README.md

@@ -1,6 +1,6 @@
 # Optional components
 
-## Monitor all components in Canarie node, both public and internal url
+## Monitor all components in Canarie node, both public and internal URL
 
 So that the url https://<PAVICS_FQDN>/canarie/node/service/stats also return
 what the end user really see (a component might work but is not accessible to
@@ -116,9 +116,13 @@ Canarie monitoring will also be automatically configured for this WPS service.
 
 ## Give public access to all resources for testing purposes
 
-By enabling this component, all WPS services and data on Thredds are completely public, please beware. 
-Once enabled, if you need to revert the change, you have to do it manually by logging into Magpie. 
-Just disabling this component will not revert the change.
+By enabling this component, all WPS services and data on Thredds are completely public, please beware.
+
+Once enabled, if you need to revert the change, you have to do it manually by logging into Magpie.
+Just disabling this component will not revert the change. Alternatively, you can create an identical file to
+[`./optional-components/all-public-access/all-public-access-magpie-permission.cfg`](all-public-access/all-public-access-magpie-permission.cfg).
+and replace all desired `action: create` entries into `action: remove` to make sure the permissions are removed as
+startup if they exist.
 
 This optional component is required for the test suite at
 https://github.com/Ouranosinc/PAVICS-e2e-workflow-tests.
@@ -128,4 +132,28 @@ How to enable in `env.local` (a copy from
 
 * Add `./optional-components/all-public-access` to `EXTRA_CONF_DIRS`.
 
-The anonymous user will now have all the permissions described in [`./optional-components/all-public-access/all-public-access-magpie-permission.cfg`](all-public-access/all-public-access-magpie-permission.cfg).
+The anonymous user will now have all the permissions described in
+[`./optional-components/all-public-access/all-public-access-magpie-permission.cfg`](all-public-access/all-public-access-magpie-permission.cfg).
+
+
+## Control secured access to resources example
+
+Optional configuration
+[`./optional-components/secure-thredds/secure-access-magpie-permission.cfg`](secure-thredds/secure-access-magpie-permission.cfg)
+is provided as example to illustrate how to apply permissions on specific THREDDS resources to limit their access publicly.
+This permission configuration can be combined with others, such as
+[all-public-access](#Give-public-access-to-all-resources-for-testing-purposes)
+ones to formulate specific permissions schemes that matches your data structure and desired access rules.
+
+How to enable in `env.local` (a copy from
+[`env.local.example`](../env.local.example)):
+
+* Add `./optional-components/secure-thredds` to `EXTRA_CONF_DIRS`.
+
+The anonymous user will *not* have access anymore to THREDDS test directory `birdhouse/testdata/secure` and any other
+directories and files under it. Directories above that one will still be accessible if
+[all-public-access](#Give-public-access-to-all-resources-for-testing-purposes) component was also employed.
+
+On a typical server, custom and private permission rules should be provided in a similar fashion to ensure that
+each time a new instance is booted, the same scheme of access configuration is applied. Permissions applied manually
+into Magpie will not be replicated onto other server instance.

birdhouse/optional-components/all-public-access/all-public-access-magpie-permission.cfg

@@ -99,3 +99,8 @@ permissions:
     permission: read
     group: anonymous
     action: create
+
+  - service: thredds
+    permission: browse
+    group: anonymous
+    action: create

birdhouse/optional-components/secure-thredds/docker-compose-extra.yml

@@ -0,0 +1,5 @@
+version: '2.1'
+services:
+  magpie:
+    volumes:
+    - ./optional-components/secure-thredds/secure-access-magpie-permission.cfg:/opt/local/src/magpie/config/permissions/secure-access-magpie-permission.cfg:ro

birdhouse/optional-components/secure-thredds/secure-access-magpie-permission.cfg

@@ -0,0 +1,68 @@
+permissions:
+  # note:
+  #   following permissions can be combined with others such as 'optional-components/all-public-access'
+  #   to provide access to 'everything' except those under 'secure' directories listed below
+
+  # following permissions only enforce security on specific directories and files under it
+  # these can be reverted or combined with other set of permissions on resources 'above' or 'under' in the hierarchy
+  # users or groups will need explicit permissions under following resources for them to access sub-directories/files
+  - service: thredds
+    resource: /birdhouse/testdata/secure
+    type: directory
+    permission:
+      name: browse
+      access: deny
+      scope: recursive
+    group: anonymous
+    action: create
+
+  - service: thredds
+    resource: /birdhouse/testdata/secure
+    type: directory
+    permission:
+      name: read
+      access: deny
+      scope: recursive
+    group: anonymous
+    action: create
+
+  - service: thredds
+    resource: /birdhouse/testdata/secure
+    type: directory
+    permission:
+      name: write
+      access: deny
+      scope: recursive
+    group: anonymous
+    action: create
+
+  # preserve access for test-suite user
+  - service: thredds
+    resource: /birdhouse/testdata/secure
+    type: directory
+    permission:
+      name: browse
+      access: allow
+      scope: recursive
+    user: authtest
+    action: create
+
+  - service: thredds
+    resource: /birdhouse/testdata/secure
+    type: directory
+    permission:
+      name: read
+      access: allow
+      scope: recursive
+    user: authtest
+    action: create
+
+  - service: thredds
+    resource: /birdhouse/testdata/secure
+    type: directory
+    permission:
+      name: write
+      access: allow
+      scope: recursive
+    user: authtest
+    action: create

birdhouse/scripts/create-magpie-users

@@ -98,7 +98,7 @@ if [ -z "$MAGPIE_CLI_CONF" ]; then
 fi
 
 if [ -z "$MAGPIE_CLI_IMAGE" ]; then
-    MAGPIE_CLI_IMAGE="pavics/magpie:2.0.1"
+    MAGPIE_CLI_IMAGE="pavics/magpie:3.2.0"
 fi
 
 # End configurable config via env var.