add docs relative to thredds-specific config for #361, relates to #353

CHANGES.rst

@@ -7,7 +7,10 @@ Changes
 `Unreleased <https://github.com/Ouranosinc/Magpie/tree/master>`_ (latest)
 ------------------------------------------------------------------------------------
 
-* Nothing yet.
+Features / Changes
+~~~~~~~~~~~~~~~~~~~~~
+* Add ``BROWSE`` permission for ``ServiceTHREDDS`` to parse request against *metadata* or *data* contents according to
+  specified configuration of the specific service (resolves `#361 <https://github.com/Ouranosinc/Magpie/issues/361>`_).
 
 `3.0.0 <https://github.com/Ouranosinc/Magpie/tree/3.0.0>`_ (2020-10-19)
 ------------------------------------------------------------------------------------
@@ -513,7 +516,7 @@ Bug Fixes
 
 Features / Changes
 ~~~~~~~~~~~~~~~~~~~~~
-* Prioritize settings (ie: `magpie.ini` values) before environment variables and ``magpie.constants`` globals.
+* Prioritize settings (ie: ``magpie.ini`` values) before environment variables and ``magpie.constants`` globals.
 * Allow specifying ``magpie.scheme`` setting to generate the ``magpie.url`` with it if the later was omitted.
 * Look in settings for required parameters for function ``get_admin_cookies``.
 * Use API definitions instead of literal strings for routes employed in ``MagpieAdapter``.

docs/configuration.rst

@@ -5,14 +5,22 @@ Configuration
 =============
 
 At startup, `Magpie` application will load multiple configuration files to define various behaviours or setup
-operations. These are defined though the configuration settings presented below.
+operations. These are defined though the configuration settings presented in below sections.
 
-All generic `Magpie` configuration settings can be defined through either the `magpie.ini`_ file
-or environment variables. Values defined in `magpie.ini`_ are expected to follow the
-``magpie.[variable_name]`` format, and corresponding ``MAGPIE_[VARIABLE_NAME]`` format is used for environment
-variables. Both of these alternatives match the constants defined in `constants.py`_ and can be used
-interchangeably. Order of resolution will prioritize setting values over environment variables in case of duplicate
-configurations resulting into different values.
+All generic `Magpie` configuration settings can be defined through either the `magpie.ini`_ file or environment
+variables. Values defined in `magpie.ini`_ are expected to follow the ``magpie.[variable_name]`` format, and
+corresponding ``MAGPIE_[VARIABLE_NAME]`` format is used for environment variables. Both of these alternatives match
+the constants defined in `constants.py`_ and can be used interchangeably.
+
+.. versionchanged:: 1.1
+    Order of resolution will prioritize *setting configurations* over *environment variables* in case of duplicates
+    resulting into different values. Environment variables will not override already specified setting values.
+
+    Previous versions of `Magpie` would instead prioritize environment variables, but this behaviour was deemed as
+    counter intuitive. This is attributed to the global scope nature of environment variables that often made it hard
+    to understand why some custom INI file would not behave as intended since those variable would inconsistently take
+    precedence whether or not they were defined. Using a common configuration file makes it easier to maintain and
+    understand the applied settings, and is therefore preferable.
 
 .. _constants.py: https://github.com/Ouranosinc/Magpie/tree/master/magpie/constants.py
 
@@ -35,15 +43,14 @@ By default, `Magpie` will try to load a ``magpie.env`` file which can define fur
 used to setup the application (see ``MAGPIE_ENV_FILE`` setting further below). An example of expected format and common
 variables for this file is presented in `magpie.env.example`_.
 
-**Important Notes:**
-
-If ``magpie.env`` cannot be found (using setting ``MAGPIE_ENV_FILE``) but ``magpie.env.example`` is available
-(after resolving any previously set ``MAGPIE_ENV_DIR`` variable), this example file will be used to make a copy
-saved as ``magpie.env`` and will be used as the base ``.env`` file to load its contained environment variables.
-This behaviour is intended to reduce initial configuration and preparation of  `Magpie` for a new user.
+.. warning::
+    If ``magpie.env`` cannot be found (e.g.: using setting ``MAGPIE_ENV_FILE``) but `magpie.env.example`_ is available
+    after resolving any previously set ``MAGPIE_ENV_DIR`` variable, this example file will be used to make a copy saved
+    as ``magpie.env`` and will be used as the base ``.env`` file to load its contained environment variables.
+    This behaviour is intended to reduce initial configuration and preparation of  `Magpie` for a new user.
 
-When loading variables from the ``.env`` file, any conflicting environment variable will **NOT** be overridden.
-Therefore, only *missing but required* values will be added to the environment to ensure proper setup of `Magpie`.
+    When loading variables from the ``.env`` file, any conflicting environment variable will **NOT** be overridden.
+    Therefore, only *missing but required* values will be added to the environment to ensure proper setup of `Magpie`.
 
 .. _magpie.env.example: https://github.com/Ouranosinc/Magpie/tree/master/env/magpie.env.example
 
@@ -67,6 +74,10 @@ to the desired URL. See ``MAGPIE_PROVIDERS_CONFIG_PATH`` setting below to setup
 configuration. Please refer to the comment header of sample file `providers.cfg`_ for specific format and parameter
 details.
 
+.. versionchanged:: 3.1
+    Some services, such as :ref:`ServiceTHREDDS` for instance, can take additional parameters to customize some of
+    their behaviour. Please refer to :ref:`Services` chapter for specific configuration supported.
+
 File: permissions.cfg
 ~~~~~~~~~~~~~~~~~~~~~~
 
@@ -84,9 +95,11 @@ behaviour of each parameter according to encountered use cases.
 Combined Configuration File
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
+.. versionadded:: 2.0
+
 Since contents of all different configurations files (`providers.cfg`_, `permissions.cfg`_) reside under distinct
-top-level objects (of same name), it is actually possible to use an unique file to define everything. For example,
-one could define a combined configuration as follows.
+top-level objects, it is actually possible to use an unique file to define everything. For example, one could define
+a combined configuration as follows.
 
 .. code-block:: YAML
 
@@ -141,7 +154,7 @@ Settings and Constants
 
 Environment variables can be used to define all following configurations (unless mentioned otherwise with
 ``[constant]`` keyword next to the parameter name). Most values are parsed as plain strings, unless they refer to an
-activable setting (e.g.: ``True`` or ``False``), or when specified with more specific ``[<type>]`` notation.
+activatable setting (e.g.: ``True`` or ``False``), or when specified with more specific ``[<type>]`` notation.
 
 Configuration variables will be used by `Magpie` on startup unless prior definition is found within `magpie.ini`_.
 All variables (i.e.: non-``[constant]`` parameters) can also be specified by their ``magpie.[variable_name]`` setting

docs/references.rst

@@ -13,6 +13,7 @@
 .. _Pyramid: https://docs.pylonsproject.org/projects/pyramid/
 .. _ReadTheDocs: https://pavics-magpie.readthedocs.io/
 .. _SQLAlchemy: https://www.sqlalchemy.org/
+.. _THREDDS: https://www.unidata.ucar.edu/software/tds/
 .. _Twitcher: https://github.com/bird-house/twitcher
 .. _Waitress: https://github.com/Pylons/waitress
 .. _Ziggurat-Foundations: https://github.com/ergo/ziggurat_foundations

docs/services.rst

@@ -49,7 +49,99 @@ ServiceAPI
 ServiceTHREDDS
 ~~~~~~~~~~~~~~~~~~~~~
 
-.. todo:
+The implementation of this service is handled by class :class:`magpie.services.ServiceTHREDDS`. It refers to a remote
+data server named `Thematic Real-time Environmental Distributed Data Services` (`THREDDS`_). The service employs two (2)
+types of :term:`Resources`, namely :class:`magpie.models.Directory` and :class:`magpie.models.File`. All the directory
+resources can be nested any number of times, and files can only reside as leaves of the hierarchy, similarly to a
+traditional file system. The :term:`Allowed Permissions` on both the :term:`Service` itself or any of its children
+:term:`Resource` are :attr:`Permission.BROWSE`, :attr:`Permission.READ`, and :attr:`Permission.WRITE` (see note below
+regarding this last permission).
+
+.. versionadded:: 3.1
+    The :attr:`Permission.BROWSE` permission is used to provide listing access of contents when targeting a
+    :term:`Resource` of type :class:`magpie.models.Directory`. When targeting a :class:`magpie.models.File`, it instead
+    provides *metadata* access to that file.
+
+Permission :attr:`Permission.READ` can be applied to all of the resources, but will only effectively make sense when
+attempting access of a specific :term:`Resource of type :class:`magpie.models.File`.
+
+.. versionchanged:: 3.1
+    Permission :attr:`Permission.READ` does not offer *metadata* content listing of :class:`magpie.models.Directory`
+    anymore. For this, :attr:`Permission.BROWSE` should be used instead. Setting :attr:`Permission.READ` on a
+    directory will only be logical when combined with :attr:`Scope.RECURSIVE`, in which case `Magpie` will interpret
+    the :term:`Effective Permissions` to allow read access to all :class:`magpie.models.File` under that directory, at
+    any depth level, unless denied by a lower-level specification.
+
+Finally, :attr:`Permission.WRITE` can also be applied on all of the resources, but are not explicitly employed during
+parsing of incoming requests.
+
+.. note::
+    The :attr:`Permission.WRITE` is not handled by `ServiceTHREDDS`_ itself during :term:`ACL` resolution as it is not
+    considered by :meth:`magpie.services.ServiceTHREDDS.permission_requested` that never returns this value. The method
+    only returns either :attr:`Permission.BROWSE` or :attr:`Permission.READ`. An :term:`User` or :term:`Group` can still
+    have this :term:`Applied Permission` to allow a third party service to interrogate `Magpie API` about the presence
+    of :attr:`Permission.WRITE` permission and perform the appropriate action with the result. The
+    :term:`Effective Permissions` API routes will provide the resolved ``access``. It is only `Twitcher`_ proxy that
+    will not be able to make use of it during incoming requests as it depends on
+    :class:`magpie.adapter.magpieowssecurity.MagpieOWSSecurity`, which in turn employs the result from the :term:`ACL`.
+
+    The :attr:`Permission.WRITE` is mostly preserved for backward compatibility of services that employed
+    `ServiceTHREDDS`_ to obtain information about which directory or files (which registered `Magpie` :term:`Resource`)
+    are writable or not, although using another upload methodology that is not specifically executed via the actual
+    remote `THREDDS`_ service.
+
+
+As presented above, the main two permissions are :attr:`Permission.BROWSE` and :attr:`Permission.READ` which
+correspondingly serve to retrieve *metadata* and actual *data* of a given :term:`Resource`. To distinguish requests
+between these two types of contents, `ServiceTHREDDS`_ employs two parts from the request path, the sub-path *prefix*
+and the file *extension*. A default methodology is employed categorize these two types of content, but can be modified
+using custom configurations as described in :ref:`Custom THREDDS Settings` section.
+
+
+
+
+Custom THREDDS Settings
+~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. note::
+    See section :ref:`ServiceTHREDDS` about the base implementation components of this service type for further details.
+
+
+With either the `providers.cfg`_ or the `Combined Configuration File`_ presented in previous sections, a service of
+type `THREDDS`_ can be slightly customized to meet the intended needs. Specifically, two additional fields
+``metadata_type`` and ``data_type
+
+
+As presented above, the main two permissions are :attr:`Permission.BROWSE` and :attr:`Permission.READ` which
+correspondingly serve to retrieve *metadata* and actual *data* of a given :term:`Resource`. To distinguish requests
+between these two types of contents, `ServiceTHREDDS`_ employs two parts from the request path, the sub-path *prefix*
+and the file *extension*. The *default* methodology employed to categorize these two types of content is presented
+below.
+
+.. code-block:: YAML
+
+    providers:
+      thredds_service:
+        data_type:
+          - prefix: fileServer
+          - prefix: dodsC
+          - prefix: wcs
+          - prefix: wms
+        metadata_type:
+          # no extra sub-path, case when navigating contents using an UI browser
+          # (`~` is the representation method of JSON's `null`, make sure to adjust accordingly with file extension)
+          - prefix: ~
+          - prefix: catalog
+          - prefix: ncml
+          - prefix: uddc
+          - prefix: iso
+
+
+
+.. note::
+    The *default* categorization between *metadata*/*data* contents can be modified using custom configurations
+    specified within a `providers.cfg`_ or :ref:`Combined Configuration File`.
+
 
 ServiceGeoserverWMS
 ~~~~~~~~~~~~~~~~~~~~~

magpie/permissions.py

@@ -23,6 +23,7 @@ class Permission(ExtendedEnum):
     READ = "read"
     WRITE = "write"
     ACCESS = "access"
+    BROWSE = "browse"
     # WPS permissions
     GET_CAPABILITIES = "getcapabilities"
     GET_MAP = "getmap"

magpie/services.py

@@ -656,10 +656,10 @@ class ServiceTHREDDS(ServiceInterface):
     service_type = "thredds"
 
     permissions = [
+        Permission.BROWSE,
+        # read are for
         Permission.READ,
-        # FIXME:
-        #   does WRITE permission even make sense here?
-        #   leave it for bw-compat, but not even considered since 'permission_requested' always returns READ
+        # NOTE: see special usage of WRITE in docs
         Permission.WRITE,
     ]