RFC.3

RCT Configuration System

We use configuration options in different places in the stack, in the form of configuration files and env variables. Alas, different conventions hold on different layers. Specifically, the RS layer configuration is very different from the RP layer. As the RS config layer turns out to be insufficient, I propose to use the opportunity to converge to a common configuration handling across the stack.

Proposal from here:

# ------------------------------------------------------------------------------
#
# We provide a json based config file parser with several custom extensions
#
#   - system config files will be merged with user configs (if those exist)
#   - python style comments are filtered out before parsing 
#   - after parsing, values are set or expanded via `os.environ`
#
#
# Config Names and Locations
# --------------------------
#
# We assume two basic locations for config files: one is installed within the
# scope of a Python module, the other one is under user control, and usually in
# the users home directory.  The  config reader accepts the following parameters
# to derive the exact locations:
#
#   - module: name of module under which the config is installed
#   - path  : config file path relative to the module home
#   - name  : config file name relative to the module home
#
# The `module` string is interpreted as follows:
#
#   m = __import__('module')
#   sys_config_dir = "%s/configs" % os.path.dirname(m.__file__)
#   usr_config_dir = "%s/.%s/"    % (os.environ['HOME'], m.replace('.', '/'))
#
# so the location of the module's `__init__.py` is used to derive the location
# of the installed system config files, and the module name is used to derive
# the location of the user provided config files.
#
# For example, the module `radical.pilot` will have the following config dirs:
#
#   sys_config_dir = /tmp/ve/lib/python2.7/site-packages/radical/utils/configs/
#   usr_config_dir = /home/merzky/.radical/utils/
#
# The remaining two arguments are exclusive (exactly *one* must be specified).  
# If `path` is given, it is interpreted as a path under those locations.  
# If `name` is given, then the same `. -> /` replacement as on the module name 
# is performed, and the result is interpreted like `path` again.  
#
# In both cases, we add the file extension `.json` if no match is found without 
# it.  It is not an error if the so specified config files do not exist -- in
# that case, the config is considered empty.
#
# After loading the system level config file, any existing user level config
# file is merged into it, via
#
#   radical.utils.dict_merge(user_cgf, system_cfg, mode='overwrite')
#
# so that the user config settings supercede the system config settings.
#
# Both path and name specifiers can contain `*` as wildcard, which is then
# interpreted as by `glob()`.  If that wirldcard exist, then all matching config
# files are read into *one* configuration dict, where each root key is set to
# the value the '*' expands to (minus the `.json` extension).
#
# For example, the name `radical.pilot.resource_*` with the following config
# files:
#
#   /tmp/ve/[...]/radical/pilot/configs/resource_xsede.json
#   /tmp/ve/[...]/radical/pilot/configs/resource_ncsa.json
#
# will result in a config dict like:
#
#   {
#     'xsede' : { 'foo' : 'bar' },
#     'ncsa'  : { 'foo' : 'baz' }
#   }
#
# User configuration files are expected to match that structure.
#
#
# Queries
# -------
#
# We support two types of queries on the resulting parsed configs:
#
#   - dict like queries (via `ru.DictMixin`)
#   - the `query(key)` method returns a single value, or 'None' if not found.
#
# In the latter `query()` case, the `key` can be specified as dot-separated
# path, so that the following two snippets are equivalent:
#
#   val = cfg['foo']['bar'].get('baz')
#   val = cfg.query('foo.bar.baz')
#
#
# Environment
# -----------
# 
# Towards `os.environ` completion, we support the following syntax in all string
# *values* (not keys):
#
#   '$env{RADICAL_UTILS_ENV:default_value}
#
# which will be replaced by 
#
#   `os.environ.get('RADICAL_UTILS_ENV', 'default_value)`
#
# The default value is optional.  Env evaluation is only performed at time of
# parsing, not at time of query.
#
#
# Validation
# ----------
#
# It probably makes sense to switch to a json schema validator at some point,
# see for example https://pypi.python.org/pypi/json-schema-validator. For we
# remain schema-less, and will thus, in a very pythonesque way, only fail once
# values are queried or used.
#
# ------------------------------------------------------------------------------