notebooks/config.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "d023d283-7e1d-4e75-95b8-5033bea42a59",
   "metadata": {},
   "source": [
    "# Config Tool\n",
    "\n",
    "The `uwtools` API's `config` module provides functions to create and manipulate configuration files, objects, and dictionaries.\n",
    "\n",
    "<div class=\"alert alert-warning\"><b>Note: </b>This notebook was tested using <code>uwtools</code> version 2.6.0.</div>\n",
    "<div class=\"alert alert-info\">For more information, please see the <a href=\"https://uwtools.readthedocs.io/en/2.5.0/sections/user_guide/api/config.html\">uwtools.api.config</a> Read the Docs page.</div>\n",
    "\n",
    "## Table of Contents\n",
    "\n",
    "* [Getting Config Objects](#Getting-Config-Objects)\n",
    "* [Config Depth Limitations](#Config-Depth-Limitations)\n",
    "* [Realizing Configs](#Realizing-Configs)\n",
    "    * [Updating Configs](#Updating-Configs)\n",
    "    * [Using the `key_path` Parameter](#Using-the-key_path-Parameter)\n",
    "    * [Using the `values_needed` Parameter](#Using-the-values_needed-Parameter)\n",
    "    * [Using the `total` Parameter](#Using-the-total-Parameter)\n",
    "* [Realizing Configs to a Dictionary](#Realizing-Configs-to-a-Dictionary)\n",
    "* [Comparing Configs](#Comparing-Configs)\n",
    "* [Validating Configs](#Validating-Configs)\n",
    "* [Working with Config Classes](#Working-with-Config-Classes)\n",
    "    * [Comparing Config Objects](#Comparing-Config-Objects)\n",
    "    * [Rendering Values](#Rendering-Values)\n",
    "    * [Writing Configs in a Specified Format](#Writing-Configs-in-a-Specified-Format)\n",
    "    * [Updating Values](#Updating-Values) \n",
    "<!--cell 0-->"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "6d02d033-0992-4990-861d-3f80d09d7083",
   "metadata": {},
   "outputs": [],
   "source": [
    "from pathlib import Path\n",
    "from uwtools.api import config\n",
    "from uwtools.api.logging import use_uwtools_logger\n",
    "\n",
    "use_uwtools_logger()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "212594bb-379f-4441-805e-af0dbabe1815",
   "metadata": {},
   "source": [
    "## Getting Config Objects\n",
    "\n",
    "The `config` tool can create configuration objects given a Python ``dict`` or a file in one of five different formats: FieldTable, INI, Fortran namelist, Shell, or YAML. `config.get_yaml_config` is demonstrated here, but the config module also has similar functions for each of the other supported formats: `get_fieldtable_config()`,  `get_ini_config()`, `get_nml_config()`, and `get_sh_config()`.\n",
    "<!--cell 2-->"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "ab0e21c3-a4b6-404c-bffd-e0d393d9b0a2",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Help on function get_yaml_config in module uwtools.api.config:\n",
      "\n",
      "get_yaml_config(config: Union[dict, str, pathlib.Path, NoneType] = None, stdin_ok: bool = False) -> uwtools.config.formats.yaml.YAMLConfig\n",
      "    Get a ``YAMLConfig`` object.\n",
      "\n",
      "    :param config: YAML file or ``dict`` (``None`` => read ``stdin``).\n",
      "    :param stdin_ok: OK to read from ``stdin``?\n",
      "    :return: An initialized ``YAMLConfig`` object.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "help(config.get_yaml_config)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "606da5b3-4bff-4148-a9a5-908aa7dd5e8c",
   "metadata": {},
   "source": [
    "The `stdin_ok` argument can be used to permit reads from `stdin`, but this is a rare use case beyond the scope of this notebook that will not be discussed here.\n",
    "\n",
    "`get_yaml_config()` can take input from a Python `dict` or a YAML file like the one below.\n",
    "<!--cell 4-->"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "c6e049df-38f6-4879-8e0d-68356226d94b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "greeting: Hello\n",
      "recipient: World\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "cat fixtures/config/get-config.yaml"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c2f72448-d35e-4a0b-a371-cb47c7b3338b",
   "metadata": {},
   "source": [
    "Paths to config files can be provided either as a string or <a href=\"https://docs.python.org/3/library/pathlib.html#pathlib.Path\">Path</a> object. Since `get_yaml_config()` is used here, a `YAMLConfig` object is returned.\n",
    "<!--cell 6-->"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "cc3020a6-4eb4-4830-9263-a9fc8fac7450",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'uwtools.config.formats.yaml.YAMLConfig'>\n",
      "greeting: Hello\n",
      "recipient: World\n"
     ]
    }
   ],
   "source": [
    "config1 = config.get_yaml_config(\n",
    "    config=Path(\"fixtures/config/get-config.yaml\")\n",
    ")\n",
    "print(type(config1))\n",
    "print(config1)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b7bcd736-ff78-4e8b-957f-b348b812c5f6",
   "metadata": {},
   "source": [
    "Providing a Python `dict` will create a UW `Config` object with format matching the function used.\n",
    "<!--cell 8-->"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "f01ac223-4a02-40ba-822f-8e66ad39f313",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "message:\n",
      "  greeting: Hi\n",
      "  recipient: Earth\n"
     ]
    }
   ],
   "source": [
    "input_config = {\"message\": {\"greeting\":\"Hi\", \"recipient\":\"Earth\"}}\n",
    "config2 = config.get_yaml_config(\n",
    "    config=input_config\n",
    ")\n",
    "print(config2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dc745e95-d1ce-435c-a488-13b761979e36",
   "metadata": {},
   "source": [
    "## Config Depth Limitations\n",
    "\n",
    "Some config formats have limitations on the depth of their nested configs. Shell configs, for example, may only contain top-level, bash-syntax `key=value` pairs.\n",
    "<!--cell 10-->"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "28d23ac5-52a0-45bc-bfee-98d9ea518ca2",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "greeting=Salutations\n",
       "recipient=Mars"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "config.get_sh_config(\n",
    "    config={\"greeting\":\"Salutations\", \"recipient\":\"Mars\"}\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3dd7292f-cbd1-4e45-b641-1e213a4ead07",
   "metadata": {},
   "source": [
    "Shell configs cannot be nested, and any attempt to do so will raise a `UWConfigError`.\n",
    "<!--cell 12-->"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "b3a0a5bc-9d1b-4d48-a05f-be6f94fb6e1d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cannot instantiate depth-1 SHConfig with depth-2 config\n"
     ]
    }
   ],
   "source": [
    "try: \n",
    "    config.get_sh_config(\n",
    "        config={\"message\": {\"greeting\":\"Salutations\", \"recipient\":\"Mars\"}}\n",
    "    )\n",
    "except Exception as e:\n",
    "    print(e)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7677bdaa-8707-4dff-b812-91b4521f4820",
   "metadata": {},
   "source": [
    "When creating INI configs, exactly one level of nesting is required so that each key-value pair is contained within a section. The top level keys become sections, which are contained within square brackets `[]`. Read more about INI configuration files <a href=\"https://en.wikipedia.org/wiki/INI_file\">here</a>.\n",
    "<!--cell 14-->"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "6837e75b-bd20-4c3b-bd33-650e4b4f9f23",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[message]\n",
       "greeting = Salutations\n",
       "recipient = Mars"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "config.get_ini_config(\n",
    "    config={\"message\": {\"greeting\":\"Salutations\", \"recipient\":\"Mars\"}}\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "04ab0d52-75cc-4227-8058-a9a1faba7b54",
   "metadata": {},
   "source": [
    "Either more or fewer levels of nesting will raise a `UWConfigError`.\n",
    "<!--cell 16-->"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "9655b36b-2d39-4fc1-b3b8-9cb3443cf4b8",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cannot instantiate depth-2 INIConfig with depth-1 config\n"
     ]
    }
   ],
   "source": [
    "try:\n",
    "    config.get_ini_config(\n",
    "        config={\"greeting\":\"Salutations\", \"recipient\":\"Mars\"}\n",
    "    )\n",
    "except Exception as e:\n",
    "    print(e)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e4403333-c247-465f-9a75-f96d1be914f6",
   "metadata": {},
   "source": [
    "## Realizing Configs\n",
    "\n",
    "The `config.realize()` function writes config files to disk or `stdout` with the ability to render Jinja2 expressions and add/update values.\n",
    "<!--cell 18-->"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "2c3f1b75-b26f-4893-beb7-37a58c09f511",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Help on function realize in module uwtools.api.config:\n",
      "\n",
      "realize(input_config: Union[uwtools.config.formats.base.Config, pathlib.Path, dict, str, NoneType] = None, input_format: Optional[str] = None, update_config: Union[uwtools.config.formats.base.Config, pathlib.Path, dict, str, NoneType] = None, update_format: Optional[str] = None, output_file: Union[str, pathlib.Path, NoneType] = None, output_format: Optional[str] = None, key_path: Optional[list[Union[bool, float, int, str]]] = None, values_needed: bool = False, total: bool = False, dry_run: bool = False, stdin_ok: bool = False) -> dict\n",
      "    Realize a config based on a base input config and an optional update config.\n",
      "\n",
      "    The input config may be specified as a filesystem path, a ``dict``, or a ``Config`` object. When it\n",
      "    is not, it will be read from ``stdin``.\n",
      "\n",
      "    If an update config is specified, it is merged onto the input config, augmenting or overriding base\n",
      "    values. It may be specified as a filesystem path, a ``dict``, or a ``Config`` object. When it is\n",
      "    not, it will be read from ``stdin``.\n",
      "\n",
      "    At most one of the input config or the update config may be left unspecified, in which case the\n",
      "    other will be read from ``stdin``. If neither filename or format is specified for the update config, no\n",
      "    update will be performed.\n",
      "\n",
      "    The output destination may be specified as a filesystem path. When it is not, it will be written to\n",
      "    ``stdout``.\n",
      "\n",
      "    If ``values_needed`` is ``True``, a report of values needed to realize the config is logged. In\n",
      "    ``dry_run`` mode, output is written to ``stderr``.\n",
      "\n",
      "    If ``total`` is ``True``, an exception will be raised if any Jinja2 variables/expressions cannot be\n",
      "    rendered. Otherwise, such variables/expressions will be passed through unchanged in the output.\n",
      "\n",
      "    Recognized file extensions are: ini, nml, sh, yaml\n",
      "\n",
      "    :param input_config: Input config file (``None`` => read ``stdin``).\n",
      "    :param input_format: Format of the input config (optional if file's extension is recognized).\n",
      "    :param update_config: Update config file (``None`` => read ``stdin``).\n",
      "    :param update_format: Format of the update config (optional if file's extension is recognized).\n",
      "    :param output_file: Output config file (``None`` => write to ``stdout``).\n",
      "    :param output_format: Format of the output config (optional if file's extension is recognized).\n",
      "    :param key_path: Path of keys to the desired output block.\n",
      "    :param values_needed: Report complete, missing, and template values.\n",
      "    :param total: Require rendering of all Jinja2 variables/expressions.\n",
      "    :param dry_run: Log output instead of writing to output.\n",
      "    :param stdin_ok: OK to read from ``stdin``?\n",
      "    :return: The ``dict`` representation of the realized config.\n",
      "    :raises: UWConfigRealizeError if ``total`` is ``True`` and any Jinja2 variable/expression was not rendered.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "help(config.realize)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "501da514-a654-4511-928f-b2ad7db102b2",
   "metadata": {},
   "source": [
    "The `input_config` parameter takes a config from a string path, <a href=\"https://docs.python.org/3/library/pathlib.html#pathlib.Path\">Path</a> object, Python `dict`, or UW `Config` object like the `YAMLConfig` object from the <a href=\"#Getting-Config-Objects\">Getting Config Objects</a> section. The `input_format` argument must be provided for `dict` inputs or for files without recognized extensions. Configs are written to `stdout` if `output_file` is unspecified or explicitly set to `None`, or to the file specified by `output_file`. The `output_format` argument must be provided when writing to `stdout` or to a file without a recognized extension. Recognized extensions are: `.ini`, `.nml`, `.sh`, and `.yaml`.\n",
    "<!--cell 20-->"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "77873e14-db3c-417d-be7a-2ba12c9a38f6",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'greeting': 'Hello', 'recipient': 'World'}"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "config.realize(\n",
    "    input_config=config1,\n",
    "    output_file=Path('tmp/config1.yaml')\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fd1823c4-6c3d-4d4a-a614-4d1238588bdd",
   "metadata": {},
   "source": [
    "The `realize()` method returns a dict version of the config regardless of input type, and the file is written in the YAML format as indicated by the file extension.\n",
    "<!--cell 22-->"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "4f237a73-da83-4632-990f-644632b15cd9",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "greeting: Hello\n",
      "recipient: World\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "cat tmp/config1.yaml"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "902c4863-e772-49c6-88f2-31e1ab76b418",
   "metadata": {},
   "source": [
    "Input and output formats are not required to match. This can be used to convert some configs from one format to another. YAML configs can be converted to configs of other recognized formats so long as the depth restrictions of the output format are met. All configs of recognized formats can be converted into YAML configs. Keep in mind that some formats are unable to express some types (for example, Shell configs can't express a value as an `int` while a Fortran namelist can) so type information may be lost when converting between formats.\n",
    "<!--cell 24-->"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "b8854dc6-9dd2-4843-99e4-278b116b9767",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'greeting': 'Hello', 'recipient': 'World'}"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "config.realize(\n",
    "    input_config='fixtures/config/get-config.yaml',\n",
    "    input_format='yaml',\n",
    "    output_file='tmp/realize-config.sh',\n",
    "    output_format='sh'\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4032ab38-f6c7-45d1-bff7-f70d23832f26",
   "metadata": {},
   "source": [
    "Here a Shell config is created from a YAML config.\n",
    "<!--cell 26-->"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "0a59a3e8-27b5-4daa-a924-941aceaad157",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "greeting=Hello\n",
      "recipient=World\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "cat tmp/realize-config.sh"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ef0136e4-f549-467f-9341-f77f84738bb0",
   "metadata": {},
   "source": [
    "### Updating Configs\n",
    "\n",
    "Configs can be updated by providing a second config with the `update_config` parameter. If the update config contains keys that match the base config, the base config values for those keys will be overwritten. Once updated, if the config contains Jinja2 expressions, like the one below, they will be rendered in the config wherever possible.\n",
    "<!--cell 28-->"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "2d72bbc1-e438-48b2-8bd7-554b598c6f24",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "&memo\n",
      "  sender_id = \"{{ id }}\"\n",
      "  message = \"{{ greeting }}, {{ recipient }}!\"\n",
      "  sent = .FALSE.\n",
      "/\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "cat fixtures/config/base-config.nml"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "67b07a34-1bea-402f-9cec-f33a2e519d27",
   "metadata": {},
   "source": [
    "Here, the update config provides values that will update two of the Jinja2 expressions and override one key with a new value.\n",
    "<!--cell 30-->"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "7648fdd5-5752-4bf3-b366-db8da1eac601",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'memo': {'sender_id': '{{ id }}',\n",
       "  'message': 'Salutations, Mars!',\n",
       "  'sent': True,\n",
       "  'greeting': 'Salutations',\n",
       "  'recipient': 'Mars'}}"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "config.realize(\n",
    "    input_config='fixtures/config/base-config.nml',\n",
    "    update_config={\"memo\": {\"greeting\":\"Salutations\", \"recipient\":\"Mars\", \"sent\": True}},\n",
    "    output_file='tmp/updated-config.nml'\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bdac7f6c-4452-4137-9104-784449584100",
   "metadata": {},
   "source": [
    "All of the key-value pairs were added to the updated config, and the base config was rendered where the appropriate values were provided. However, not all Jinja2 expressions are required to be rendered: An `id` key was not provided in the update config, so the expression referencing it was not rendered.\n",
    "<!--cell 32-->"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "f4538965-d3b0-4c0c-a878-b6852f8d8ab0",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "&memo\n",
      "    sender_id = '{{ id }}'\n",
      "    message = 'Salutations, Mars!'\n",
      "    sent = .true.\n",
      "    greeting = 'Salutations'\n",
      "    recipient = 'Mars'\n",
      "/\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "cat tmp/updated-config.nml"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a20024ac-0f33-4000-942d-29c99dc0502e",
   "metadata": {},
   "source": [
    "### Using the `key_path` Parameter\n",
    "\n",
    "Consider the following config file, where the desired keys and values are not at the top level.\n",
    "<!--cell 34-->"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "7ce0e917-f0d0-4302-9c8c-b136ffc5410a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "keys:\n",
      "  to:\n",
      "    config:\n",
      "      message: \"{{ greeting }}, {{ recipient }}!\"\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "cat fixtures/config/keys-config.yaml"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "633778ca-391c-4ba4-b3b4-b3f599af5d41",
   "metadata": {},
   "source": [
    "The `key_path` parameter allows only a portion of the config, identified by following a given list of keys, to be written to a file or, in this case, to `stdout`. Note that the key-value pairs from the update config are used to render values, but don't appear in the config written to `stdout`.\n",
    "<!--cell 36-->"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "00150efa-848c-44eb-ac0c-dab3845546b8",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "message: Good morning, Venus!\n"
     ]
    }
   ],
   "source": [
    "_ = config.realize(\n",
    "    input_config=\"fixtures/config/keys-config.yaml\",\n",
    "    update_config={\"greeting\": \"Good morning\", \"recipient\": \"Venus\"},\n",
    "    output_file=None,\n",
    "    output_format='yaml',\n",
    "    key_path=['keys', 'to', 'config']\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "903ea23f-4625-48f0-9efc-1c0a106d5bf6",
   "metadata": {},
   "source": [
    "### Using the `values_needed` Parameter\n",
    "\n",
    "Consider the config file below, which contains unrendered Jinja2 expressions.\n",
    "<!--cell 38-->"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "928a23d7-8ba9-4217-935d-01563bb36cb6",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "&memo\n",
      "  sender_id = \"{{ id }}\"\n",
      "  message = \"{{ greeting }}, {{ recipient }}!\"\n",
      "  sent = .FALSE.\n",
      "/\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "cat fixtures/config/base-config.nml"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7d878080-55ae-4233-bb65-37cfa5ef7cff",
   "metadata": {},
   "source": [
    "Setting `values_needed` to `True` will allow logging of keys that contain unrendered Jinja2 expressions and their values. A logger needs to be initialized for this information to be displayed. The config is not written and the returned `dict` is empty.\n",
    "<!--cell 40-->"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "d22c692d-e98e-4f88-bdac-a369f0a1962f",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[2024-11-19T23:12:52]     INFO Keys that are complete:\n",
      "[2024-11-19T23:12:52]     INFO   memo\n",
      "[2024-11-19T23:12:52]     INFO   memo.sent\n",
      "[2024-11-19T23:12:52]     INFO \n",
      "[2024-11-19T23:12:52]     INFO Keys with unrendered Jinja2 variables/expressions:\n",
      "[2024-11-19T23:12:52]     INFO   memo.sender_id: {{ id }}\n",
      "[2024-11-19T23:12:52]     INFO   memo.message: {{ greeting }}, {{ recipient }}!\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{}"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "config.realize(\n",
    "    input_config='fixtures/config/base-config.nml',\n",
    "    output_file=None,\n",
    "    output_format='nml',\n",
    "    values_needed=True\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ddbc989a-a9ed-4649-8e87-f74d1ff43b89",
   "metadata": {},
   "source": [
    "### Using the `total` Parameter\n",
    "\n",
    "The `total` parameter is used to specify that all Jinja2 expressions must be rendered before the final config is written. Consider the config below which contains multiple expressions.\n",
    "<!--cell 42-->"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "647538d9-cd22-4f94-b15b-c34d68a324da",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "&memo\n",
      "  sender_id = \"{{ id }}\"\n",
      "  message = \"{{ greeting }}, {{ recipient }}!\"\n",
      "  sent = .FALSE.\n",
      "/\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "cat fixtures/config/base-config.nml"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7c0d64be-c97c-4d07-b636-54b1c8bb5d0c",
   "metadata": {},
   "source": [
    "As was shown in the <a href=\"#Updating-Configs\">Updating Configs</a> section, by default not all Jinja2 expressions are required to be rendered. However, when `total` is set to `True` and not enough values are provided to fully realize the config, a `UWConfigRealizeError` is raised. Notice that values are provided for `greeting` and `recipient`, but not for `id`.\n",
    "<!--cell 44-->"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "ba360102-f558-4dba-b0b6-c6f550c7d40f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Config could not be totally realized\n"
     ]
    }
   ],
   "source": [
    "try:\n",
    "    config.realize(\n",
    "        input_config='fixtures/config/base-config.nml',\n",
    "        update_config={\"memo\": {\"greeting\":\"Salutations\", \"recipient\":\"Mars\", \"sent\":True}},\n",
    "        output_file='tmp/config-total.nml',\n",
    "        total=True\n",
    "    )\n",
    "except Exception as e:\n",
    "    print(e)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4048efdd-42f4-4400-bdba-1c8e0001d8f6",
   "metadata": {},
   "source": [
    "With all values provided to fully render the config, `realize()` writes the complete config without error.\n",
    "<!--cell 46-->"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "e891a446-f699-460d-b4a8-568d9d4cf631",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'memo': {'sender_id': '321',\n",
       "  'message': 'Salutations, Mars!',\n",
       "  'sent': True,\n",
       "  'greeting': 'Salutations',\n",
       "  'recipient': 'Mars',\n",
       "  'id': 321}}"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "config.realize(\n",
    "    input_config='fixtures/config/base-config.nml',\n",
    "    update_config={\"memo\": {\"greeting\":\"Salutations\", \"recipient\":\"Mars\", \"sent\":True, \"id\":321}},\n",
    "    output_file='tmp/config-total.nml',\n",
    "    total=True\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "07876d41-dad2-4c30-adba-f635050708ed",
   "metadata": {},
   "source": [
    "The newly created config file is free from any unrendered Jinja2 expressions.\n",
    "<!--cell 48-->"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "aeab0ec5-7e6e-4309-b484-4de5dd9324b5",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "&memo\n",
      "    sender_id = '321'\n",
      "    message = 'Salutations, Mars!'\n",
      "    sent = .true.\n",
      "    greeting = 'Salutations'\n",
      "    recipient = 'Mars'\n",
      "    id = 321\n",
      "/\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "cat tmp/config-total.nml"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e121dab6-5d16-40b0-aa16-161c329a8e9a",
   "metadata": {},
   "source": [
    "## Realizing Configs to a Dictionary\n",
    "\n",
    "The `config.realize_to_dict()` function has the ability to manipulate config values, and returns the config as a Python `dict` just as `realize()` does. However, a config won't be written to a file or to `stdout`. Like `realize()`, input or update configs can be Python dictionaries, UW `Config` objects, or files like the one below.\n",
    "<!--cell 50-->"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "bdc98ce1-e213-41ac-b1f2-03bd52238e30",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "greeting: Hello\n",
      "recipient: World\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "cat fixtures/config/get-config.yaml"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eeacc37f-b808-4d0e-9e29-ff8aa1fc2ff2",
   "metadata": {},
   "source": [
    "`realize_to_dict()` has the same parameters as `realize()`, with the exception of `output_file` and `output_format`. Instead, configs can be manipulated or converted to a `dict` without the need to specify an output file or format.\n",
    "<!--cell 52-->"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "ae7f648e-6586-4700-87e7-492ca3a02a06",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'id': '456', 'greeting': 'Hello', 'recipient': 'World'}"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "config.realize_to_dict(\n",
    "    input_config={\"id\": \"456\"},\n",
    "    update_config=\"fixtures/config/get-config.yaml\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "be66b870-a42c-46f2-a045-b769354403de",
   "metadata": {},
   "source": [
    "For more details on usage and parameters, see the <a href=\"#Realizing-Configs\">Realizing Configs</a> section above.\n",
    "\n",
    "## Comparing Configs\n",
    "\n",
    "The `config` tool can be used to compare two configuration files using `config.compare()`.\n",
    "<!--cell 54-->"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "249ae7a9-1ad3-4401-98b0-bc3c433f22f4",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Help on function compare in module uwtools.api.config:\n",
      "\n",
      "compare(config_1_path: Union[pathlib.Path, str], config_2_path: Union[pathlib.Path, str], config_1_format: Optional[str] = None, config_2_format: Optional[str] = None) -> bool\n",
      "    Compare two config files.\n",
      "\n",
      "    Recognized file extensions are: ini, nml, sh, yaml\n",
      "\n",
      "    :param config_1_path: Path to 1st config file.\n",
      "    :param config_2_path: Path to 2nd config file.\n",
      "    :param config_1_format: Format of 1st config file (optional if file's extension is recognized).\n",
      "    :param config_2_format: Format of 2nd config file (optional if file's extension is recognized).\n",
      "    :return: ``False`` if config files had differences, otherwise ``True``.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "help(config.compare)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "908b17fe-ae2d-4964-a659-0481d063c037",
   "metadata": {},
   "source": [
    "Consider the following config files, which have similar values, with the exception of `sent`'s value.\n",
    "<!--cell 56-->"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "0218e5de-2c25-4d7a-a7b6-0f05ad81afb2",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "&memo\n",
      "  sender_id = \"{{ id }}\"\n",
      "  message = \"{{ greeting }}, {{ recipient }}!\"\n",
      "  sent = .FALSE.\n",
      "/\n",
      "----------------------------------------------\n",
      "&memo\n",
      "  sender_id = \"{{ id }}\"\n",
      "  message = \"{{ greeting }}, {{ recipient }}!\"\n",
      "  sent = .TRUE.\n",
      "/\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "cat fixtures/config/base-config.nml\n",
    "echo ----------------------------------------------\n",
    "cat fixtures/config/alt-config.nml "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c4361a7c-9c16-4bc0-b3e6-985d85f1450c",
   "metadata": {},
   "source": [
    "`compare()` returns `True` if the configs contain identical key-value pairs, and `False` otherwise. If a logger has been initialized, information is logged on which files are being compared and the values that differ, if any. Files are passed to `config_1_path` and `config_2_path` as a string filename or <a href=\"https://docs.python.org/3/library/pathlib.html#pathlib.Path\">Path</a> object. Corresponding optional formats may be passed using `config_1_format` and `config_2_format` and are only needed if the format suffix is not recognized.\n",
    "<!--cell 58-->"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "988b72bc-7983-4d24-9d37-c9ba32a31ac4",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[2024-11-19T23:12:52]     INFO - fixtures/config/base-config.nml\n",
      "[2024-11-19T23:12:52]     INFO + fixtures/config/alt-config.nml\n",
      "[2024-11-19T23:12:52]     INFO ---------------------------------------------------------------------\n",
      "[2024-11-19T23:12:52]     INFO ↓ ? = info | -/+ = line unique to - or + file | blank = matching line\n",
      "[2024-11-19T23:12:52]     INFO ---------------------------------------------------------------------\n",
      "[2024-11-19T23:12:52]     INFO   memo:\n",
      "[2024-11-19T23:12:52]     INFO     message: '{{ greeting }}, {{ recipient }}!'\n",
      "[2024-11-19T23:12:52]     INFO     sender_id: '{{ id }}'\n",
      "[2024-11-19T23:12:52]     INFO -   sent: false\n",
      "[2024-11-19T23:12:52]     INFO +   sent: true\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "False"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "config.compare(\n",
    "    config_1_path=Path('fixtures/config/base-config.nml'),\n",
    "    config_2_path='fixtures/config/alt-config.nml',\n",
    "    config_1_format='nml',\n",
    "    config_2_format='nml'\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "655717b7-b7db-4711-9dc4-c00557e6c2df",
   "metadata": {},
   "source": [
    "To see the behavior of `compare()` when key-value pairs are identical, one of the configs from above is copied in the cell below.\n",
    "<!--cell 60-->"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "ceb7f893-398b-41e4-818a-49cd036c2bfe",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "cp fixtures/config/base-config.nml tmp/config-copy.nml"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "02c75cf7-9f24-44a0-97c6-abd7b44fb60f",
   "metadata": {},
   "source": [
    "When these two files are compared, `True` is returned and the log reports no differences.\n",
    "<!--cell 62-->"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "8b495735-2396-436e-87db-d886ac6769fa",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[2024-11-19T23:12:52]     INFO - fixtures/config/base-config.nml\n",
      "[2024-11-19T23:12:52]     INFO + tmp/config-copy.nml\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "config.compare(\n",
    "    config_1_path='fixtures/config/base-config.nml',\n",
    "    config_2_path='tmp/config-copy.nml',\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eb4536f8-d198-46f8-ac74-ee9d2010704e",
   "metadata": {},
   "source": [
    "If a comparison is attempted between two files whose formats that don't match, `compare()` returns `False` and the mismatch is reported.\n",
    "<!--cell 64-->"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "438ccd5b-30d5-49b4-ba12-6e6e89f46d28",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[2024-11-19T23:12:52]    ERROR Formats do not match: yaml vs nml\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "False"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "config.compare(\n",
    "    config_1_path=Path('fixtures/config/get-config.yaml'),\n",
    "    config_2_path=Path('fixtures/config/base-config.nml')\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e14c0158-baa4-45d0-a86c-d9a245021229",
   "metadata": {},
   "source": [
    "## Validating Configs\n",
    "\n",
    "The `config.validate()` function checks if a given config conforms to a specified JSON schema.\n",
    "<!--cell 66-->"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "2eb5b2ce-bebd-449d-90a5-764484aa03aa",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Help on function validate in module uwtools.api.config:\n",
      "\n",
      "validate(schema_file: Union[pathlib.Path, str], config_data: Union[bool, dict, float, int, list, str, uwtools.config.formats.yaml.YAMLConfig, NoneType] = None, config_path: Union[str, pathlib.Path, NoneType] = None, stdin_ok: bool = False) -> bool\n",
      "    Check whether the specified config conforms to the specified JSON Schema spec.\n",
      "\n",
      "    Specify at most one of config_data or config_path. If no config is specified, ``stdin`` is read\n",
      "    and will be parsed as YAML and then validated.\n",
      "\n",
      "    :param schema_file: The JSON Schema file to use for validation.\n",
      "    :param config_data: A config to validate.\n",
      "    :param config_path: A path to a file containing a config to validate.\n",
      "    :param stdin_ok: OK to read from ``stdin``?\n",
      "    :raises: TypeError if both config_* arguments specified.\n",
      "    :return: ``True`` if the YAML file conforms to the schema, ``False`` otherwise.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "help(config.validate)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "57cdde4c-b3ab-40b1-8e52-1f99ea6d9696",
   "metadata": {},
   "source": [
    "Consider the simple YAML config below. `validate()` used together with an appropriate JSON schema ensures that the config meets expectations before it's used elsewhere.\n",
    "<!--cell 68-->"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "dc7d2b80-ea92-4301-ae85-4edbf61bf510",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "greeting: Hello\n",
      "recipient: World\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "cat fixtures/config/get-config.yaml"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1eeca5ad-e969-4670-bddd-8e20b38f3d6c",
   "metadata": {},
   "source": [
    "Below is an example of a schema used to validate a config. It ensures that the required keys are present and the value types match expectations. For information on the keys used here and more, please refer to <a href=\"https://json-schema.org/docs\">JSON Schema documentation</a>.\n",
    "<!--cell 70-->"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "f9c66a24-2a7e-43be-839f-3ad5b136b646",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "  \"additionalProperties\": false,\n",
      "  \"properties\": {\n",
      "    \"greeting\": {\n",
      "      \"type\": \"string\"\n",
      "    },\n",
      "    \"recipient\": {\n",
      "      \"type\": \"string\"\n",
      "    }\n",
      "  },\n",
      "  \"required\": [\n",
      "    \"greeting\", \"recipient\"\n",
      "  ],\n",
      "  \"type\": \"object\"\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "cat fixtures/config/validate.jsonschema"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8c61a2d2-473c-45c6-9c6c-6c07fc5bf940",
   "metadata": {},
   "source": [
    "The schema file and config from above are passed to the respective `schema_file` and `config_path` parameters. Config file paths should be passed as a string or <a href=\"https://docs.python.org/3/library/pathlib.html#pathlib.Path\">Path</a> object. Files should be of YAML format, or parseable as YAML. Alternatively, a `YAMLConfig` object or a Python `dict` can be provided. `validate()` returns `True` if the config conforms to the JSON schema, and `False` otherwise. With a logger initialized, details about any validation errors are reported.\n",
    "<!--cell 72-->"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "f3873050-d857-490d-aeeb-f9217a7f808c",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[2024-11-19T23:12:53]     INFO 0 schema-validation errors found in config\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "config.validate(\n",
    "    schema_file='fixtures/config/validate.jsonschema',\n",
    "    config_path='fixtures/config/get-config.yaml'\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8d151205-4a95-4b46-aa1d-30ec30d96e88",
   "metadata": {},
   "source": [
    "A mutually-exclusive alternative to the `config_path` argument, the `config_data` argument commonly accepts a `dict` object, but can also validate configs based on `bool`, `float`, `int`, `list`, or `str` values. In the next example, validation errors exist, and the logger reports the number of errors found along with their locations and details.\n",
    "<!--cell 74-->"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "b5664e58-8ddc-438c-8180-1e2911838744",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[2024-11-19T23:12:53]    ERROR 1 schema-validation error found in config\n",
      "[2024-11-19T23:12:53]    ERROR Error at recipient:\n",
      "[2024-11-19T23:12:53]    ERROR   47 is not of type 'string'\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "False"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "config.validate(\n",
    "    schema_file='fixtures/config/validate.jsonschema',\n",
    "    config_data={'greeting':'Hello', 'recipient':47}\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fffeb768-7109-472f-916b-1407520b0f5e",
   "metadata": {},
   "source": [
    "## Working with Config Classes\n",
    "\n",
    "The `config` tool provides five classes that can be used to work with configs in an object-oriented way. The five different classes each work with a single format: `config.FieldTableConfig`, `config.INIConfig`, `config.NMLConfig`, `config.SHConfig`, and `config.YAMLConfig`. `config.INIConfig` is demonstrated here, but the other classes all use methods of the same names for working with each respective format.\n",
    "<!--cell 76-->"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "6f4df804-cda2-48c4-bf42-a5b57de5e066",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Help on class INIConfig in module uwtools.config.formats.ini:\n",
      "\n",
      "class INIConfig(uwtools.config.formats.base.Config)\n",
      " |  INIConfig(config: Union[dict, pathlib.Path, NoneType] = None)\n",
      " |\n",
      " |  Work with INI configs.\n",
      " |\n",
      " |  Method resolution order:\n",
      " |      INIConfig\n",
      " |      uwtools.config.formats.base.Config\n",
      " |      abc.ABC\n",
      " |      collections.UserDict\n",
      " |      collections.abc.MutableMapping\n",
      " |      collections.abc.Mapping\n",
      " |      collections.abc.Collection\n",
      " |      collections.abc.Sized\n",
      " |      collections.abc.Iterable\n",
      " |      collections.abc.Container\n",
      " |      builtins.object\n",
      " |\n",
      " |  Methods defined here:\n",
      " |\n",
      " |  __init__(self, config: Union[dict, pathlib.Path, NoneType] = None)\n",
      " |      :param config: Config file to load (None => read from stdin), or initial dict.\n",
      " |\n",
      " |  as_dict(self) -> dict\n",
      " |      Returns a pure dict version of the config.\n",
      " |\n",
      " |  dump(self, path: Optional[pathlib.Path] = None) -> None\n",
      " |      Dump the config in INI format.\n",
      " |\n",
      " |      :param path: Path to dump config to (default: stdout).\n",
      " |\n",
      " |  ----------------------------------------------------------------------\n",
      " |  Class methods defined here:\n",
      " |\n",
      " |  dump_dict(cfg: dict, path: Optional[pathlib.Path] = None) -> None\n",
      " |      Dump a provided config dictionary in INI format.\n",
      " |\n",
      " |      :param cfg: The in-memory config object to dump.\n",
      " |      :param path: Path to dump config to (default: stdout).\n",
      " |\n",
      " |  ----------------------------------------------------------------------\n",
      " |  Data and other attributes defined here:\n",
      " |\n",
      " |  __abstractmethods__ = frozenset()\n",
      " |\n",
      " |  ----------------------------------------------------------------------\n",
      " |  Methods inherited from uwtools.config.formats.base.Config:\n",
      " |\n",
      " |  __repr__(self) -> 'str'\n",
      " |      Return the string representation of a Config object.\n",
      " |\n",
      " |  compare_config(self, dict1: 'dict', dict2: 'Optional[dict]' = None, header: 'Optional[bool]' = True) -> 'bool'\n",
      " |      Compare two config dictionaries.\n",
      " |\n",
      " |      Assumes a section/key/value structure.\n",
      " |\n",
      " |      :param dict1: The first dictionary.\n",
      " |      :param dict2: The second dictionary (default: this config).\n",
      " |      :return: True if the configs are identical, False otherwise.\n",
      " |\n",
      " |  dereference(self, context: 'Optional[dict]' = None) -> 'Config'\n",
      " |      Render as much Jinja2 syntax as possible.\n",
      " |\n",
      " |  update_from(self, src: 'Union[dict, UserDict]') -> 'None'\n",
      " |      Update a config.\n",
      " |\n",
      " |      :param src: The dictionary with new data to use.\n",
      " |\n",
      " |  ----------------------------------------------------------------------\n",
      " |  Readonly properties inherited from uwtools.config.formats.base.Config:\n",
      " |\n",
      " |  config_file\n",
      " |      Return the path to the config file from which this object was instantiated, if applicable.\n",
      " |\n",
      " |  ----------------------------------------------------------------------\n",
      " |  Data descriptors inherited from uwtools.config.formats.base.Config:\n",
      " |\n",
      " |  __dict__\n",
      " |      dictionary for instance variables\n",
      " |\n",
      " |  __weakref__\n",
      " |      list of weak references to the object\n",
      " |\n",
      " |  ----------------------------------------------------------------------\n",
      " |  Methods inherited from collections.UserDict:\n",
      " |\n",
      " |  __contains__(self, key)\n",
      " |      # Modify __contains__ and get() to work like dict\n",
      " |      # does when __missing__ is present.\n",
      " |\n",
      " |  __copy__(self)\n",
      " |\n",
      " |  __delitem__(self, key)\n",
      " |\n",
      " |  __getitem__(self, key)\n",
      " |\n",
      " |  __ior__(self, other)\n",
      " |\n",
      " |  __iter__(self)\n",
      " |\n",
      " |  __len__(self)\n",
      " |\n",
      " |  __or__(self, other)\n",
      " |      Return self|value.\n",
      " |\n",
      " |  __ror__(self, other)\n",
      " |      Return value|self.\n",
      " |\n",
      " |  __setitem__(self, key, item)\n",
      " |\n",
      " |  copy(self)\n",
      " |\n",
      " |  get(self, key, default=None)\n",
      " |      D.get(k[,d]) -> D[k] if k in D, else d.  d defaults to None.\n",
      " |\n",
      " |  ----------------------------------------------------------------------\n",
      " |  Class methods inherited from collections.UserDict:\n",
      " |\n",
      " |  fromkeys(iterable, value=None)\n",
      " |\n",
      " |  ----------------------------------------------------------------------\n",
      " |  Methods inherited from collections.abc.MutableMapping:\n",
      " |\n",
      " |  clear(self)\n",
      " |      D.clear() -> None.  Remove all items from D.\n",
      " |\n",
      " |  pop(self, key, default=<object object at 0xffffa39d01c0>)\n",
      " |      D.pop(k[,d]) -> v, remove specified key and return the corresponding value.\n",
      " |      If key is not found, d is returned if given, otherwise KeyError is raised.\n",
      " |\n",
      " |  popitem(self)\n",
      " |      D.popitem() -> (k, v), remove and return some (key, value) pair\n",
      " |      as a 2-tuple; but raise KeyError if D is empty.\n",
      " |\n",
      " |  setdefault(self, key, default=None)\n",
      " |      D.setdefault(k[,d]) -> D.get(k,d), also set D[k]=d if k not in D\n",
      " |\n",
      " |  update(self, other=(), /, **kwds)\n",
      " |      D.update([E, ]**F) -> None.  Update D from mapping/iterable E and F.\n",
      " |      If E present and has a .keys() method, does:     for k in E.keys(): D[k] = E[k]\n",
      " |      If E present and lacks .keys() method, does:     for (k, v) in E: D[k] = v\n",
      " |      In either case, this is followed by: for k, v in F.items(): D[k] = v\n",
      " |\n",
      " |  ----------------------------------------------------------------------\n",
      " |  Methods inherited from collections.abc.Mapping:\n",
      " |\n",
      " |  __eq__(self, other)\n",
      " |      Return self==value.\n",
      " |\n",
      " |  items(self)\n",
      " |      D.items() -> a set-like object providing a view on D's items\n",
      " |\n",
      " |  keys(self)\n",
      " |      D.keys() -> a set-like object providing a view on D's keys\n",
      " |\n",
      " |  values(self)\n",
      " |      D.values() -> an object providing a view on D's values\n",
      " |\n",
      " |  ----------------------------------------------------------------------\n",
      " |  Data and other attributes inherited from collections.abc.Mapping:\n",
      " |\n",
      " |  __hash__ = None\n",
      " |\n",
      " |  __reversed__ = None\n",
      " |\n",
      " |  ----------------------------------------------------------------------\n",
      " |  Class methods inherited from collections.abc.Collection:\n",
      " |\n",
      " |  __subclasshook__(C)\n",
      " |      Abstract classes can override this to customize issubclass().\n",
      " |\n",
      " |      This is invoked early on by abc.ABCMeta.__subclasscheck__().\n",
      " |      It should return True, False or NotImplemented.  If it returns\n",
      " |      NotImplemented, the normal algorithm is used.  Otherwise, it\n",
      " |      overrides the normal algorithm (and the outcome is cached).\n",
      " |\n",
      " |  ----------------------------------------------------------------------\n",
      " |  Class methods inherited from collections.abc.Iterable:\n",
      " |\n",
      " |  __class_getitem__ = GenericAlias(...)\n",
      " |      Represent a PEP 585 generic type\n",
      " |\n",
      " |      E.g. for t = list[int], t.__origin__ is list and t.__args__ is (int,).\n",
      "\n"
     ]
    }
   ],
   "source": [
    "help(config.INIConfig)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "91a793e9-d1d1-41bb-84fd-c0c279606473",
   "metadata": {},
   "source": [
    "An object can be initialized by providing a config either as a Python `dict` or a <a href=\"https://docs.python.org/3/library/pathlib.html#pathlib.Path\">Path</a> to the file.\n",
    "<!--cell 78-->"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "0e8d6c8a-cc3c-49d7-9b97-32187cd5f754",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[fruit count]\n",
      "apples = 3\n",
      "grapes = {{ grape_count }}\n",
      "kiwis = 2\n"
     ]
    }
   ],
   "source": [
    "fruits = config.INIConfig(\n",
    "    config=Path('fixtures/config/fruit-config.ini')\n",
    ")\n",
    "print(fruits)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1a50a362-d063-4ae1-9b6a-d98d0bf84c4f",
   "metadata": {},
   "source": [
    "### Comparing Config Objects\n",
    "\n",
    "The `compare_config()` method compares two config `dict`s and returns `True` when they match and `False` otherwise. Two config `dict`s can be passed to the `dict1` and `dict2` parameters. Config objects of every format use the same method demonstrated here, and it stands as an alternative to `config.compare()`, which compares files rather than dictionaries. See the [Comparing Configs](#Comparing-Configs) section above for more details on `config.compare()`. The configs compared using `compare_config()` can be compared without regard for their intended format, since they are compared as dictionaries, but they must have a section/key/value structure.\n",
    "<!--cell 80-->"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "73925576-6b07-4af6-a9b0-7bd6f4235987",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fruits.compare_config(\n",
    "    dict1={'fruit count':{'apples':'3', 'grapes':'8', 'kiwis':'1'}},\n",
    "    dict2={'fruit count':{'apples':'3', 'grapes':'8', 'kiwis':'1'}}\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3caa77b7-2eca-498b-841f-b37c91068974",
   "metadata": {},
   "source": [
    "If `dict2` is left unspecified or set to `None`, the `dict1` config is compared to the config stored in the object itself. When there are differences between the two configs, as is the case here, `False` is returned. When a logger is initialized, the values that differ are displayed.\n",
    "<!--cell 82-->"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "id": "9d94a325-93f6-43b4-bc56-0ce5a1830647",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[2024-11-19T23:12:53]     INFO ---------------------------------------------------------------------\n",
      "[2024-11-19T23:12:53]     INFO ↓ ? = info | -/+ = line unique to - or + file | blank = matching line\n",
      "[2024-11-19T23:12:53]     INFO ---------------------------------------------------------------------\n",
      "[2024-11-19T23:12:53]     INFO   fruit count:\n",
      "[2024-11-19T23:12:53]     INFO     apples: '3'\n",
      "[2024-11-19T23:12:53]     INFO -   grapes: '{{ grape_count }}'\n",
      "[2024-11-19T23:12:53]     INFO +   grapes: '8'\n",
      "[2024-11-19T23:12:53]     INFO -   kiwis: '2'\n",
      "[2024-11-19T23:12:53]     INFO ?           ^\n",
      "[2024-11-19T23:12:53]     INFO +   kiwis: '1'\n",
      "[2024-11-19T23:12:53]     INFO ?           ^\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "False"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fruits.compare_config(\n",
    "    dict1={'fruit count':{'apples':'3', 'grapes':'8', 'kiwis':'1'}}\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a9c0020f-db59-4af3-a099-ebcdfbb017c8",
   "metadata": {},
   "source": [
    "### Rendering Values\n",
    "\n",
    "If the object's config contains unrendered Jinja2 expressions, the `dereference()` method will render as many as possible. The optional `context` parameter can be used to provide additional values with a Python `dict`.\n",
    "<!--cell 84-->"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "id": "65315885-28f7-4e34-b5e9-f07b51b85e42",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[fruit count]\n",
      "apples = 3\n",
      "grapes = 15\n",
      "kiwis = 2\n"
     ]
    }
   ],
   "source": [
    "fruits.dereference(\n",
    "    context={'grape_count':'15'}\n",
    ")\n",
    "print(fruits)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d9a330d1-35ff-4843-90fa-dd19e6eb98e9",
   "metadata": {},
   "source": [
    "### Writing Configs in a Specified Format\n",
    "\n",
    "Each of the `config` tool's classes provide methods that write configs of their format. With the `fruits` object, which is an instance of `INIConfig`, INI configs are written. `dump()` is one of these methods, which writes the config stored in the object to a file specified by providing the `path` parameter with a <a href=\"https://docs.python.org/3/library/pathlib.html#pathlib.Path\">Path</a> object. If `path` is `None`, the config is written to `stdout`.\n",
    "<!--cell 86-->"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "id": "7cd35285-9eb9-4796-b1d9-cf84160c0c0e",
   "metadata": {},
   "outputs": [],
   "source": [
    "fruits.dump(\n",
    "    path=Path('tmp/fruits.ini')\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0c0170d2-16ab-4b61-a3e5-2840b4f59271",
   "metadata": {},
   "source": [
    "Below we can see that the config was written in the INI format at the specified path.\n",
    "<!--cell 88-->"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "id": "f2ce9e59-f03c-4203-b929-87b4c0eae9de",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[fruit count]\n",
      "apples = 3\n",
      "grapes = 15\n",
      "kiwis = 2\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "cat tmp/fruits.ini"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "83a9d6b2-161f-4bc0-891c-477bd5c7a013",
   "metadata": {},
   "source": [
    "To write a config that is not stored in the object, the `dump_dict()` method is used. This method takes a config in the form of a `dict` and, like `dump()`, writes the config in the INI format to `stdout` if `path` is `None` or to the path that a <a href=\"https://docs.python.org/3/library/pathlib.html#pathlib.Path\">Path</a> object indicates.\n",
    "<!--cell 90-->"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "id": "b896632e-6a5a-4756-a32b-0630bc02d504",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[fruit count]\n",
      "oranges = 4\n",
      "blueberries = 9\n"
     ]
    }
   ],
   "source": [
    "other_fruits = {'fruit count':{'oranges':4, 'blueberries':9}}\n",
    "fruits.dump_dict(\n",
    "    cfg=other_fruits,\n",
    "    path=None\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "945d1e71-2704-49de-9a7c-dc8bf58c12fe",
   "metadata": {},
   "source": [
    "### Updating Values\n",
    "\n",
    "The `update_from()` method adds new or updated key-value pairs to the stored config, and these are provided as a dictionary via the `src` parameter. \n",
    "<!--cell 92-->"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "id": "7a3f8247-75f6-4cc6-9a9d-7e69896120e7",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[fruit count]\n",
      "apples = 3\n",
      "grapes = 15\n",
      "kiwis = 4\n",
      "raspberries = 12\n"
     ]
    }
   ],
   "source": [
    "fruits.update_from(\n",
    "    src={'fruit count':{'kiwis': '4', 'raspberries': '12'}}\n",
    ")\n",
    "print(fruits)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}