Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
158 changes: 158 additions & 0 deletions doc/source/composites.rst
Original file line number Diff line number Diff line change
Expand Up @@ -495,6 +495,164 @@ image) for both of the static images::
min_stretch: [0, 0, 0]
max_stretch: [255, 255, 255]

.. _composite_variants:

Composite variants
------------------

.. versionadded:: 0.60

Satpy supports defining multiple *variants* of a composite (e.g., one that
applies WMO-recommended recipe and one that does not). This feature is controlled by optional
fields in the composite YAML configuration.

Tagging composite variants
^^^^^^^^^^^^^^^^^^^^^^^^^^

A composite can carry a list of **tags** that describe which processing variant
it represents. Tags are plain strings (e.g., ``wmo``, ``crefl``,
``nocorr``) and have no special meaning to Satpy beyond being matched during
compositor lookup.

.. code-block:: yaml

sensor_name: visir

composites:
true_color_wmo:
compositor: !!python/name:satpy.composites.SomeCompositor
prerequisites:
- name: red
modifiers: [rayleigh_corrected_wmo]
- name: green
- name: blue
standard_name: true_color
tags: [wmo]

true_color_crefl:
compositor: !!python/name:satpy.composites.SomeCompositor
prerequisites:
- name: red
modifiers: [rayleigh_corrected_crefl]
- name: green
- name: blue
standard_name: true_color
tags: [crefl]

true_color: # default, no tags
compositor: !!python/name:satpy.composites.SomeCompositor
prerequisites:
- name: red
- name: green
- name: blue
standard_name: true_color

The ``standard_name`` field acts as the *base name* shared by all variants.
Tag-based resolution searches for a compositor whose ``standard_name`` matches
and whose ``tags`` list contains the requested tag.

Loading a tagged variant explicitly
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Append ``:<tag>`` to the composite name when calling :meth:`~satpy.scene.Scene.load`:

.. code-block:: python

scene.load(["true_color:wmo"]) # loads true_color_wmo

This syntax is interpreted as: "find a compositor with
``standard_name='true_color'`` that has ``'wmo'`` in its ``tags``". It never
performs a plain string match, so ``true_color:wmo`` will not accidentally
resolve to a compositor named ``true_color`` or ``true_color_wmo``.

Multiple tags can be combined with additional colons. All listed tags must be
present on the compositor (AND semantics):

.. code-block:: python

scene.load(["true_color:wmo:pyspectral"]) # compositor must carry wmo AND pyspectral
scene.load(["true_color:wmo"]) # also matches a compositor with tags [wmo, pyspectral]

A single-tag request is a subset match, so requesting ``"true_color:wmo"``
will find a compositor tagged ``["wmo", "pyspectral"]`` just as readily as one
tagged ``["wmo"]`` alone.

Session-wide tag preferences
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

You can configure an **ordered** list of preferred tags so that loading an
unqualified name like ``"true_color"`` automatically selects a tagged variant
when one is available:

.. code-block:: python

import satpy
with satpy.config.set(preferred_composite_tags=["crefl", "wmo"]):
scene.load(["true_color"]) # picks true_color_crefl (crefl listed first)

Resolution order for ``preferred_composite_tags``:

1. Try each tag in the list, in order, looking for a compositor with matching
``standard_name`` and that tag.
2. If no tagged variant is found, fall back to the normal name-based lookup.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Which variant is used if the tag is not defined in the load and the global env variable is not set and all/some of the following composites are defined?

  • foo_wmo
  • foo_crefl
  • foo

Each of them have the same standard_name: foo defined, no name is set. Only the first line in the YAML definition is different and for the first two there are tags set.

So what would I get with scn.load(["foo"]) call in the following cases:

  1. all variants are defined (I'd expect foo variant without any tags)
  2. only the two tagged versions are defined

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If tags are not provided, neither at load time or in the configuration, the current behaviour is preserved, ie we will use the name (not standard name) to choose the composite to load.

  1. you get the composite name "foo"
  2. crash if you don't provide a tag (at load time or as config).

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So something like this for the first case? I'm not sure how this works, but hopefully it's close enough:

        pytest.param(
            {"comp1": None, "comp1_wmo": ["wmo"]},
            None,
            "comp1",
            "comp1",
            id="use_plain_version_when_no_tag",
        ),

For case 2. the failure might need a completely separate test?


An explicit ``name:tag`` in the load call always overrides the session-wide
preference for that specific dataset.

The setting can also be provided as an environment variable (comma-separated):

.. code-block:: bash

export SATPY_PREFERRED_COMPOSITE_TAGS=crefl,wmo

Or as a YAML configuration key:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In which config file should this be defined?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

at the top of this file, you have your answer :)

YAML Configuration

YAML files that include these parameters can be in any of the following
locations:

  1. <python environment prefix>/etc/satpy/satpy.yaml
  2. <user_config_dir>/satpy.yaml (see below)
  3. ~/.satpy/satpy.yaml
  4. <SATPY_CONFIG_PATH>/satpy.yaml (see :ref:config_path_setting below)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sorry, it was actually in config.rst

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, I didn't even know we had a satpy.yaml config file 😅


.. code-block:: yaml

preferred_composite_tags:
- crefl
- wmo

Enhancements for tagged variants
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

No extra configuration is needed on the enhancement side. The enhancement
decision tree already matches by composite ``name`` (the YAML key) *before*
it falls back to ``standard_name``. A composite named ``true_color_wmo`` will
therefore automatically pick up an enhancement entry keyed ``true_color_wmo``
if one exists, and fall back to the ``standard_name: true_color`` entry
otherwise.

Deprecating a composite
------------------------

To emit a warning when a composite is used, add a ``warnings`` mapping to the
composite YAML entry. Each key is a Python warning category name and the
value is the warning message:

.. code-block:: yaml

composites:
old_true_color:
compositor: !!python/name:satpy.composites.SomeCompositor
prerequisites:
- name: red
- name: green
- name: blue
standard_name: true_color
warnings:
DeprecationWarning: "old_true_color is deprecated, use true_color_wmo instead."

The warning is emitted only when the compositor is actually *loaded* (i.e.
when :meth:`~satpy.scene.Scene.load` is called with the deprecated name), not
during composite discovery calls such as
:meth:`~satpy.scene.Scene.available_dataset_names`.

Supported warning categories are any that exist in the Python ``builtins``
module (e.g., ``DeprecationWarning``, ``FutureWarning``,
``PendingDeprecationWarning``, ``UserWarning``). If a category name is not
recognised, ``UserWarning`` is used as a fallback.

.. _enhancing-the-images:

Enhancing the images
Expand Down
40 changes: 40 additions & 0 deletions doc/source/config.rst
Original file line number Diff line number Diff line change
Expand Up @@ -279,6 +279,46 @@ Clipping of negative radiances is currently implemented for the following reader
* ``abi_l1b``, ``ami_l1b``, ``fci_l1c_nc``


Preferred Composite Tags
^^^^^^^^^^^^^^^^^^^^^^^^

* **Environment variable**: ``SATPY_PREFERRED_COMPOSITE_TAGS``
* **YAML/Config Key**: ``preferred_composite_tags``
* **Default**: ``[]``

Ordered list of composite variant tags that Satpy should prefer when resolving
an unqualified composite name. When a user requests a composite such as
``"true_color"`` and this list is non-empty, Satpy will first search for a
compositor whose ``standard_name`` matches and whose ``tags`` list contains
the first tag in the preference list, then the second tag, and so on. If no
tagged variant is found the normal name-based lookup is used as a fallback.

For example, to prefer Pyspectral-based variants:

.. code-block:: python

import satpy
satpy.config.set(preferred_composite_tags=["pyspectral"])

Or to prefer CREFL Rayleigh correction over Pyspectral:

.. code-block:: python

satpy.config.set(preferred_composite_tags=["crefl", "pyspectral"])

An explicit ``name:tag`` syntax in the ``scene.load()`` call always overrides
this setting for that specific dataset.

When setting this as an environment variable, it should be a comma-separated
list of tag names, for example:

.. code-block:: bash

export SATPY_PREFERRED_COMPOSITE_TAGS=crefl,pyspectral

See :ref:`composite_variants` for a full description of
composite tagging and the ``name:tag`` load syntax.

Temporary Directory
^^^^^^^^^^^^^^^^^^^

Expand Down
83 changes: 76 additions & 7 deletions satpy/dependency_tree.py
Original file line number Diff line number Diff line change
Expand Up @@ -19,10 +19,13 @@

from __future__ import annotations

import builtins
import warnings
from typing import Container, Iterable, Optional

import numpy as np

import satpy
from satpy import DataID, DatasetDict
from satpy.dataset import ModifierTuple, create_filtered_query
from satpy.dataset.data_dict import TooManyResults, get_key
Expand Down Expand Up @@ -492,14 +495,80 @@ def _promote_query_to_modified_dataid(self, query, dep_key):
return dep_key.from_dict(orig_dict)

def get_compositor(self, key):
"""Get a compositor."""
for sensor_name in sorted(self.compositors):
try:
return self.compositors[sensor_name][key]
except KeyError:
continue
"""Get a compositor.

Resolves in order:
1. Tag-based: explicit ``name:tag`` syntax or ``preferred_composite_tags`` config.
2. Normal name-based lookup in the compositor registry.

If the compositor has a ``warnings`` attribute dict, those warnings are emitted here.
"""
compositor = self._get_compositor_by_tag(key)
if compositor is None:
for sensor_name in sorted(self.compositors):
try:
compositor = self.compositors[sensor_name][key]
break
except KeyError:
continue

if compositor is None:
raise KeyError("Could not find compositor '{}'".format(key))

raise KeyError("Could not find compositor '{}'".format(key))
self._emit_compositor_warnings(compositor)
return compositor

@staticmethod
def _emit_compositor_warnings(compositor):
for category_name, message in compositor.attrs.get("warnings", {}).items():
category = getattr(builtins, category_name, UserWarning)
warnings.warn(message, category, stacklevel=4)

def _get_compositor_by_tag(self, key):
"""Find a compositor by tag syntax (``'name:tag1:tag2'``) or ``preferred_composite_tags`` config.

For explicit tag syntax the returned compositor must carry all listed tags; among
multiple matches the ``preferred_composite_tags`` config is used as a tiebreaker,
falling back to the first candidate in alphabetical sensor order.

For plain names (no colon) each entry in ``preferred_composite_tags`` is tried in
order; ``None`` is returned when none match so that normal name-based lookup can
proceed.
"""
try:
name = key["name"]
except (KeyError, TypeError):
return None
if not isinstance(name, str):
return None

parts = name.split(":")
standard_name, required_tags = parts[0], set(parts[1:])
candidates = self._find_tag_candidates(standard_name, required_tags)
preferred = self._pick_preferred_candidate(candidates)
if preferred is not None:
return preferred
# Explicit-tag requests fall back to the first candidate; plain-name requests
# return None so that normal name-based lookup can proceed.
return candidates[0] if (required_tags and candidates) else None

def _find_tag_candidates(self, standard_name, required_tags):
"""Return compositors whose standard_name matches and that carry all required_tags."""
return [
comp
for sensor_name in sorted(self.compositors)
for comp in self.compositors[sensor_name].values()
if comp.attrs.get("standard_name") == standard_name
and required_tags.issubset(set(comp.attrs.get("tags", [])))
]

def _pick_preferred_candidate(self, candidates):
"""Return the first candidate that matches any preferred_composite_tags entry, in order."""
for tag in satpy.config.get("preferred_composite_tags", []):
for comp in candidates:
if tag in comp.attrs.get("tags", []):
return comp
return None

def get_modifier(self, comp_id):
"""Get a modifer."""
Expand Down
72 changes: 72 additions & 0 deletions satpy/tests/compositor_tests/test_config_loader.py
Original file line number Diff line number Diff line change
Expand Up @@ -56,3 +56,75 @@ def _create_fake_composite_config(yaml_filename: str):
},
comp_file,
)


def test_composite_warning_is_emitted_on_compositor_use(tmp_path):
"""Test that composite 'warnings' fires when the compositor is fetched, not when configs are loaded."""
import pytest

from satpy.composites.config_loader import load_compositor_configs_for_sensors
from satpy.dataset import DataQuery
from satpy.dependency_tree import DependencyTree

comp_dir = tmp_path / "composites"
comp_dir.mkdir()
_create_fake_composite_config_with_warnings(comp_dir / "fake_sensor.yaml")

with satpy.config.set(config_path=[tmp_path]):
comps, _ = load_compositor_configs_for_sensors(["fake_sensor"]) # no warning here

tree = DependencyTree({}, comps, {})
with pytest.warns(DeprecationWarning, match="Use new_composite instead"):
tree.get_compositor(DataQuery(name="old_composite"))


def _create_fake_composite_config_with_warnings(yaml_filename):
import yaml

from satpy.composites.aux_data import StaticImageCompositor

with open(yaml_filename, "w") as comp_file:
yaml.dump({
"sensor_name": "fake_sensor",
"composites": {
"old_composite": {
"compositor": StaticImageCompositor,
"url": "http://example.com/image.png",
"warnings": {"DeprecationWarning": "Use new_composite instead"},
},
},
}, comp_file)


def test_composite_tags_stored_on_compositor(tmp_path):
"""Test that a composite with 'tags' in its YAML has those tags stored in its attrs."""
from satpy.composites.config_loader import load_compositor_configs_for_sensors

comp_dir = tmp_path / "composites"
comp_dir.mkdir()
comp_yaml = comp_dir / "fake_sensor.yaml"
_create_fake_composite_config_with_tags(comp_yaml, tags=["wmo"])

with satpy.config.set(config_path=[tmp_path]):
comps, _ = load_compositor_configs_for_sensors(["fake_sensor"])

compositor = next(iter(comps["fake_sensor"].values()))
assert compositor.attrs.get("tags") == ["wmo"]


def _create_fake_composite_config_with_tags(yaml_filename, tags):
import yaml

from satpy.composites.aux_data import StaticImageCompositor

with open(yaml_filename, "w") as comp_file:
yaml.dump({
"sensor_name": "fake_sensor",
"composites": {
"tagged_composite": {
"compositor": StaticImageCompositor,
"url": "http://example.com/image.png",
"tags": tags,
},
},
}, comp_file)
Loading
Loading