Modularisation: the Submodel concept

[tyralla]

We currently aim to support LARSIM-like simulations with HydPy-L that better take infiltration excess into account via a Green & Ampt (GA) infiltration approach. However, HydPy-L is already very complex, and a GA approach suitable for continuous simulations is not too simple either. So, combining the equations of both models is something we should avoid. Instead, we strive for an independently usable GA model that allows for coupling with HydPy-L (and, at best, other "complete" hydrological models like HydPy-H). Unfortunately, this is beyond HydPy's current modularisation functionalities. Extending these functionalities will be non-trivial, but we think it will pay off very soon due to ways to keep our application models smaller and increase the users' flexibility in combining different model modules.

The necessary work will touch on many topics and require many decisions. Hence, we split the discussion into three issues that separately deal with the GA methodology itself (#89), the general approach to couple "complete" application models like HydPy-L with "submodels" like GA (#90), and the specific coupling interface for HydPy-L and GA (#91).

[tyralla]

The application model ga_garto is now implemented as a stand-alone model. We discussed the general aspects of how to couple it with other models before starting with ga_garto. I hope the following short list contains all relevant discussed points:

• Technically speaking, we follow the strategy pattern.
• We speak of "submodels", a term that conflicts with the existing Submodel class. However, the new submodel concept we target here is far more relevant from the users' perspective. So we decided to leave this name conflict for now as is. Maybe, we need to rename the existing Submodel class and the related documentation later.
• When speaking about submodels, "main model" would mean a model controlling a submodel. I think we need this term just for the documentation, not for the source code.
• Principally, a submodel can control another submodel and so be a main model and a submodel simultaneously. We do not put any effort into explicitly supporting this, as we are unsure if it will ever be required. However, we also do not implement things in an unnecessarily restricting way.
• It should be possible to define the main model and its submodel(s) within a single control file and its initial conditions in a single condition file. Our first idea is to use separate "with" blocks for individual models, which should result in a readable design and help deal with possible name conflicts for the parameters or sequences of the different models involved.
• We still need to think about shared information. For example, HydPy-L and HydPy-GA both require soil depth data. For now, users must define it twice, but (if the submodel concept proves successful), we should somehow circumvent this redundancy later.
• We postpone any discussion on configuring submodels via the XML support for now.
• There is no way around clear interfaces here. And these must be implemented both in Python and (manually or automatically?) in Cython. This should be the first (and possibly hardest) point to solve.

[tyralla]

At the time of writing, #91 defines a SoilInterface prototype semantically. Here, we discuss the general technical implementation of this and other submodel interfaces and their concrete realisations.

Like for "normal" models, we always require two implementations: pure Python classes and Cython extension classes. At least, we should support the automatic generation of Cython realisations based on Python code. Preferably, we should also support the automatic generation of Cython interfaces based on Python code. I start with the optimistic assumption complete automatisation is possible.

[tyralla]

If fully automatic generation is possible, we do not need to write any pyx or pxd files manually. Hence, there is no good reason for defining interfaces in the cythons subpackage. I would prefer to create a new subpackage, called interfaces, for this purpose. That should also help to keep the package structure and the documentation structure in sync. (For users, knowing the available interfaces is much more relevant than the very specific functionalities implemented in the cythons subpackage.)

[tyralla]

Within the interfaces subpackage, there could be modules for different "topics" (for now, only soil.py). The assignment of each interface to a single topic may not always be clear. Still, it currently seems better than collecting all interfaces in a single module (in case the submodel concept is successful and we will implement many interfaces later).

[tyralla]

I prefer to write "real" python modules (.py) of over stub files (.pyi) because we want to load them at least for converting their interfaces to Cython.

[tyralla]

Reusing the "normal" model conversion mechanisms seems favourable. So far, we have the interfaces subpackage including a soil module defining the following interface:

class SoilInterface:
    def set_initial_surface_water(self, k: int, v: float) -> None:
        ...

    def set_actual_surface_water(self, k: int, v: float) -> None:
        ...

    def set_soil_water_supply(self, k: int, v: float) -> None:
        ...

    def set_soil_water_demand(self, k: int, v: float) -> None:
        ...

    def execute_infiltration(self, k: int) -> None:
        ...

    def add_soil_water(self, k: int) -> None:
        ...

    def remove_soil_water(self, k: int) -> None:
        ...

    def get_infiltration(self, k: int) -> float:
        ...

    def get_percolation(self, k: int) -> float:
        ...

    def get_soil_water_addition(self, k: int) -> float:
        ...

    def get_soil_water_removal(self, k: int) -> float:
        ...

    def get_soil_water_content(self, k: int) -> float:
        ...

Speaking of the pure Python mode, we just need to define the corresponding methods in the usual way (derived from modeltools.Method) and select them as "additional methods" (ADD_METHODS). However, adding a new tuple for selecting methods should be the tidier solution if going this route. Therefore, I suggest INTERFACE_METHODS.

Following this idea, we would need to create a GARTO-like GA submodel designed to follow a specific interface. But it is not directly clear whether such a submodel follows the interface (no inheritance). Theoretically, we could think about making the interface class a Protocol. At least, this would hint at its actual usage, but I am unsure if the protocol mechanism (issubclass, issintance) works for cythonized methods.

Maybe we can should a mandatory marker for each interface. We could name our first soil interface SoilInterface1 instead of SoilInterface (perhaps a good idea anyhow) and define it so:

from typing import Literal
class SoilInterface1:
    mark: Literal[1]

Then, the main model would require no isinstance check the identify the interface supported by the model selected by the user. But, on the downside, it requires casting in Cython (I am confident we can automate it quickly), and each application model would relate to a single interface only, even if providing enough methods for supporting multiple interfaces.

Maybe the last point is more an advantage than a disadvantage because it could prevent using submodels in cases that work technically but are hydrologically misleading. A GARTO submodel would, for example, also satisfy an interface that does not require calculating percolation. But GARTO calculates percolation in any case. So if the main model does not consider this calculated percolation, it will likely make serious (water balance) errors.

The code from the main model's perspective could then look like this:

SupportedSoilInterfaces: Literal[SoilInterface1, SoilInterface2]

model.soilmodel: SupportedSoilInterfaces

if model.soilmodel is None:
    model.apply_standard_soil_routine()
elif model.soilmodel.mark== 1:
    model.apply_soilsubmodel1()
elif model.soilmodel.mark == 2:
   model.apply_soilsubmodel2()
else:
    assert_never(model.soilmodel.mark)

Hence, we need to declare somewhere for each model which submodel interfaces it supports, which is also helpful for users. The assert_never approach helps to update the main model's code as soon as we manage to type-check our models with Mypy. In Cython, we can eliminate the assert_never line because we should check that a given submodel follows a supported interface already during initialisation. If (always) providing a standard routine as a fallback is a good idea, I am not sure.

Converting the Python implementation of an interface into a Cython implementation (.pxd) should pose no big problem. If we somehow connect a cythonized submodel with such a cythonized interface, we hopefully get type-checking for free (not during programming but at least during compiling). I have to try, but I guess that simple inheritance should work here. Of course, we must ensure the interfaces are cythonized first, even when starting cythonizing a single submodel from the interpreter.

[tyralla]

After discussing the details of the current proposal: Maybe we can avoid the "marks" and use the standard isinstance approach instead. I do not know what type-checkers and linters say, but we could make the interface's methods abstract without making the interface an abstract class and add the class to the submodel's inheritance tree. The abstract methods would be "replaced" by concrete Python- or Cython-methods during model instantiation.

from abc import abstractmethod

class SoilInterface:
    @abstractmethod
    def set_initial_surface_water(self, k: int, v: float) -> None:
        ...

SupportedSoilInterfaces: Literal[SoilInterface1, SoilInterface2]

model.soilmodel: Optional[SupportedSoilInterfaces]

if model.soilmodel is None:
    model.apply_standard_soil_routine()
if isinstance(model.soilmodel, SoilInterface1):
    model.apply_soilsubmodel1()
if isinstance(model.soilmodel, SoilInterface2):
   model.apply_soilsubmodel2()
else:
    assert_never(model.soilmodel)

It looks similar but a little cleaner and would allow further checks (e.g. in apply_soilsubmodel1) to differentiate between potential sub-interfaces (e.g. SoilInterface1A and SoilInterface1B) or so if such a subclassing makes sense hydrologically (we will see).

However, we need to remember not to inherit interface SoilInterface2 from interface SoilInterface1, even if all methods of SoilInterface1 are also defined by SoilInterface2, but both are hydrologically incompatible, as discussed above.

[tyralla]

Base model lland provides a "switch method" Calc_BoWa_V1, its default implementation Calc_BoWa_Default_V1, and (so far) the handler of the interface Calc_BoWa_SoilInterface1_V1. Application model lland_v2 selects all these methods.

Base model ga provides the different interface methods (e.g. Set_InitialSurfaceWater_V1). The submodel (I would say, submodels are always application models) ga_garto_sub1 selects the appropriate (currently all) interface methods.

How much testing is necessary from the hydrological perspective, and where do we do it?

• Calc_BoWa_SoilInterface1_V1
  • Detailed testing that lland uses the specific interface correctly (by selecting a random submodel following this interface).
  • At least, we need to clarify we keep the water balance under all circumstances.
• lland_v2
  • Short demonstration of how to use a submodel following this interface (by selecting a random submodel following this interface).
• ga_garto_sub1
  • Semi-detailed testing by repeating the most striking examples of the related stand-alone model ga_garto (by selecting a random main model using its interface).

[tyralla]

After discussing the details of the current proposal: Maybe we can avoid the "marks" and use the standard isinstance approach instead. I do not know what type-checkers and linters say, but we could make the interface's methods abstract without making the interface an abstract class and add the class to the submodel's inheritance tree. The abstract methods would be "replaced" by concrete Python- or Cython-methods during model instantiation.

Unfortunately, I cannot get the prefered isinstance approach running. Cython generally supports it, but not when releasing Python's Global Interpreter Lock (GIL) via the nogil option, which we still want to do for the sake of not risking unnecessary speed bottlenecks and not blocking later (potential) parallelisation efforts.

Hence, the first lines of our Python interface now look like this:

class SoilModel_V1(modeltools.SubmodelInterface):

    typeid: Literal[1] = 1

    @abstractmethod
    def set_initialsurfacewater(self, k: int, v: float) -> None:
        ...

I now use typeid instead of mark, which seems clearer. By convention, the integer value of typeid (1) corresponds to the "version number" of the interface ("V1").

In Python, ga_garto_sub1 inherits from this soil model interface:

class Model(modeltools.AdHocModel, soilinterfaces.SoilModel_V1):
    """The GARTO algorithm (assuming a hydrostatic groundwater table), implemented as
    a submodel meeting the requirements of the |SoilModel_V1| interface."""
    ....

In Cython, we derive the application model from the (cythonized) interface only:

@cython.final
cdef class Model(soilinterfaces.SoilModel_V1):
    cdef public int idx_sim
    ...

The base extension class is now defined in two modules. First, in a pxd module (for declaring the types):

cdef class SoilModel_V1:
    cdef numpy.int32_t typeid
    cdef void add_soilwater(self, numpy.int32_t k) nogil
    ...

Second, in a pyx module (for implementing the implementation):

cdef class SoilModel_V1:
    cdef void add_soilwater(self, numpy.int32_t k) nogil:
        pass
    ...
    cdef double get_infiltration(self, numpy.int32_t k) nogil:
        return 0.0
    ...
    def __init__(self):
        self.typeid = 1

I don't like that we define useless default method implementations in pyx, but I have no idea how to circumvent it. However, it's no real problem, as they are automatically generated and users won't be able to access them (accessible via C only).

[tyralla]

Each submodel inherits from a single interface only and selects the suitable methods via the new class attribute INTERFACE_METHODS:

class Model(modeltools.AdHocModel, soilinterfaces.SoilModel_V1):
    """The GARTO algorithm (assuming a hydrostatic groundwater table), implemented as
    a submodel meeting the requirements of the |SoilModel_V1| interface."""
    
    INLET_METHODS = ()
    RECEIVER_METHODS = ()
    RUN_METHODS = (ga_model.Perform_GARTO_V1,)
    INTERFACE_METHODS = (
        ga_model.Set_InitialSurfaceWater_V1,
        ga_model.Set_ActualSurfaceWater_V1,
        ga_model.Set_SoilWaterDemand_V1,
        ga_model.Execute_Infiltration_V1,
        ga_model.Remove_SoilWater_V1,
        ga_model.Get_Percolation_V1,
        ga_model.Get_Infiltration_V1,
        ga_model.Get_SoilWaterRemoval_V1,
        ga_model.Get_SoilWaterContent_V1,
    )
    ADD_METHODS = (
        ga_model.Return_RelativeMoisture_V1,
    ...

I make INTERFACE_METHODS a member of modeltools.SubmodelInterface so that models which cannot be submodels do not need to define empty INTERFACE_METHODS tuples.

Main models which support using specific submodel interfaces declare so by selecting them under the new class attribute SUBMODELINTERFACES:

class Model(modeltools.AdHocModel):
    """Base model for HydPy-L-Land."""
    ...
    SUBMODELINTERFACES = (soilinterfaces.SoilModel_V1,)
    ...

Similar to the usual "method selection" mechanism, such an "interface selection" implies there is an instance attribute named submodel (which we generate automatically and set to None during model initialisation). Assume the following situation:

class Model(modeltools.AdHocModel):
    ...
    SUBMODELINTERFACES = (
        soilinterfaces.SoilModel_V1,
        soilinterfaces.SoilModel_V3,
        soilinterfaces.InterceptionModel_V1,
    )

For such a setting, the main model would have an attributed named soilmodel that allows submodels following the interfaces SoilModel_V1 and SoilModel_V3 and an attribute named interceptionmodel that must follow InterceptionModel_V1 (or be set to None).

For documentation and consistency checking purposes, the methods relying on SoilModel_V1 should also select this class under the new (optional) class attribute SUBMODELINTERFACES.