Link to data object

[vgro]

In our group meeting yesterday, we once more discussed the need for a central 'data' object that is accessible for all modules and stores all relevant variables that go in and out of the modules (and between modules). I would like to discuss the mechanism of accessing and updating this object. As mentioned yesterday, it is crucial that processes in different modules and updating of the data object happen in a coordinated way and that the correct time step is accessed; at the same time we want to store as little data as possible. (For those who did not attend the meeting yesterday, we currently use different dummy versions of the data object to develop the modules which is not ideal.)
So the question is: Should each module load the whole data object, do it's thing for the current time step, update the data object and return the info to a central location? How does it work that it is pushed back to where it is accessible for everyone?

[davidorme]

I've got a sketch in my head for this. At the moment the BaseModel class has an __init__ that looks like this:

    def __init__(
        self, update_interval: timedelta64, start_time: datetime64, **kwargs: Any
    ):
        self.update_interval = update_interval
        self.next_update = start_time + update_interval
        # Save variables names to be used by the __repr__
        self._repr = ["update_interval", "next_update"]

I think that is altered to:

```python
    def __init__(
        self, data: Data, update_interval: timedelta64, start_time: datetime64, **kwargs: Any
    ):
        self.data = data
        self.update_interval = update_interval
        self.next_update = start_time + update_interval
        # Save variables names to be used by the __repr__
        self._repr = ["update_interval", "next_update"]

So the flow of the program is:

• load the config
• create a Data instance and populate from the config.
• create the models, all which then get passed a reference to that Data object when they are created.
• all of the models can now use self.data['varname'] to access data and self.data['newvarname'] = ... to make their own variables available to other models and for export.

One other thing we discussed at various points was having a way of declaring required variables for initialising a model. The idea I had for this is making a class variable to hold a list of variable names that have to be present and aligned onto the right core axes in Data before anything can start. Obviously the specific model __init__ might do any amount of further inspection of the initial data, but having a single shared function to do a baseline check seems useful.

As a sketch:

class BaseModel(ABC):
    """A superclass for all `vr` models.
    Describes the common functions and attributes that all `vr` models should have. This
    includes functions to setup, spin up and update the specific model, as well as a
    function to cleanup redundant model data. At this level these functions are not
    define and are mere placeholders to be overwritten (where appropriate) by the
    inheriting classes.
    Args:
        start_time: Point in time that the model simulation should be started.
        end_time: Time that the model simulation should end
        update_interval: Time to wait between updates of the model state.
    Attributes:
        name: Names the model that is described
        required_variables: A tuple of required variable names and the core axes 
            they must match. For example:  ``(('temperature', ('spatial', 'temporal')))``
    """

    name = "base"
    required_variables: tuple[tuple[str, tuple[str]]] = tuple()

    def __init__(
        data: Data, update_interval: timedelta64, start_time: datetime64, **kwargs: Any
    ):
        self.data = data
        self.check_required_variables()
        self.update_interval = update_interval
        self.next_update = start_time + update_interval
        # Save variables names to be used by the __repr__
        self._repr = ["update_interval", "next_update"]

    def check_required_variables(self):
        """A docstring"""
        all_required = True
        for varname, core_axes in self.required_variables:
            if not varname in self.data:
                LOGGER.error(f"{self.name} model missing required variable: {varname}")
                all_required = False
                continue

            for axname in core_axes:
                if not self.data.on_core_axis(varname, axname):
                    LOGGER.error(
                        f"{self.name} model: variable {varname} not on core axis {axname}"
                    )
                    all_required = False

        if not all_required:
            LOGGER.critical(f"{self.name} model missing required variable: {varname}")
            raise ConfigurationError(
                f"Required data not correctly configured for {self.name} model"
            )

[vgro]

Is the first example the blueprint for the full VR model or for the individual modules(plants, abiotic,..;)? If the latter, I don't fully understand how this ensures there being only one data object that all modules access when it's their turn, it looks to me like each module initializes it's own data object?

[davidorme]

So the docs that @jacobcook1995 is writing at the moment will clarify this. The definition of the AbioticModel would use the BaseModel as a template and would use and extend the same __init__. So as a sketch you could have:

data = Data()
data.load_from_config()

abiotic_model = AbioticModel(data, update_interval, start_time, abiotic_arg1, abiotic_arg2)

The other models would do the same and so everything points to one Data store. The command line use is set up to create those models from the config in a more flexible way, but that example would be perfectly valid code.

[davidorme]

Just to be clear, BaseModel is the template for the individual models. It defines the interface by which all the individual models communicate with the core system and timing loop.

[jacobcook1995]

Not completely up on how class inheritance works under the hood in python, but would this actually ensure that the data object is common to all inherited classes?

Like the shared __init__ function ensures that the same processing of input is followed across models, but different inputs could be provided for different models and there wouldn't be a conflict.

I guess it's fine so long as we provide the same data object to all models when they are created and crucially that we are sure that a shallow copy of the data object has been made, and that this remains a shallow copy when modifications are made

[davidorme]

It isn't even a copy - its a reference to the exact same object:
https://realpython.com/python-pass-by-reference/

An instance of the BaseModel ABC is never actually created - it is just a template - so that the contents of self.data are not passed among actual model subclasses. Instead, a given Data instance is attached to self.data when any real model deriving from BaseModel is created . We could create the models with individual Data instances but if we pass them all the same object, then all the models are going to be looking at the same data store.

If this is not the case (@dalonsoa , @alexdewar ) then I have utterly dropped the ball on the core development 😰

[davidorme]

To be clear, the __init__ for the derived class has to include (as your soil model does):

super().__init__(data, update_interval, start_time, **kwargs)

Essentially that is saying "now run all the stuff from BaseModel.__init__" which includes setting self.data = data.

[davidorme]

Ooops - didn't reply in the right place.

[jacobcook1995]

Ahh right so something about the setup of Data means that it is automatically passed by reference? Because self.next_update can vary between models despite being in the ABC __init__. (or if it can't the timing loop has a fairly major problem 😅)

[davidorme]

Actually, this might not technically be passing 'by reference'. I think the distinction is that Data is mutable and so its contents can be altered globally from within a function. As in:

In [31]: x = []
In [32]: def func(x):
    ...:     x.append([1])
    ...: 
In [33]: func(x)
In [34]: x
Out[34]: [[1]]

I remember that feature surprising people earlier.

[alexdewar]

It is passing by reference. As you say @davidorme, it is a mutable reference -- but then again you don't have immutable references in Python. Basically everything is mutable in Python. (You can make something that looks like an immutable variable with Python @propertys, but mutability isn't a property of variables in the way it is for e.g. C.)

@jacobcook1995 Objects are always passed by reference in Python (and this also includes dicts, lists etc.), so there's nothing special in Data's implementation. If you want to clone your object (so you can effectively pass-by-value) then you need to do a deep copy: https://docs.python.org/3/library/copy.html

[dalonsoa]

As Alex says, the object will be passed by reference, so there will be no copy of information.

I think that the key aspect is that of ownership of the Data object. If it is assigned as the attribute of half a dozen models, where it is actually living? Objects in Python are kept alive for as long as there is a reference to them. I am not sure if this aspect will matter much in this particular workflow, but I think it will be cleaner if Data is passed to the models using weak references, leaving the ownership of the object to the caller.

Here you have a detailed description of a use case for this and how to implement it: https://docs.python.org/3/library/weakref.html

[davidorme]

I think the flow would always be:

• Create a Data object
• Pass references to it into different models
• Stuff happens
• Models finish
• Data is shutdown, exporting any required variables.

So the single Data instance will always be owned by the root process (not sure of the terminology there!).