Refactor/load series through schema #1436
Conversation
Signed-off-by: F.N. Claessen <felix@seita.nl>
Signed-off-by: F.N. Claessen <felix@seita.nl>
Signed-off-by: F.N. Claessen <felix@seita.nl>
Signed-off-by: F.N. Claessen <felix@seita.nl>
Signed-off-by: F.N. Claessen <felix@seita.nl>
Signed-off-by: F.N. Claessen <felix@seita.nl>
…stock-commitments
Signed-off-by: F.N. Claessen <felix@seita.nl>
Signed-off-by: F.N. Claessen <felix@seita.nl>
Signed-off-by: F.N. Claessen <felix@seita.nl>
Signed-off-by: F.N. Claessen <felix@seita.nl>
…or resolution Signed-off-by: F.N. Claessen <felix@seita.nl>
Signed-off-by: F.N. Claessen <felix@seita.nl>
Signed-off-by: F.N. Claessen <felix@seita.nl>
Signed-off-by: F.N. Claessen <felix@seita.nl>
…llback attribute Signed-off-by: F.N. Claessen <felix@seita.nl>
…stock-commitments
…stock-commitments
…stock-commitments
Signed-off-by: F.N. Claessen <felix@seita.nl>
…stock-commitments
…stock-commitments
Signed-off-by: F.N. Claessen <felix@seita.nl>
Signed-off-by: F.N. Claessen <felix@seita.nl>
Signed-off-by: F.N. Claessen <felix@seita.nl>
Signed-off-by: F.N. Claessen <felix@seita.nl>
Signed-off-by: F.N. Claessen <felix@seita.nl>
Signed-off-by: F.N. Claessen <felix@seita.nl>
Signed-off-by: F.N. Claessen <felix@seita.nl>
Signed-off-by: F.N. Claessen <felix@seita.nl>
Signed-off-by: F.N. Claessen <felix@seita.nl>
…time series data Signed-off-by: F.N. Claessen <felix@seita.nl>
Signed-off-by: F.N. Claessen <felix@seita.nl>
Signed-off-by: F.N. Claessen <felix@seita.nl>
Signed-off-by: F.N. Claessen <felix@seita.nl>
Signed-off-by: F.N. Claessen <felix@seita.nl>
…_price loading to schema Signed-off-by: F.N. Claessen <felix@seita.nl>
…ce loading to schema Signed-off-by: F.N. Claessen <felix@seita.nl>
Signed-off-by: F.N. Claessen <felix@seita.nl>
Signed-off-by: F.N. Claessen <felix@seita.nl>
Signed-off-by: F.N. Claessen <felix@seita.nl>
…time series Signed-off-by: F.N. Claessen <felix@seita.nl>
…mw and ems_production_capacity_in_mw loading to schema Signed-off-by: F.N. Claessen <felix@seita.nl>
…izing flex-model Signed-off-by: F.N. Claessen <felix@seita.nl>
…city_in_mw and ems_production_capacity_in_mw loading to schema" This reverts commit a3b9811.
…mw and ems_production_capacity_in_mw loading to schema Signed-off-by: F.N. Claessen <felix@seita.nl>
Signed-off-by: F.N. Claessen <felix@seita.nl>
…mw loading to schema Signed-off-by: F.N. Claessen <felix@seita.nl>
…ex-context schema Signed-off-by: F.N. Claessen <felix@seita.nl>
Signed-off-by: F.N. Claessen <felix@seita.nl>
…/load-Series-through-schema # Conflicts: # documentation/api/change_log.rst # flexmeasures/data/models/planning/storage.py # flexmeasures/data/models/planning/tests/test_storage.py # flexmeasures/data/schemas/scheduling/__init__.py
Signed-off-by: F.N. Claessen <felix@seita.nl>
| def _deserialize( | ||
| self, value: dict[str, int] | list[dict] | str, attr, obj, **kwargs | ||
| ) -> Sensor | list[dict] | ur.Quantity: | ||
|
|
||
| if isinstance(value, dict): | ||
| return self._deserialize_dict(value) | ||
| elif isinstance(value, list): | ||
| return self._deserialize_list(value) | ||
| elif isinstance(value, str): | ||
| return self._deserialize_str(value) | ||
| elif isinstance(value, numbers.Real) and self.default_src_unit is not None: | ||
| return self._deserialize_numeric(value, attr, obj, **kwargs) | ||
| else: | ||
| raise FMValidationError( | ||
| f"Unsupported value type. `{type(value)}` was provided but only dict, list and str are supported." | ||
| ) -> Sensor | list[dict] | ur.Quantity | timely_beliefs.BeliefsSeries: |
There was a problem hiding this comment.
@nhoening Need advice.
I've basically repurposed each VariableQuantityField in the FlexContextSchema to (upon deserialization) either:
- Do what it did before, so either:
- Turn a string into a Quantity
- Turn a sensor reference (a dict) into a Sensor
- Turn a time series specification (a list of dicts) into a list of timed events (still a list of dicts)
- Or do a new thing, which is to turn any of the previous outputs (Quantity, Sensor or list of timed events) into a
pd.Series.
I don't particularly like this mixing of two deserialization steps in a single Field, but I do like having a single place where each field is defined, incl. all the steps needed to get to a pd.Series.
When we create a scheduling job, we're only doing step (1) above (so some validation, but not loading any sensor data from the db). When the job is run, we do both steps.
I could clean the current approach up a bit, of course, but if you have a suggestion on taking a different direction, I'm open to that.
There was a problem hiding this comment.
I really like the outcome of this PR and I agree with you in that it doesn't feel right to mix two deserialization approaches. It would be nice to have an alternative deserealization method but, of course, it would require too many changes.
A small improvement could be to move the two deserealization to different methods and just switch between them in def _deserialize(...).
Perhaps we should create a new field as the two deserealization point to two different resulting objects:
- A union of types ➡️ Quantity | Sensor | TimeSeries
- A TimeSeries generator: potentially incomplete description of a time series.
After all, some of the original descriptions (i.e. Quantity and Sensor) require some context to become proper TimeSeries (start, end, resolution, etc...). Later, we transform data to the unit/resolution we find convenient.
| ) | ||
|
|
||
|
|
||
| def series_range_validator(min=None, max=None): |
There was a problem hiding this comment.
Perhaps the name of the function should say that this is returning a validator, something like get_series...
| def _deserialize( | ||
| self, value: dict[str, int] | list[dict] | str, attr, obj, **kwargs | ||
| ) -> Sensor | list[dict] | ur.Quantity: | ||
|
|
||
| if isinstance(value, dict): | ||
| return self._deserialize_dict(value) | ||
| elif isinstance(value, list): | ||
| return self._deserialize_list(value) | ||
| elif isinstance(value, str): | ||
| return self._deserialize_str(value) | ||
| elif isinstance(value, numbers.Real) and self.default_src_unit is not None: | ||
| return self._deserialize_numeric(value, attr, obj, **kwargs) | ||
| else: | ||
| raise FMValidationError( | ||
| f"Unsupported value type. `{type(value)}` was provided but only dict, list and str are supported." | ||
| ) -> Sensor | list[dict] | ur.Quantity | timely_beliefs.BeliefsSeries: |
There was a problem hiding this comment.
I really like the outcome of this PR and I agree with you in that it doesn't feel right to mix two deserialization approaches. It would be nice to have an alternative deserealization method but, of course, it would require too many changes.
A small improvement could be to move the two deserealization to different methods and just switch between them in def _deserialize(...).
Perhaps we should create a new field as the two deserealization point to two different resulting objects:
- A union of types ➡️ Quantity | Sensor | TimeSeries
- A TimeSeries generator: potentially incomplete description of a time series.
After all, some of the original descriptions (i.e. Quantity and Sensor) require some context to become proper TimeSeries (start, end, resolution, etc...). Later, we transform data to the unit/resolution we find convenient.
Description
documentation/changelog.rstLook & Feel
Internal refactoring only.