Added ML Model Handling and Prophet model support

[dominiquekleeven]

Closes #8

---

PR adds the following:

• Generic configuration schema for ML Models (Pydantic Model)

  • Model uses Pydantic union discriminators to construct the concrete type based on a specified discriminator field.
  • Extended configurations can override or add additional properties.

• Extended configuration schema for Prophet Model

• ProphetModelProvider

  • Supports training, saving, loading, and forecasting with Prophet models.
  • Implements multi-variable training and forecasting.
  • Regressors (variables correlated with the predicted variable) have their Y values interpolated (nearest) to match the timestamps of the target variable.
  • Future improvement: Make interpolation and missing data handling configurable.

• Simple provider factory for ModelProvider creation

  • Selects the appropriate implementation based on the type field in the config.
  • Easily extendable for new providers.
  • Scheduler automatically applies the correct implementation based on the config.

• ModelConfigService – CRUD service for managing model configurations.

• ModelStorageService – Handles persistence for serialized ML models.

• OpenRemoteDataService – Abstraction for the OpenRemote Client, with the focus on having a more specific interface that is suitable for the ml forecast service.

• ModelScheduler

  • Manages scheduling for training and forecasting jobs.
  • Updates jobs when configurations change.
  • Removes stale jobs when configurations are deleted.
  • Uses APScheduler.

• Utility classes

  • Time handling.
  • Filesystem handling.
  • Singleton decorator for the Scheduler .

• Other changes

  • Re-ordered the startup process so that uvicorn can use more than 1 worker (when configured)

Testing

All tests utilize mocked datasets, mocked data retrieval, and various fixtures.

I have also manually tested the scheduling, training and forecasting. I let the service run for roughly 12 hours against a OpenRemote demo instance. No issues occured during this window.

Notes

Stricter validation of the model configuration will be part of the endpoint implementation #14 - Examples are: checking whether the assets are present in the provided realm, regressor has predicted datapoints, timestamps are valid etc.

---

Acceptance Criteria

• No linter errors
• All tests pass (integration tests are allowed to skip)
• Application starts and the scheduler is started

Reach out if you'd like help with manually adding a config to test against an actual OR instance.

• The code structure/implementation makes sense
• Implementation is well documented (comments)
• Tests cover the core functionality, e.g. data preparation, training, forecast and scheduling.

Sample Config

{
    "id": "d3c119a6-1018-4ebd-932b-a509eb7ab741",
    "realm": "smartcity",
    "name": "Power Total Consumers Forecast",
    "enabled": true,
    "type": "prophet",
    "target": {
      "asset_id": "44ORIhkDVAlT97dYGUD9n5",
      "attribute_name": "powerTotalConsumers",
      "cutoff_timestamp": 1716153600000
    },
    "forecast_interval": "PT1M",
    "training_interval": "PT2M",
    "forecast_periods": 96,
    "forecast_frequency": "1h",
    "daily_seasonality": true,
    "weekly_seasonality": true,
    "yearly_seasonality": true
  }

[wborn]

I noticed these small issues while checking the changes in IntelliJ.

[wborn]

I am a bit unsure about using the filesystem directly for storing the config files, but it should be fine with the scale the service will be used at. I made sure to make the writes atomic, and since it will run in docker you can map the volume.

I'm fine with config files for now but it might be something to add in the future when there is a need for database anyhow. Will everything also be configurable via the API/UI?

[dominiquekleeven]

I am a bit unsure about using the filesystem directly for storing the config files, but it should be fine with the scale the service will be used at. I made sure to make the writes atomic, and since it will run in docker you can map the volume.

I'm fine with config files for now but it might be something to add in the future when there is a need for database anyhow. Will everything also be configurable via the API/UI?

Yes, configs will be able to be created via an UI and via the REST API.

Thanks for the review, will make the necessary changes tomorrow.