Add unit field for sensor data upload form

[joshuaunity]

Description

This PR introduces a new field to the form for uploading data to a sensor, followed by the expansion of the backend validation to cover this new field.

Look & Feel

Before

After

How to test

1. Visit a sensor page. (any sensor is fine)
2. Navigate to the bottom left to find the form to upload data to the sensor
3. confirm that the unit field is visible
4. try uploading some data with while selecting a unit which cannot be converted to the main sensor unit.
5. Observe the error message toast that comes up talks about th eunit being not convertable

Further Improvements

None

Related Items

This PR closes #1781

Sign-off

• I agree to contribute to the project under Apache 2 License.
• To the best of my knowledge, the proposed patch is not based on code under GPL or other license that is incompatible with FlexMeasures

To Do

• Add unit field to "upload data" form
• Add validation for new unit field
• Data conversion integration
• Fix bug with error messages tied to this form. (Toast messages are broken here)
• New test case
• Add changelog

[read-the-docs-community]

Documentation build overview

📚 flexmeasures | 🛠️ Build #30839569 | 📁 Comparing 4b9bf0d against latest (1e28403)

🔍 Preview build

Show files changed (9 files in total): 📝 9 modified | ➕ 0 added | ➖ 0 deleted

File	Status
changelog.html	📝 modified
genindex.html	📝 modified
http-routingtable.html	📝 modified
_autosummary/flexmeasures.api.v3_0.assets.html	📝 modified
_autosummary/flexmeasures.api.v3_0.sensors.html	📝 modified
_autosummary/flexmeasures.data.queries.generic_assets.html	📝 modified
_autosummary/flexmeasures.data.schemas.sensors.html	📝 modified
api/change_log.html	📝 modified
api/v3_0.html	📝 modified

[nhoening]

Thanks.

Great that you used the schemas, and added tests, also! I see some failing tests, mainly in flexmeasures/api/v3_0/tests/test_sensors_api.py::test_upload_csv_file, so that needs to be looked after.

And I have a few other things I'd like to be done.

Also, you show the images in the wrong order in the PR description, I believe :)

[nhoening]

It works well!

I have some smaller coding and UX comments, then we can merge.

[nhoening]

Thanks, the web logic work well, but we are adding data conversion here (if input unit differs from target unit), and therefore our testing game needs to step up.

• I added some code that might already fix what I found not to be working
• I added a list of expansion TODOs you should do in the new test

I also believe the dialogue is not looking very clean. It did not look clean before, and our new field is not helping, no matter how we style it.
Maybe we should make it cleaner, by adding simple headings before each element:

1. The data
2. When was it measured?
3. What is the meaurement unit?

[nhoening]

move import to the top

[nhoening]

Why is this test in the module test_asset_api(again)?

It tests the sensor API, so move it there.

And do the renaming (remove "_auth"), I even made a clickable suggestion for that around two reviews ago.

[nhoening]

When you call this variable "price", I'm not sure what you mean. The units are not EUR, and they differ. Probably better to call this data_value.

Even better to use a few different values in the test file, e.g. 1,2,2,1 - and then we can test even better how the rescaling affects the values. But that could happen in the next step.

[nhoening]

I was experimenting with this very approach myself (I also discovered bdf.event_frequency), but I guess I did something small differently.

[Flix6x]

I also discovered the bdf.event_frequency currently fails on a 1-hour frequency, see SeitaBV/timely-beliefs#217.

[Flix6x]

The test cases themselves were not the problem. In fact, they were really helpful to replicate and identify the underlying issue. I pushed some commits which I hope fix the issue.

Other than that and a few minor review comments I advise to:

• add a few more test cases, covering:
  • downsampling (e.g. from 15-minute data to 30-minute data) in combination with kW to MW
  • uploading data to an instaneous sensor (0 resolution)
  • a price sensor (e.g. in EUR/kWh)
  • a costs sensor (e.g. in EUR)
• also write assert statements against the validation error being returned to the user for the cases where we currently expect a 422 status code
• resolve the merge conflicts with main

[Flix6x]

This isn't returning what you want it to return if the data represents a stock change rather than a flow. For instance, if the data unit is kWh (not a flow!) and its resolution is hourly, and the sensor resolution is 15 minutes, this method assumes the resampling is on a flow and it should upsample by forward filling. So 1 kWh over an hour becomes 1 kWh over 15 minutes, and 1 kWh over the next 15 minutes, etc., which is incorrect.

In contrast if the data unit is kW (a flow!) and its resolution is hourly, and the sensor resolution is 15 minutes, 1 kW over an hour becomes 1 kW over 15 minutes, and 1 kW over the next 15 minutes, etc., which is correct.

[Flix6x]

I pushed some commits that I feel address this issue as best as currently possible without requiring further changes to timely-beliefs to support resampling sensor data recording stock changes (e.g. in kWh) or even recording stocks themselves (e.g. also in kWh, but then representing cumulative readings).

[Flix6x]

@joshuaunity thanks for discussing the test cases with me today. I fixed the failing tests and will now leave this PR in your capable hands again. I also made a checklist out of my previous comment, feel free to mark things as done in it (assuming you are able to edit the comment.)