Missing ehv- and hvmv-voronoi

[ClaraBuettner]

When I run the current dev branch I don't see any ehv or hvm-substation voronoi which results in error in several tasks (e.g . hydro-biomass-powerplants).
The log of create-voronoi says:

> [2021-07-19 10:24:20,540] {standard_task_runner.py:78} INFO - Job 4508: Subtask create-voronoi
> [2021-07-19 10:24:20,577] {logging_mixin.py:120} INFO - Running <TaskInstance: egon-data-processing-pipeline.create-voronoi 2021-07-19T08:13:56.373752+00:00 [running]> on host clara-LIFEBOOK-U749
> [2021-07-19 10:24:20,624] {logging_mixin.py:120} WARNING - 2021-07-19 10:24:20.622 | INFO     | egon.data.datasets:skip_task:186
> Dataset 'MvGridDistricts' version '0.0.0' already executed.
> Skipping.

But the voronoi tables are dropped and created in task create-substation-tables. So currently the table gets dropped and recreated but not filled because task create-voronoi is skipped.

The voronoi creation is part of the MV grid district dataset, was that the intention?

eGon-data/src/egon/data/datasets/mv_grid_districts.py

Lines 773 to 779 in be4c69a

	mv_grid_districts_setup = partial(
	Dataset,
	name="MvGridDistricts",
	version="0.0.0",
	dependencies=[],
	tasks=(create_voronoi, define_mv_grid_districts),
	)

[nesnoj]

Frankly, I'm not very familiar with @gplssm's code.
I ran the test mode on Friday from scratch and the tables grid.egon_hvmv_substation_voronoi and grid.egon_ehv_substation_voronoi were created and filled properly.

I just re-ran that task and get the same message like you but nothing gets dropped in my DB. The new MVGD dataset version (#317) was merged on Friday, it might somehow conflict with your existing data? Did u try to clear and re-run?

[ClaraBuettner]

Frankly, I'm not very familiar with @gplssm's code.
I ran the test mode on Friday from scratch and the tables grid.egon_hvmv_substation_voronoi and grid.egon_ehv_substation_voronoi were created and filled properly.

I think this was added with the merge on Friday, the voronoi creation separated from the MVGD before.

I just re-ran that task and get the same message like you but nothing gets dropped in my DB. The new MVGD dataset version (#317) was merged on Friday, it might somehow conflict with your existing data? Did u try to clear and re-run?

If you re-run only the task it works. The problem occurs when I trigger the whole DAG again and all tasks run. Then the tables are dropped and created in create-substation-tables, but since the version of the MVGD is already there, the tables are not filled again.

[nesnoj]

As agreed f2f, I'll move the create_voronoi task back to (unversioned) substations.

[nesnoj]

@nailend will take care of this

[nailend]

The voronoi creation is part of the MV grid district dataset, was that the intention?

I assume, some osmtgmod functions are a dependency for the voronoi-creation and therefore executed afterwards but I am not entirely sure. I do not fully understand if the same tables (grid.egon_ehv_substation and grid.egon_hvmv_substation) are used and altered in create_voronoi and osmtgmod_substation or subsequent parts of osmtgmod. To fix this for now I will assume a dependency but maybe we will have to look into this a bit deeper.

[nailend]

In osmtgmod_substation the bus_ids are altered which are then used in create_voronoi ?

[ClaraBuettner]

I think it is the easiest option to restore the old dependencies of create_voronoi:

eGon-data/src/egon/data/airflow/dags/pipeline.py

Lines 346 to 350 in 972b3c1

	create_voronoi = PythonOperator(
	task_id="create_voronoi",
	python_callable=substation.create_voronoi
	)
	osmtgmod_substation >> create_voronoi