Dealing with WEL IPFs from a project file in open_projectfile_data

[Huite]

There are a number of issues with the current implementation of open_projectfile_data:

It loads identical dataframes over and over, because it treats every row of every WEL block separately. This is very inefficient.
Secondly, it doesn't reproduce the behavior of iMOD5 unless very specific conditions are met. For simplicity, I'll categorize the IPFs into two categories: "simple" when it's just a single file with rows of (x, y, rate), and "associated" when it contains references to the external time series (txt) files.

The key part to understand here is that iMOD5 clears all wells whenever it reads a new block. Due to the way that we work, by essentially creating discharge time series of each well first, we can only reproduce it by introducing a zero rate value at the the next stress period. Essentially, whenever we define some input, we could duplicate the stress period data, set its rate value to 0.0, and append it -- I'm not sure this works though, it depends on how the dataframes are combined afterwards.

Anyway, to give some example, a single definition:

0001,(WEL),1
1900-01-01
001,002
 1,2, 001, 1.0, 0.0, -999.9900 ,"simple.ipf"
 1,2, 001, 1.0, 0.0, -999.9900 ,"associated.ipf"

Here, the values of simple are simply true for the all subsequent stress periods.
For associated, however, iMOD will read the associated text files every stress period and determine the rate value.

0001,(WEL),1
1900-01-01
001,002
 1,2, 001, 1.0, 0.0, -999.9900 ,"simple.ipf"
 1,2, 001, 1.0, 0.0, -999.9900 ,"associated.ipf"
1900-01-02
001,001
 1,2, 001, 1.0, 0.0, -999.9900 ,"simple.ipf"

Note: This will turn off the associated wells at the second stress period!
Essentially, there's a duplication of time series data: once in project file, and then again in the associated text files.

Alternatively:

0001,(WEL),1
1900-01-01
001,001
 1,2, 001, 1.0, 0.0, -999.9900 ,"simple.ipf"
1900-01-02
001,001
 1,2, 001, 1.0, 0.0, -999.9900 ,"associated.ipf"

In this case, associated is only turned on in the second stress period. Unfortunately, for us, this would mean that we would have to cut off any data in the associated files that exists prior to 1900-01-02. I think this is quite undesirable: it forces time selection data into the data loading routines, rather complicating it.

The best solution to deal with this is to disallow the redundant time specification. Ideally, for the WEL, there's only one stress period block, where the associated ipf is given once. Then, all time dependent data is available from the TXT file only. However, due to the setup and history of many project files, they currently tend to repeat the associated ipf every time. So my suggestion is to allow repeated associated ipf files, but they must be present in all stress periods. This creates an unambiguous time series specification.

To illustrate, these would be equivalent and both valid:

0001,(WEL),1
1900-01-01
001,001
 1,2, 001, 1.0, 0.0, -999.9900 ,"associated.ipf"

identical to:

0001,(WEL),1
1900-01-01
001,001
 1,2, 001, 1.0, 0.0, -999.9900 ,"associated.ipf"
1900-01-02
001,001
 1,2, 001, 1.0, 0.0, -999.9900 ,"associated.ipf"
1900-01-03
001,001
 1,2, 001, 1.0, 0.0, -999.9900 ,"associated.ipf"

However, it must be checked that factor and addition values are also equal for all stress periods!

Tasks

@JoerivanEngelen added these tasks

• Create test cases for #1116 development #1125
• Change how wells are returned by imod.formats.prj.open_projectfile_data #1136
• Update Well.from_imod5_data to work with new data shape returned by imod.formats.prj.open_projectfile_data #1137

[Huite]

Second, we need to deal with the simple IPFs.

1900-01-01
001,001
 1,2, 001, 1.0, 0.0, -999.9900 ,"simple1.ipf"
1900-01-02
001,001
 1,2, 001, 1.0, 0.0, -999.9900 ,"simple2.ipf"
1900-01-03
001,001
 1,2, 001, 1.0, 0.0, -999.9900 ,"simple3.ipf"

Note, simple2 may have nothing in common with simple1!
Unfortunately, the way that we have things setup -- and if I understand correctly -- this means that the wells of simple1 are never turned off.

What we should do instead in this case is to "stop" simple1 when we come across the second stress period.
Essentially, we should treat every IPF file here as completely separate -- unless maybe the IPF files are exactly the same. We could identify them as such by appending an ID column here, which is equal to the name of the IPF file plus the row index in the dataframe. This should hold true for factor and addition as well: the changed rate is reasonably interpreted as a different rate for the same well.

What makes this significantly more complicated, I think, is what happens when you assign the same IPF to different layers. Even worse if it's assigned to a fixed layer one stress period, and then with auto-placement (layer <= 0) in another. I haven't quite thought that through yet...

[Huite]

Ideally, we can mix both. This should be allowed:

0001,(WEL),1
1900-01-01
001,001
 1,2, 001, 1.0, 0.0, -999.9900 ,"associated.ipf"
1900-01-02
001,002
 1,2, 001, 1.0, 0.0, -999.9900 ,"associated.ipf"
 1,2, 001, 1.0, 0.0, -999.9900 ,"simple.ipf"
1900-01-03
001,001
 1,2, 001, 1.0, 0.0, -999.9900 ,"associated.ipf"

I.e. associated with an external text file is "always" present. But the simple file is turned only during stress period 2; so our routines should take care of turning it off for stress period 3.

[Huite]

More thinking, I think this should not be allowed:

0001,(WEL),1
1900-01-01
001,002
 1,1, 001, 1.0, 0.0, -999.9900 ,"associated.ipf"
 1,2, 001, 1.0, 0.0, -999.9900 ,"associated.ipf"

I.e. assigning the same time series to different layers. In particular, because it would also enable this (unless we add complicated checks), setting different factor and addition values:

0001,(WEL),1
1900-01-01
001,002
 1,1, 001, 1.0, 0.0, -999.9900 ,"associated.ipf"
 1,2, 001, 0.75, 0.25, -999.9900 ,"associated.ipf"

Which gets complex to check in case the next stress period block is not consistent (layer = 0):

0001,(WEL),1
1900-01-01
001,002
 1,1, 001, 1.0, 0.0, -999.9900 ,"associated.ipf"
 1,2, 001, 1.0, 0.0, -999.9900 ,"associated.ipf"

1900-01-02
001,002
 1,0, 001, 1.0, 0.0, -999.9900 ,"associated.ipf"
 1,0, 001, 1.0, 0.0, -999.9900 ,"associated.ipf"

[Huite]

See draft PR making some of these things more concrete: #1117

[JoerivanEngelen]

Refinement

Some context/reminder for those less familiar with the .IPF format:

• An IPF can contain multiple wells
• Associated text files contain timeseries (ITYPE=1), for the established groundwater models, we can safely assume all associated text files are of this type. To be on the safe side, we can still check for this.
• Data with other ITYPE is irrelevant to the MODFLOW 6 simulation and can be discarded.

To summarize the use cases nicely outlined by @Huite :

Associated

We will restrict ourselves to:

• Associated.ipf is defined either on the first timestep or on all timesteps. Addition and factors remain constant. If not, raise error.
• Multiple layers cannot point to the same associated.ipf

Simple

• Different IPFs can be specified per timestep

Mixed

• Different simple IPFs can be specified at different timesteps, associated need to be kept constant.

Next steps

• Convert examples here to simple test cases to test against. Also include tests for things that explicitly should fail
• Work on implementation, salvage Start addressing #1116 #1117