split settings for the first pass and multi-pass

[alexlib]

the problem
at the moment windef.piv() takes windef.Settings() with only one type of correlation method. it prevents us from a more robust operation when the first pass is for instance linear and normalized and other passes are faster due to already deformed windows circular and not normalized.

Describe the solution you'd like
the suggestion is to move to YAML type of settings that could be even stored in a file next to the results. then we could have

[first pass]
correlation: linear 
normalized : True

[multi pass]
correlation: circular
normalized: False

it would also help us to get extended search into the first pass using different interrogaton window sizes and
overlaps and also will open the path to the rectangular windows

[first pass]
interrogation size A : [64, 128]
interrogation size B : [128, 256] 

[alexlib]

@eguvep what do you think about changing settings to YAML format and unify those between openpiv and GUI version?

[eguvep]

Hi Alex,
YAML ist just the format, not the structure, right? For example: For the GUI, we are using an OpenPivParams object, which is holding basically just a single dictionary (and some convenient functions). That dictionary could of course be saved as a YAML file. windef.Settings() is an object containing separate member variables, that could also be saved as a YAML file. So the storage format would be the same, the data structure would still be different. Both settings must still be translated to each other, because the structure is different. We earlier discussed a similar topic for the vector data format. OpenPIV/openpiv_tk_gui#22 (comment)
In my point of view, the question is broader:

• Do we want a single vector format?
• Do we want a single settings format?
• Do we want to merge them in a single multi purpose file (like suggested by Erich in the linked thread)?
• Should every contributor care about converting the formats by himself?
• Should we start a separate format conversion project (DaVis to OpenPiv to NetCDF to ...)?

[alexlib]

Thanks for a quick reply.

The parameters issue shall be probably separated from the results storage. Since parameters file is small and it's really for the simple adjustment of those between the runs and for storage for reproducibility, the data storage format is a different beast and it requires some thinking about being a proprietary (our) binary format or some general binary format or maybe ASCII (human readable) format and what it should contain.

As far as I see it is that a working folder will contain the raw image files and the settings parameter file next to it leading to the data in the output folder. The path can be a Python script, Jupyter notebook or GUI, but the result should be the same to a certain extent. If GUI creates a different output depending on additional filters, sub-settings, etc. - we need to have it also in the parameters file, so the Jupyter notebook can be re-created for the same path from input to output.
All the internal things are fine to be adjusted as long as those are transparent for the user.

I believe we have to have a single settings format and more so - to have a single format for Matlab and Python and other versions. Therefore it's best to store in YAML format (or similar, but there are no many that are readable so easy by C, Matlab and Python) .

Let's have the data output format discussion on OpenPIV/openpiv_tk_gui#22 (comment)

[alexlib]

See how nice I can draw :)

[alexlib]

as far as I see your JSON format is very close to YAML format. I do not mind to keep parameters in JSON format as well, but I think that YAML today is a bit more popular and more human readable.
I suggest the following:
GUI reads the piv_parameters.yaml that will be a single format with a list of a) common parameters, b) additional parameters. Since YAML is a flexible format, the reader (either Jupyter notebook or script or GUI) does not have to know a-priori all the settings it might include, but only those which are necessary and common will be checked. All the rest is populated by GUI or shown to the user in the notebook and could be adjusted there (and stored back). I.e. a simple parameters file will include only what is in windef.Settings() and anything below it can be for GUI, but won't be read by the notebook as irrelevant.
The main thing is still that x,y,u,v should be at least within a small error reproducible through the various systems if they all use underneath the same extended_search_piv. It can be different between Matlab and Python, of course, but for the simple set of settings like 32 x 32 no filtering no smoothing we'd expect to get the same sort of result, at least on average.

[alexlib]

I guess we need to figure out first between us what is crucial in the structure of the parameters. It will be then easier to decide if we have the same minimal set with the translators or we can agree about the same structure for the minimal set.

[alexlib]

I have a prototype here
https://github.com/OpenPIV/openpiv-python/blob/windef_yaml/openpiv/examples/notebooks/windef_settings_yaml.ipynb
https://github.com/OpenPIV/openpiv-python/blob/windef_yaml/openpiv/examples/notebooks/default_settings.yaml
in the branch windef_yaml https://github.com/OpenPIV/openpiv-python/tree/windef_yaml