Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[utils] netCDF converter #2654

Merged
merged 30 commits into from
Dec 9, 2019
Merged

[utils] netCDF converter #2654

merged 30 commits into from
Dec 9, 2019

Conversation

rinkk
Copy link
Member

@rinkk rinkk commented Sep 10, 2019

Command line utility to conver netCDF-files into OGS meshes. Works for both 2d and 3d meshes + time. Multiple time steps can be saved as seperate meshes or as one mesh with one scalar array per time step.

(A follow-up PR will move all non-ui functionality into FileIO and adjust the DataExplorer interface to use that as well.)

  1. Feature description was added to the changelog

Please note:

  • Conan minimum version was bumped to 1.20, do pip install --upgrade conan!
  • Updated Conan Qt package to 5.13.2

@endJunction
Copy link
Member

Is there a test for the command line tool?

@rinkk
Copy link
Member Author

rinkk commented Sep 12, 2019

No, there isn't. The smallest nc-file I have is ~170 mb. From what I understand, this is too large for our tests?

@endJunction
Copy link
Member

endJunction commented Sep 12, 2019

That's too large. There must be small files. As usual, the reason is to have working examples, s.t. in case of code changes the utility is covered.

std::array<std::size_t, 4> const& dim_idx_map,
bool is_time_dep)
{
std::size_t const temp_offset = (is_time_dep) ? 1 : 0;
Copy link
Member

@endJunction endJunction Sep 12, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

temp means temperature?
why no simply offset?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

temporal offset ... in contrast to the spatial offset checked further on.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd rename the variable for readability.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If it's the same to you I'll keep it as it is because the UI explicitely says "temporal dimension".

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
std::size_t const temp_offset = (is_time_dep) ? 1 : 0;
std::size_t const temporal_offset = (is_time_dep) ? 1 : 0;

Copy link
Member

@endJunction endJunction left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The converter looks to be very specific for a given input type and relies on series of inputs from the user.
It is not a generic netcdf converter/extractor, so I'd recommend to rename it to the specific actions it does for that particular file format.

@rinkk
Copy link
Member Author

rinkk commented Sep 23, 2019

What do you mean by "very specific for a given input type"? It uses arbitrary 2D/3D/4D rasters (gives an error msg for 1D) and creates a mesh out of these. That's the data netCDF contains. I could add a conversion for 1D, too, but it wouldn't make much sense since that is usually diagrams from my experience.

What do you think is specific about that? What other data does netCDF contain that you are aware of? In what sense is it not generic?

@rinkk
Copy link
Member Author

rinkk commented Sep 23, 2019

The smallest nc-file I have is ~170 mb. From what I understand, this is too large for our tests?

That's too large. There must be small files. As usual, the reason is to have working examples, s.t. in case of code changes the utility is covered.

I understand but don't have a small file.

Unidata offers a number of test files ... I've seen a 22 mb test example for surface data. Would that be acceptable? There might be smaller files but I'd have to check it's not just 1D data. There's also the license issue ... Unidata software is offered under 3 clause BSD but there's no mention if the same rules apply for their example data sets. What's your opinion on that?

@endJunction
Copy link
Member

https://www.unidata.ucar.edu/software/netcdf/examples/files.html has several files smaller than 10MiB, e.g. the testrh.nc (only 90KiB).

@rinkk
Copy link
Member Author

rinkk commented Sep 23, 2019

Yes, but I need a file containing a 2D, 3D, or 4D array. That is why I said I'd have to check for a small file with a 2D array assuming you think the license automatically extends to datasets as well.

@bilke
Copy link
Member

bilke commented Dec 6, 2019

@endJunction @rinkk Ready to merge, everything is green!

@bilke bilke merged commit f0e7c8a into ufz:master Dec 9, 2019
@bilke bilke deleted the NetCdfSplit branch December 9, 2019 08:46
@ogsbot
Copy link
Member

ogsbot commented Jun 19, 2020

OpenGeoSys development has been moved to GitLab.

See this pull request on GitLab.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants