Allow subdirectories for add & delete granules #20
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Codecov Report
@@ Coverage Diff @@
## main #20 +/- ##
==========================================
+ Coverage 93.51% 94.16% +0.64%
==========================================
Files 4 4
Lines 833 1028 +195
==========================================
+ Hits 779 968 +189
- Misses 54 60 +6
Flags with carried forward coverage won't be shown. Click here to find out more.
|
If we're defining multiple layers with the same config file, it might make sense to have the files in subdirectories per layer this allows keeping the subdir specified in argument or keeping the path from the geoserver granule location Note! Deleting files with this might fail if the path in geoserver is different than path on host server!
ritvje
force-pushed
the
feature-subdirs-add-granule
branch
from
4b7a305
to
acb190b
Compare
lahtinep
reviewed
Oct 3, 2023
georest/__init__.py
Outdated
| @@ -286,7 +286,7 @@ def add_file_to_mosaic(config, fname_in, filesystem='posix'): | |||
| This function wraps some boilerplate around adding a granule to a layer. | |||
|
|
|||
| """ | |||
| fname = utils.convert_file_path(config, fname_in) | |||
| fname = utils.convert_file_path(config, fname_in, keep_subpath=config["keep_subpath"]) | |||
There was a problem hiding this comment.
Could also be
Suggested change
| fname = utils.convert_file_path(config, fname_in, keep_subpath=config["keep_subpath"]) | |
| fname = utils.convert_file_path(config, fname_in, keep_subpath=config.get("keep_subpath", False)) |
so forgetting the extra option from the config doesn't cause a crash.
georest/__init__.py
Outdated
|
|
||
|
|
||
| def _delete_files_from_fs(config, gs_location): | ||
| delete_files = config.get("delete_files", False) | ||
| if not delete_files: | ||
| return | ||
| fs_path = utils.convert_file_path(config, gs_location, inverse=True) | ||
| fs_path = utils.convert_file_path(config, gs_location, inverse=True, keep_subpath=config["keep_subpath"]) |
There was a problem hiding this comment.
Suggested change
| fs_path = utils.convert_file_path(config, gs_location, inverse=True, keep_subpath=config["keep_subpath"]) | |
| fs_path = utils.convert_file_path(config, gs_location, inverse=True, keep_subpath=config.get("keep_subpath", False)) |
| @@ -85,9 +85,12 @@ def _write_property_zip(prop_paths, zip_path): | |||
| return zip_path | |||
|
|
|||
|
|
|||
| def convert_file_path(config, file_path, inverse=False): | |||
| def convert_file_path(config, file_path, inverse=False, keep_subpath=False): | |||
There was a problem hiding this comment.
Apparently I haven't had a test case for this function, but we could add something now to test_utils.py. I'll write some and push them to this branch in a moment.
|
@ritvje do you think the cases I added for |
|
Looks good. Your version of the inverse option of |
ritvje
force-pushed
the
feature-subdirs-add-granule
branch
from
47167d0
to
2f8d905
Compare
lahtinep
approved these changes
Oct 4, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Add a config option that allows keeping the subpath given in arguments when adding granules.
E.g. if config file has
if the filepath for
add_granule.pyis given asfivih_3067/20230926_1300_fivih_ppi-0.3_optclass.tif, the filepath is interpreted as/smartmet/radar/geotiff/radar-optclass/fivih_3067/20230926_1300_fivih_ppi-0.3_optclass.tif