Conversation
for more information, see https://pre-commit.ci
mgrover1
left a comment
mgrover1
left a comment
There was a problem hiding this comment.
Great! Thanks for adding a test here too
mgrover1
left a comment
mgrover1
left a comment
There was a problem hiding this comment.
Oh wait - we need to make sure kerchunk is installed in the testing environment too
|
Thanks for the quick review! I did add the kerchunk package in the previous PR #758 in requirement.txt. Or is the test environment set somewhere else? Thanks! |
It needs to included here As in the testing infrastructure, it does not install all the dependencies in the requirements.txt file |
|
Looks like python3.10 is in conflict with the kerchunk version for the python3.10 CI test. |
can we drop python3.10 in a separate PR and close this issue ? |
|
@mgrover1 @andersy005 just checking to see do I need to do anything for the failed testing? Many thanks! |
correct! I am planning on submitting a PR tonight with the fix; then we can just merge the main branch in here |
andersy005
left a comment
andersy005
left a comment
There was a problem hiding this comment.
This looks solid to me 👍🏽
@mgrover1, when you get a chance, can you take a final pass at reviewing this and merge it?
3 participants
Change Summary
This pull request improves support for opening kerchunk datasets with the kerchunk engine, particularly distinguishing between local and remote reference files. It also adds new tests to ensure the correct behavior for both remote HTTPS URLs and local files when using the kerchunk engine.
Related issue number
Related to the PR #758. This fix bug of local access on the PR and also add unittest on both local and remote reference file access using Kerchunk engine.
Checklist
Enhancements to dataset opening logic:
_open_datasetfunction insource.pyto avoid using fsspec when loading local reference files while engine set tokerchunk, ensuring correct handling by checking thatxarray_open_kwargs['engine'] != 'kerchunk'before usingfsspec.open_local.Expanded test coverage:
test_open_dataset_kerchunk_engineto verify that remote kerchunk reference files accessed via HTTPS are properly opened with the kerchunk engine.test_open_dataset_kerchunk_engine_localto verify that local kerchunk reference files are handled correctly, including testing with specificstorage_optionsfor remote S3 access.