Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Avoid BioImage.IO Core downloading everything from model zoo entries #386

Merged
merged 2 commits into from
Jan 27, 2025

Conversation

qin-yu
Copy link
Collaborator

@qin-yu qin-yu commented Jan 27, 2025

From this

2025-01-24 16:29:51.676 | INFO     | bioimageio.spec._internal.io_utils:open_bioimageio_yaml:131 - loading philosophical-panda from https://uk1s3.embassy.ebi.ac.uk/public-datasets/bioimage.io/philosophical-panda/0.0.11/files/rdf.yaml
Downloading data from 'https://uk1s3.embassy.ebi.ac.uk/public-datasets/bioimage.io/philosophical-panda/0.0.11/files/rdf.yaml' to file '/home/runner/.cache/bioimageio/520a69782c7dafb6478e43ebcc4679b0-rdf.yaml'.
Downloading data from 'https://uk1s3.embassy.ebi.ac.uk/public-datasets/bioimage.io/philosophical-panda/0.0.11/files/README.md' to file '/home/runner/.cache/bioimageio/42620dae3e3850cefbf0475c7cf590dd-README.md'.
SHA256 hash of downloaded file: fc6e1292ca309bedaca504260cecc9a7bc9f26e9328eb7a051f82a2ceec475e3
Use this value as the 'known_hash' argument of 'pooch.retrieve' to ensure that the file hasn't changed if it is downloaded again in the future.
Downloading data from 'https://uk1s3.embassy.ebi.ac.uk/public-datasets/bioimage.io/philosophical-panda/0.0.11/files/test_input.npy' to file '/home/runner/.cache/bioimageio/101853864f8c8e986b2819c9ac44d0f9-test_input.npy'.

computing SHA256 of 101853864f8c8e986b2819c9ac44d0f9-test_input.npy:   0%|          | 0/5529[72](https://github.com/kreshuklab/plant-seg/actions/runs/12953461209/job/36133179407#step:12:73)8 [00:00<?, ?it/s]
computing SHA256 of 101853864f8c8e986b2819c9ac44d0f9-test_input.npy (result: 6810255f5b5260fe39153f2192bedf30d9899ec4e770976b7813116c467579f0): 100%|██████████| 5529728/5529728 [00:00<00:00, 1340501691.67it/s]
computing SHA256 of 101853864f8c8e986b2819c9ac44d0f9-test_input.npy (result: 6810255f5b5260fe39153f2192bedf30d9899ec4e770976b7813116c467579f0): 100%|██████████| 5529728/5529728 [00:00<00:00, 1267119769.96it/s]
Downloading data from 'https://uk1s3.embassy.ebi.ac.uk/public-datasets/bioimage.io/philosophical-panda/0.0.11/files/test_output.npy' to file '/home/runner/.cache/bioimageio/57b964a123db3ff8400bcce3ef902b18-test_output.npy'.

computing SHA256 of 57b964a123db3ff8400bcce3ef902b18-test_output.npy:   0%|          | 0/8294528 [00:00<?, ?it/s]
computing SHA256 of 57b964a123db3ff8400bcce3ef902b18-test_output.npy (result: d802e3024da80bff93a9ec50fbe50b9c3946534aab1b60b911511111a8e2dbca): 100%|██████████| 8294528/8294528 [00:00<00:00, 13320[73](https://github.com/kreshuklab/plant-seg/actions/runs/12953461209/job/36133179407#step:12:74)820.44it/s]
computing SHA256 of 57b964a123db3ff8400bcce3ef902b18-test_output.npy (result: d802e3024da80bff93a9ec50fbe50b9c3946534aab1b60b911511111a8e2dbca): 100%|██████████| 8294528/8294528 [00:00<00:00, 1282808700.90it/s]
Downloading data from 'https://uk1s3.embassy.ebi.ac.uk/public-datasets/bioimage.io/philosophical-panda/0.0.11/files/test_style.npy' to file '/home/runner/.cache/bioimageio/0b712fa8abf8707021a71[74](https://github.com/kreshuklab/plant-seg/actions/runs/12953461209/job/36133179407#step:12:75)7e726bf7c-test_style.npy'.

computing SHA256 of 0b712fa8abf8707021a71747e726bf7c-test_style.npy:   0%|          | 0/76928 [00:00<?, ?it/s]
computing SHA256 of 0b712fa8abf8707021a71747e726bf7c-test_style.npy (result: ab464b406f9050561b40f7d76700ab5edf3aca97e31fe9a6069a51aeeca8bc81): 100%|██████████| 76928/76928 [00:00<00:00, 517912388.62it/s]
computing SHA256 of 0b712fa8abf8707021a71747e726bf7c-test_style.npy (result: ab464b406f9050561b40f7d76700ab5edf3aca97e31fe9a6069a51aeeca8bc81): 100%|██████████| 76928/76928 [00:00<00:00, 19251[75](https://github.com/kreshuklab/plant-seg/actions/runs/12953461209/job/36133179407#step:12:76)52.57it/s]
Downloading data from 'https://uk1s3.embassy.ebi.ac.uk/public-datasets/bioimage.io/philosophical-panda/0.0.11/files/test_downsampled_0.npy' to file '/home/runner/.cache/bioimageio/d0abd68ef3844b0d6fbed811e0fed878-test_downsampled_0.npy'.

computing SHA256 of d0abd68ef3844b0d6fbed811e0fed878-test_downsampled_0.npy:   0%|          | 0/88473728 [00:00<?, ?it/s]
computing SHA256 of d0abd68ef3844b0d6fbed811e0fed878-test_downsampled_0.npy (result: 67df53fb440e94dbb9c8e4003dcbde158646a7975c4878cacdd251e1fcfb4225): 100%|██████████| 88473728/88473728 [00:00<00:00, 1368234468.02it/s]
computing SHA256 of d0abd68ef3844b0d6fbed811e0fed878-test_downsampled_0.npy (result: 67df53fb440e94dbb9c8e4003dcbde158646a7975c4878cacdd251e1fcfb4225): 100%|██████████| 88473728/88473728 [00:00<00:00, 136152[76](https://github.com/kreshuklab/plant-seg/actions/runs/12953461209/job/36133179407#step:12:77)08.58it/s]
Downloading data from 'https://uk1s3.embassy.ebi.ac.uk/public-datasets/bioimage.io/philosophical-panda/0.0.11/files/test_downsampled_1.npy' to file '/home/runner/.cache/bioimageio/3765cac1d92a49daf0d6ec949919aeb1-test_downsampled_1.npy'.

computing SHA256 of 3765cac1d92a49daf0d6ec949919aeb1-test_downsampled_1.npy:   0%|          | 0/44236928 [00:00<?, ?it/s]
computing SHA256 of 3765cac1d92a49daf0d6ec949919aeb1-test_downsampled_1.npy (result: cb4addbd763d96731ebd18ed001b87ab7195ec9198f01a753a363a06c27bfb1c): 100%|██████████| 44236928/44236928 [00:00<00:00, 1367172813.64it/s]
computing SHA256 of 3765cac1d92a49daf0d6ec949919aeb1-test_downsampled_1.npy (result: cb4addbd763d96731ebd18ed001b87ab7195ec9198f01a753a363a06c27bfb1c): 100%|██████████| 44236928/44236928 [00:00<00:00, 1354794156.08it/s]
Downloading data from 'https://uk1s3.embassy.ebi.ac.uk/public-datasets/bioimage.io/philosophical-panda/0.0.11/files/test_downsampled_2.npy' to file '/home/runner/.cache/bioimageio/7251078a2afa8713384a3103878dd09d-test_downsampled_2.npy'.

computing SHA256 of 7251078a2afa8713384a3103878dd09d-test_downsampled_2.npy:   0%|          | 0/22118528 [00:00<?, ?it/s]
computing SHA256 of 7251078a2afa8713384a3103878dd09d-test_downsampled_2.npy (result: 9c0225b94d84fcc3adfb9a73eef1303d6adb318b57a5a801e0e2e1638b458e72): 100%|██████████| 22118528/22118528 [00:00<00:00, 1361668410.33it/s]
computing SHA256 of 7251078a2afa8713384a3103878dd09d-test_downsampled_2.npy (result: 9c0225b94d84fcc3adfb9a73eef1303d6adb318b57a5a801e0e2e1638b458e72): 100%|██████████| 22118528/22118528 [00:00<00:00, 1338930701.[77](https://github.com/kreshuklab/plant-seg/actions/runs/12953461209/job/36133179407#step:12:78)it/s]
Downloading data from 'https://uk1s3.embassy.ebi.ac.uk/public-datasets/bioimage.io/philosophical-panda/0.0.11/files/test_downsampled_3.npy' to file '/home/runner/.cache/bioimageio/e434cdc3ea3e7ecfb752cdc00161[78](https://github.com/kreshuklab/plant-seg/actions/runs/12953461209/job/36133179407#step:12:79)75-test_downsampled_3.npy'.

computing SHA256 of e434cdc3ea3e7ecfb752cdc001617875-test_downsampled_3.npy:   0%|          | 0/11059328 [00:00<?, ?it/s]
computing SHA256 of e434cdc3ea3e7ecfb752cdc001617875-test_downsampled_3.npy (result: 1ea789ff37d47197c847b585[79](https://github.com/kreshuklab/plant-seg/actions/runs/12953461209/job/36133179407#step:12:80)9f7d063e7592b0c5e9c3094fd0e3ac209b7fc2): 100%|██████████| 11059328/11059328 [00:00<00:00, 1373144183.65it/s]
computing SHA256 of e434cdc3ea3e7ecfb752cdc001617875-test_downsampled_3.npy (result: 1ea789ff37d47197c847b585799f7d063e7592b0c5e9c3094fd0e3ac209b7fc2): 100%|██████████| 11059328/11059328 [00:00<00:00, 1329269362.33it/s]
Downloading data from 'https://uk1s3.embassy.ebi.ac.uk/public-datasets/bioimage.io/philosophical-panda/0.0.11/files/cpnet_wrapper.py' to file '/home/runner/.cache/bioimageio/00bd170d6c9de6a391d6869f59058847-cpnet_wrapper.py'.

computing SHA256 of 00bd170d6c9de6a391d6869f59058847-cpnet_wrapper.py:   0%|          | 0/11053 [00:00<?, ?it/s]
computing SHA256 of 00bd170d6c9de6a391d6869f59058847-cpnet_wrapper.py (result: b8b947cdd0ea8f5b98bd7be5f12f38bb1ea1ebe0b455c62d9a6389cd21d134bf): 100%|██████████| 11053/11053 [00:00<00:00, 147173467.02it/s]
computing SHA256 of 00bd170d6c9de6a391d6869f59058847-cpnet_wrapper.py (result: b8b947cdd0ea8f5b98bd7be5f12f38bb1ea1ebe0b455c62d9a6389cd21d134bf): 100%|██████████| 11053/11053 [00:00<00:00, 28687897.35it/s] 
Downloading data from 'https://uk1s3.embassy.ebi.ac.uk/public-datasets/bioimage.io/philosophical-panda/0.0.11/files/cp_state_dict_1135_gold.pth' to file '/home/runner/.cache/bioimageio/8dbb20d5a3cb3a3dfdb5101a671861ce-cp_state_dict_1135_gold.pth'.

computing SHA256 of 8dbb20d5a3cb3a3dfdb5101a671861ce-cp_state_dict_1135_gold.pth:   0%|          | 0/26556687 [00:00<?, ?it/s]
computing SHA256 of 8dbb20d5a3cb3a3dfdb5101a671861ce-cp_state_dict_1135_gold.pth (result: 26c277f3b8f6ca5aab30b4b0a832601aea60183cbed1c2333576f4135a643eb2): 100%|██████████| 26556687/26556687 [00:00<00:00, 1385235897.41it/s]
computing SHA256 of 8dbb20d5a3cb3a3dfdb5101a671861ce-cp_state_dict_1135_gold.pth (result: 26c277f3b8f6ca5aab30b4b0a832601aea60183cbed1c2333576f4135a643eb2): 100%|██████████| 26556687/26556687 [00:00<00:00, 1366910692.51it/s]
Downloading data from 'https://uk1s3.embassy.ebi.ac.uk/public-datasets/bioimage.io/philosophical-panda/0.0.11/files/cp_traced_1135_gold.pt' to file '/home/runner/.cache/bioimageio/17fee110c39ccad7c3cb36d00d2fdd2c-cp_traced_1135_gold.pt'.

computing SHA256 of 17fee110c39ccad7c3cb36d00d2fdd2c-cp_traced_1135_gold.pt:   0%|          | 0/26[81](https://github.com/kreshuklab/plant-seg/actions/runs/12953461209/job/36133179407#step:12:82)2339 [00:00<?, ?it/s]
computing SHA256 of 17fee110c39ccad7c3cb36d00d2fdd2c-cp_traced_1135_gold.pt (result: f61bae146ab522902350eadda1d509ac1037726fe6d7fb63f6a8a314021d63e7): 100%|██████████| 26812339/26812339 [00:00<00:00, 1384043871.28it/s]
computing SHA256 of 17fee110c39ccad7c3cb36d00d2fdd2c-cp_traced_1135_gold.pt (result: f61bae146ab522902350eadda1d509ac1037726fe6d7fb63f6a8a314021d63e7): 100%|██████████| 26812339/26812339 [00:00<00:00, 136429[82](https://github.com/kreshuklab/plant-seg/actions/runs/12953461209/job/36133179407#step:12:83)01.10it/s]
tests/functionals/prediction/test_prediction_biio.py INFO: P [MainThread] 2025-01-24 16:30:48,206 plantseg.functionals.prediction.prediction - Model expects these inputs: ['raw'].
2025-01-24 16:30:48.316 | WARNING  | bioimageio.spec.model.v0_5:get_axis_size:2568 - Ignoring unexpected size increment factor (n) for fixed size axis 'channel' of tensor 'raw'.

predict sample raw with philosophical-panda:   0%|          | 0/1 [00:00<?, ?block/s]
predict sample raw with philosophical-panda: 100%|██████████| 1/1 [00:04<00:00,  4.65s/block]
predict sample raw with philosophical-panda: 100%|██████████| 1/1 [00:04<00:00,  4.65s/block]
WARNING: P [MainThread] 2025-01-24 16:30:53,024 plantseg.functionals.prediction.prediction - Model has more than one output tensor. PlantSeg does not support this yet.

to this

2025-01-27 14:21:39.538 | INFO     | bioimageio.spec._internal.io_utils:open_bioimageio_yaml:131 - loading philosophical-panda from https://uk1s3.embassy.ebi.ac.uk/public-datasets/bioimage.io/philosophical-panda/0.0.11/files/rdf.yaml
Downloading data from 'https://uk1s3.embassy.ebi.ac.uk/public-datasets/bioimage.io/philosophical-panda/0.0.11/files/rdf.yaml' to file '/home/runner/.cache/bioimageio/520a69782c7dafb6478e43ebcc4679b0-rdf.yaml'.
tests/functionals/prediction/test_prediction_biio.py INFO: P [MainThread] 2025-01-27 14:21:41,092 plantseg.functionals.prediction.prediction - Model expects these inputs: ['raw'].
Downloading data from 'https://uk1s3.embassy.ebi.ac.uk/public-datasets/bioimage.io/philosophical-panda/0.0.11/files/cpnet_wrapper.py' to file '/home/runner/.cache/bioimageio/00bd170d6c9de6a391d6869f59058847-cpnet_wrapper.py'.
SHA256 hash of downloaded file: b8b947cdd0ea8f5b98bd7be5f12f38bb1ea1ebe0b455c62d9a6389cd21d134bf
Use this value as the 'known_hash' argument of 'pooch.retrieve' to ensure that the file hasn't changed if it is downloaded again in the future.
Downloading data from 'https://uk1s3.embassy.ebi.ac.uk/public-datasets/bioimage.io/philosophical-panda/0.0.11/files/cp_state_dict_1135_gold.pth' to file '/home/runner/.cache/bioimageio/8dbb20d5a3cb3a3dfdb5101a671861ce-cp_state_dict_1135_gold.pth'.
2025-01-27 14:22:44.008 | WARNING  | bioimageio.spec.model.v0_5:get_axis_size:2568 - Ignoring unexpected size increment factor (n) for fixed size axis 'channel' of tensor 'raw'.

predict sample raw with philosophical-panda:   0%|          | 0/1 [00:00<?, ?block/s]
predict sample raw with philosophical-panda: 100%|██████████| 1/1 [00:04<00:00,  4.62s/block]
predict sample raw with philosophical-panda: 100%|██████████| 1/1 [00:04<00:00,  4.62s/block]
WARNING: P [MainThread] 2025-01-27 14:22:48,690 plantseg.functionals.prediction.prediction - Model has more than one output tensor. PlantSeg does not support this yet.

Copy link

codecov bot commented Jan 27, 2025

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 67.59%. Comparing base (564bfa4) to head (c4a5154).

Additional details and impacted files
@@           Coverage Diff           @@
##           master     #386   +/-   ##
=======================================
  Coverage   67.59%   67.59%           
=======================================
  Files          76       76           
  Lines        5506     5506           
=======================================
  Hits         3722     3722           
  Misses       1784     1784           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@qin-yu
Copy link
Collaborator Author

qin-yu commented Jan 27, 2025

For some reason, Fynn's official solution in bioimage-io/core-bioimage-io-python#451 doesn’t seem to work, but there is a workaround.

@qin-yu qin-yu self-assigned this Jan 27, 2025
@qin-yu qin-yu added the enhancement New feature or request label Jan 27, 2025
@qin-yu qin-yu merged commit 40a77d8 into master Jan 27, 2025
4 checks passed
@qin-yu qin-yu deleted the qy/avoid-biio-tensor-download branch January 27, 2025 21:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant