Problems with NifTI

[CloseChoice]

Describe the bug

There are currently 2 problems with the new NifTI feature:

1. dealing with zipped files, this is mentioned and explained here
2. when uploading via the niftifolder feature, the resulting parquet only contains relative paths to the nifti files:

table['nifti']
<pyarrow.lib.ChunkedArray object at 0x798245d37d60>
[
  -- is_valid: all not null
  -- child 0 type: binary
    [
      null,
      null,
      null,
      null,
      null,
      null
    ]
  -- child 1 type: string
    [
      "/home/tobias/programming/github/datasets/nifti_extracted/T1.nii",
      "/home/tobias/programming/github/datasets/nifti_extracted/T2-interleaved.nii",
      "/home/tobias/programming/github/datasets/nifti_extracted/T2.nii",
      "/home/tobias/programming/github/datasets/nifti_extracted/T2_-interleaved.nii",
      "/home/tobias/programming/github/datasets/nifti_extracted/T2_.nii",
      "/home/tobias/programming/github/datasets/nifti_extracted/fieldmap.nii"
    ]
]

instead of containing bytes. The code is copy pasted from PDF, so I wonder what is going wrong here.

Steps to reproduce the bug

see the linked comment

Expected behavior

downloading should work as smoothly as for pdf

Environment info

• datasets version: 4.4.2.dev0
• Platform: Linux-6.14.0-33-generic-x86_64-with-glibc2.39
• Python version: 3.12.3
• huggingface_hub version: 0.35.3
• PyArrow version: 21.0.0
• Pandas version: 2.3.3
• fsspec version: 2025.9.0

[lhoestq]

2. when uploading via the niftifolder feature, the resulting parquet only contains relative paths to the nifti files:

what did you use to upload the dataset ? iirc push_to_hub() does upload the bytes as well, but to_parquet() doesn't

[CloseChoice]

2. when uploading via the niftifolder feature, the resulting parquet only contains relative paths to the nifti files:

what did you use to upload the dataset ? iirc push_to_hub() does upload the bytes as well, but to_parquet() doesn't

I used push_to_hub but the problem is that the nifti feature does not have an embed_storage function