Performance of contiguous datasets

[fmckeogh]

On my contiguous dataset, any read (even of a single element) results in all 35GB being read into memory.

use hidefix::{prelude::Index, reader::ReaderExt};
use std::sync::Arc;

const PATH: &str = "/home/fm208/Downloads/pubmed/benchmark-dev-pubmed23.h5";

fn main() {
    let i = Arc::new(Index::index(PATH).unwrap());
    let mut r = i.reader("train").unwrap();
    let values = r.values::<f32, _>([0..1, 0..1]).unwrap();

    panic!("{values:?}");
}

In read_to on CacheReader, self.ds.chunk_slices returns one giant chunk, which when passed into read_chunk, loads the entire file. hdf5-metno does not have this behavior.

I'm not sure what changes would be needed to hidefix to improve this, do you have any suggestions?

[gauteh]

To avoid reading the one giant chunk, the dataset also needs to be uncompressed. Then it would be possible to just read that part of the file into the target array, rather than reading the entire chunk into memory first: https://github.com/gauteh/hidefix/blob/main/src/reader/direct.rs#L137.

[fmckeogh]

use hidefix::{
    idx::DatasetD,
    prelude::Index,
    reader::{direct::Direct, ReaderExt},
};
use std::sync::Arc;

const PATH: &str = "/home/fm208/Downloads/pubmed/benchmark-dev-pubmed23.h5";

fn main() {
    let i = Arc::new(Index::index(PATH).unwrap());
    let ds = i.dataset("train").unwrap();
    let DatasetD::D2(ds) = ds else { panic!() };
    let mut direct = Direct::with_dataset(ds, PATH).unwrap();

    let values = direct.values::<f32, _>([0..1, 0..1]).unwrap();

    panic!("{values:?}");
}

This still reads in the entire file, did I use it incorrectly? The dataset is not compressed.

[gauteh]

Yeah, sorry, the direct reader still reads in the whole chunk - but it could be modified to only read the necessary byte range (when the dataset is uncompressed).