global shuffling

[grez72]

I have mxnet version 1.4.0, and I'm using ImageRecordIter or ImageRecordIter_v1 and wondering if it is possible to globally shuffle images, rather than shuffling by batch? I'm training a model where the structure resulting from batch-wise shuffling actually dramatically impairs learning, so I need a way to globally shuffle.

Thanks!

[zachgk]

Thank you for submitting the issue! I'm labeling it so the MXNet community members can help resolve it.

@mxnet-label-bot add [Feature request, Backend, Data-loading]

[ptrendx]

Adding idx file to ImagerecordIter should result in globally shuffled images.

[grez72]

Thanks for the quick response. I added the idx file, and I still don't seem to be getting a global shuffle.

Here's how I'm setting up the ImageRecordIter_v1 (I'm using v1 so that I can directly access the index. It seems ImageRecordIter returns only zeros for the index).

train_data = ImageRecordIter_v1(
    path_imglist    = path_imglist,
    path_imgrec     = rec_file,
    path_imgidx     = idx_file,
    data_shape      = (3, 224, 224),
    batch_size      = 32,
    shuffle_chunk_seed = 12,
    shuffle         = True,
    prefetch_buffer = 4,
    preprocess_threads = 8,
)

Does the global shuffle only work with ImageRecordIter? If so, is there any way to get ImageRecordIter to return the correct indices?

Thanks!

[ptrendx]

I'm not sure what do you mean by returning index. The v1 version of the ImageRecordIter is pretty old and deprecated, and I'm not even sure it does anything with the index file, let alone use it for global shuffling.

[grez72]

Sorry I wasn't clear about that. By index I mean the value in the first column of the .lst file used to generate the .rec and .idx files. Each batch of the iterator has fields for batch.data, batch.label, and batch.index.

batch = data_iter.next()
data = batch.data[0]
labels = batch.label[0]
indexes = batch.index

When using ImageRecordIter_v1, batch.index returns the index of the image, corresponding to the first column of the .lst file. However, when using ImageRecordIter, batch.index returns all zeros!

The reason I was using ImageRecordIter_v1 was because it returns the correct index value, which I need to train my model (it also seemed to be loading images faster).

Is it possible to have ImageRecordIter return the correct image index (first column of .lst file)?

Thanks!

[ptrendx]

Hmmm, seems like a bug in ImageRecordIter then.

[grez72]

I can confirm that ImageRecordIter only returns zeros for the index values. Should I submit a separate issue for this?

[ptrendx]

Please do. Did you confirm that you get global shuffling with ImageRecordIter with index provided?