datasets.load_datasets('parquet',...) fails to cache .map calls. Sometimes.

[el-hult]

I have tried to make a minimal reproduction, but have not really managed. So bear with me for a second.

I have a file foo.py with contents

import os
import datasets
import lib.render
import polars as pl

def batch_render(EXAMPLE: dict):
  lib.render.render_candidates()
  return {"ImageData": []}


if __name__ == "__main__":
  fname = "test.parquet"

  if not os.path.exists(fname):
    d = [("a", 1), ("b", 2), ("c", 3), ("d", 4), ("e", 5), ("f", 6)]
    df = pl.DataFrame(d, schema=["a", "b"])
    df.write_parquet(fname)

  dataset: datasets.Dataset = datasets.load_dataset("parquet", data_files=(fname,))[
    "train"
  ]  # only one split, with default name 'train'
  dataset = dataset.select(range(2)) # deleting 
  dataset = dataset.map(batch_render)

If I call this a few times with command line rm -rf ~/.cache/huggingface/datasets/ && python foo.py && python foo.py, the output looks like

~/S/P/p/e/250512_faircompare (main)> rm -rf ~/.cache/huggingface/datasets/ &&  python foo.py && python foo.py
/Users/elhult/SynologyDrive/Postdoktor/picasso/experiments/250512_faircompare/foo.py:16: DataOrientationWarning: Row orientation inferred during DataFrame construction. Explicitly specify the orientation by passing `orient="row"` to silence this warning.
  df = pl.DataFrame(d, schema=["a", "b"])
Generating train split: 6 examples [00:00, 3745.47 examples/s]
Map: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:00<00:00, 1364.22 examples/s]
Map: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:00<00:00, 966.10 examples/s]
~/S/P/p/e/250512_faircompare (main)> rm -rf ~/.cache/huggingface/datasets/ &&  python foo.py && python foo.py
Generating train split: 6 examples [00:00, 3486.05 examples/s]
Map: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:00<00:00, 981.93 examples/s]
~/S/P/p/e/250512_faircompare (main)> rm -rf ~/.cache/huggingface/datasets/ &&  python foo.py && python foo.py
Generating train split: 6 examples [00:00, 3617.34 examples/s]
Map: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:00<00:00, 1098.70 examples/s]
Map: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:00<00:00, 1051.47 examples/s]
~/S/P/p/e/250512_faircompare (main)> rm -rf ~/.cache/huggingface/datasets/ &&  python foo.py && python foo.py
Generating train split: 6 examples [00:00, 3710.68 examples/s]
Map: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:00<00:00, 1111.07 examples/s]
~/S/P/p/e/250512_faircompare (main)> rm -rf ~/.cache/huggingface/datasets/ &&  python foo.py && python foo.py
Generating train split: 6 examples [00:00, 3316.53 examples/s]
Map: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:00<00:00, 1056.50 examples/s]
~/S/P/p/e/250512_faircompare (main)> 

so clearly the caching mechanism fails intermittently. The second tqdm progress bar appears only when the cached versions of the map call comes to an invvalidated cache. I need to understand why the caching fails.

The contents of lib.render looks like

def render_candidates():
  return  []

import polars as pl

a = [(1,2),(3,4)]
df = pl.DataFrame(a, schema=["a", "b"])
df = df.group_by("a").agg(pl.all())

There are a couple of confusing things:

• renaming the folder fixes the problem for a few seconds
• deleting some of the other files in the same directory fixes the problem
• renaming lib/render.py fixes the problem
• Adding comments at the beggining of the file tends to make the cache failures deterministic -- it always fails to use the cache.

How can I debug the cache invalidation behavior? Where can I find exactly a description of the caching logic?

[el-hult]

This seems to be an issue with dill. Simply put, caching in huggingface relies on a library that sometimes fails. However the discussion I found is quite old, and it is hard for me to verify this is the problem. uqfoundation/dill#19 (comment)

What I did was to check what dumps(val) became in

datasets/src/datasets/fingerprint.py

Line 188 in 53f958e

return cls.hash_bytes(dumps(value))

as the map function tries to create hash of batch_render in part of determining on whether to use cache, it creates a dill serialization of the function. That representation is stochastically different between runs.

What can I do to enforce reproducibility and cache utilization?

[el-hult]

I will for the time being use the new_fingerprint argument to map, in order to get a deterministic fingerprint.