Add some mechanism to clear the cache

[robmarkcole]

🚀 Feature

I often update the dataset on S3 without actually changing the URL, however then a cached version may be used without my knowing. I would like a way/command to clear the cache, without needing to search for it and perform the clear manually

[tchaton]

Hey @robmarkcole Do you merge datasets together ?

[robmarkcole]

Never done that, yet.!

[tchaton]

So how do you add new data ?

[robmarkcole]

I just overwrote the whole dataset

[deependujha]

this bothered me too when I was working on append and overwrite feature. Even after updating (appending data), streamingDataset used to return cached data only. So, I'd to manually clear the cache.

---

Proposed solution

We can use an environment variable that when set to true, will clear cache before running StreamingDataset function. (Default False).

[tchaton]

Right now, we are hashing the input_dir to create a path for the cache. A better option would be to hash also the index.json file, so if he has changed, this becomes a new path or we clean the old ones as we know the index.json has changed.

[deependujha]

Right now, we are hashing the input_dir to create a path for the cache. A better option would be to hash also the index.json file, so if he has changed, this becomes a new path or we clean the old ones as we know the index.json has changed.

I think, this could be one part of the implementation, but it lacks in case the index.json file doesn't change on updating, like, function returns (i, i**3) instead of (i, i**2). The dataset will change, but even the new implementation will use cache only.

So, having an environment variable that you can set like DO_NOT_CACHE. If it's set, in the function _try_create_cache_dir clear the cache_dir, and then proceed. This will be mostly useful in development.

Your thoughts?

[bhimrazy]

I also encountered similar issues with hashing based solely on the index.json content, which results in the same hash when the content remains unchanged. I also explored using the last modification time, but unfortunately, the original timestamp was lost once the index file was downloaded.

so, IMO setting an environment variable like DO_NOT_CACHE seems like a good solution.

or, Alternatively, we could introduce a new field such as timestamp in the index.json to record the last creation or modification time.

[deependujha]

Adding last updation timestamp seems better option to me, We won't be required to hash the whole index.json file, neither bother about setting any environment variable.

Just hashing the timestamp will do the trick.

[tchaton]

Yeah @deependujha. Let's do that.