Skip to content

Option to disable temporary file writing for in-memory workflows #303

@cmayet

Description

@cmayet

Hi !

I’d like to propose an improvement / change regarding EOReader’s use of temporary files.

Context

EOReader writes temporary files after loading bands, in order to:

  • cache processed data
  • speed up subsequent reads

This is very useful in many workflows and probably responds perfectly to the initial use cases !

Problem

In some use cases, this caching mechanism is unnecessary and introduces overhead.
Example workflow:

  1. Load a product's bands once (fits in memory)
  2. Compute a spectral index
  3. Discard the data

In this case, bands are never reloaded, temporary files are never reused, disk I/O becomes pure overhead

Expected behavior

It would be useful to have an option to disable temporary file writing entirely, so that all processing stays in memory, no intermediate GeoTIFFs are written.

Proposed solution

Introduce a parameter such as:

product.load(..., use_cache=False)
# OR 
product.load(..., write_tmp=False)

I understand that a significant part of EOReader’s internal design may rely on this temporary file mechanism. However, this would be a very useful feature for my use case, which involves processing large volumes (many thousands) of cropped Sentinel/Landsat products that fit in memory, whereminimizing I/O and optimizing processing time is a key concern.

Would this fit EOReader’s design philosophy?

Thanks in advance for your feedback!

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels
    No fields configured for Feature.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions