LocalFileSystem: offset for `list_with_offset` can't be identified / List results *must* be sorted

[timwaizenegger]

Describe the bug

The list / [list_with_offset](https://docs.rs/object_store/0.12.0/object_store/trait.ObjectStore.html#method.list_with_offset) functions on LocalFileSystem stores do not (reliably) return results in sorted order.
Now the interface docs for this interface state that:

Note: the order of returned ObjectMeta is not guaranteed

so it would be consistent with that but I believe this is not useful:

• offset filtering/listing works by skipping any input objects that are (lexicographically) smaller than the offset
• in order to use offsets to list the all objects in multiple iterations, each iteration must contain a slice of the sorted overall results
• if I have such a slice, the last element will be correct offset to the get the next slice/batch
• if results are in a random order overall, offset listing is impossible to use. I can never find any useful offset value out of the ones I received
  • unless I list everything and sort it then. But that defeats the purpose
• object stores typically list results in sorted order specifically to allow for offset listing
• Having a usable offset listing feature is key for object-sort to achieve its design goal of a stateless API; the only alternative for users is to use a stateful iterator.

To Reproduce

generate a set of files:

mkdir -p /tmp/manyfiles/
cd /tmp/manyfiles/
    for i in $(seq 1 5000); do
echo "hello world" > "hello world.txt.$i"
done

Run this code to show the (random) ordering and show that offset listing isn't usable:

#[tokio::main]
async fn main() -> Result<()> {
    let store = LocalFileSystem::new_with_prefix("/tmp/manyfiles")?;

    // list() returns a Stream of Result<ObjectMeta>
    let mut list_stream = store.list(None);

    // pull each ObjectMeta out of the stream
    while let Some(result) = list_stream.next().await {
        let meta = result?;
        // print its path
        println!("-> {} - {}", meta.location.to_string(), meta.last_modified.to_string());
    }



    println!("\nListing files in batches of 10 using list_with_offset...");
    let mut offset: Option<String> = None;
    loop {
        // Choose list or list_with_offset based on whether we have an offset
        let mut batch_stream = if let Some(ref off) = offset {
            store.list_with_offset(None, &Path::from(off.as_str()))
        } else {
            store.list(None)
        };

        let mut count = 0;
        let mut last_path: Option<String> = None;
        while let Some(result) = batch_stream.next().await {
            let meta = result?;
            println!("-> {} - {}", meta.location.to_string(), meta.last_modified.to_string());
            count += 1;
            last_path = Some(meta.location.to_string());
            if count >= 10 {
                break;
            }
        }
        // Stop if fewer than 10 items were returned
        if count < 10 {
            break;
        }
        // Update offset for the next batch
        offset = last_path;
    }

    Ok(())
}

Here is an example of the results:

cargo run
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.26s
     Running `target/debug/objectstore-list-bug`
-> hello world.txt.822 - 2025-05-29 19:55:29.630128009 UTC
-> hello world.txt.3678 - 2025-05-29 19:55:29.979447372 UTC
-> hello world.txt.1913 - 2025-05-29 19:55:29.765314276 UTC
-> hello world.txt.2598 - 2025-05-29 19:55:29.844496358 UTC
-> hello world.txt.1779 - 2025-05-29 19:55:29.749811871 UTC
-> hello world.txt.3812 - 2025-05-29 19:55:29.994586274 UTC
...

Expected behavior

I should be able to use offset listing with local files just like it work with S3 and others.

Additional context

I found this behavior on macos and redhat linux.

A POC for a possible fix is here: https://github.com/apache/arrow-rs-object-store/pull/389/files

[tustvold]

Changed to a feature request, as the current behaviour is as documented

[timwaizenegger]

Changed to a feature request, as the current behaviour is as documented

that's a stretch IMO.
The docs for LocalFile do not mention sorting order: https://docs.rs/object_store/0.12.0/object_store/local/struct.LocalFileSystem.html#method.list_with_offset only the generic interface docs have that comment.

with the current behavior, offset listing just can't be used for local files. But the feature is implemented & documented and none of the docs say that this doesn't work on local files. Moreover, it works on the other stores.

[tustvold]

The docs for LocalFile do not mention sorting order: https://docs.rs/object_store/0.12.0/object_store/local/struct.LocalFileSystem.html#method.list_with_offset only the generic interface docs have that comment.

That's a quirk of how docs.rs renders, if you click the read more link it shows more than just the first line.

I think there is a misunderstanding as to what list_with_offset is designed for, it is for when listing everything after a given prefix. This is very common pattern for data lake changelogs and is why we support it. However, it is not intended as a mechanism to DIY pagination, nor is it documented as supporting this

[Xuanwo]

We've had similar discussions about adding sort support for fs several times. For example, see apache/arrow-rs#6375.

Let me share my previous comment:

LocalFileSystem should mimic what ListObjectsV2 does, returning objects in lexicographical order of their keys.

Hi, this behavior is expected. We can't perform sorting over the LocalFileSystem since we don't have such an API, and we can't collect keys and sort them in-memory. The best we can do is return objects in the order that the filesystem provides.

Intended only for small stores that fin in memory?

The most interesting part is this: we don't know how many items there are until we list them all. This inconsistent behavior based on item size might introduce more issues. For example, users might find the object_store behaves one way when there are only 10 items, but differently in production services.

---

The only thing we can rely on is that the file system will return the same results if there are no external changes. So, we can simply list from the given offset without worrying about missing any entries.

with the current behavior, offset listing just can't be used for local files. But the feature is implemented & documented and none of the docs say that this doesn't work on local files. Moreover, it works on the other stores.

This is incorrect because offset listing doesn't require the results to be ordered; it only requires the filesystem to have stable results, meaning that the order remains consistent within the filesystem itself.

If we implement sorting for file systems or other services that do not return entries in order (as mentioned by @tustvold, such as AWS S3 Express), our users might experience the following strange issues:

• The list operation hangs indefinitely without returning any entries.
• Memory usage keeps increasing, potentially leading to an out-of-memory (OOM) situation.
• The number of API requests grows significantly, even when users only want to fetch the first entry.

So my suggestion is to keep our current behavior AS IS, and perhaps clarify this in our documentation in case it might confuse users for now.

[timwaizenegger]

This is incorrect because offset listing doesn't require the results to be ordered; it only requires the filesystem to have stable results, meaning that the order remains consistent within the filesystem itself.

I though so too initially. But the offset filter is just that; a filter. It doesn't "skip until offset reached", it just filters based on lexicographical comparisons. The "skip until" couldn't even work without sorted results. What if the token file gets deleted? Now you can't know how far to skip any more.

How about we add sorted output for LocalFileSystem as an opt-in feature; as mentioned here? #389 (comment)

No harm in doing that, right? Can document the pros and cons and it would enable this paged-listing use case for local files. I'm happy to propose changes if you can give me a few pointers. Could be a cargo "feature" since you probably decide at compile time if you need this and not at runtime.

[tustvold]

What if the token file gets deleted? Now you can't know how far to skip any more.

The start offset is a filter, so it would just yield the values after that point. The same "issue" would occur if you inserted a new file before or after the iteration position. FWIW no object stores provide stable pagination AFAIK.

it would enable this paged-listing use case for local files.

But it doesn't, it just fudges around it by fetching everything from the filesystem, sorting it and then discarding those less than a given filter. This isn't materially different from your application doing this... Perhaps it might be worth taking a step back and explaining what your use-case and we can then better advise on how to achieve it. Ultimately pagination isn't something filesystems support.

[timwaizenegger]

Sorry for the late reply. Our use case is this:

our application processes objects/files in batches. Each batch execution is an independent stateless function call. I can only pass strings/primitive types between these function calls. So today, we use a continuation token. The very first invocation has an empty string so we start processing from the beginning. Then we return the last object name and the execution framework feeds that into the next invocation.

With S3 and the other object stores, I can do paginated listing with such a token.

A bit more background:
Our code is part of a custom postgres DB extension where each batch runs in its own "transaction context"; PG memory safety imposes constraints on what data we can pass.

Options I can see

1. application re-write; could e.g. separate the listing from the processing logic. It's just a heavy lift for us so I'm looking for other options
2. lift the sorting/pagination logic into the app; rather than ask object_store to do it

• possibly; but only needed for local files not other object stores. So I'd have to break the generic access pattern I can use today and introduce a special case for a certain type of object_store back end

3. Have a feature/config option on object_store that makes it return sorted results to match behavior of other stores

(3) is just a nice and clean solution for our use case. I agree it will have poor performance; it's a tradeoff users can decide to make.

Would you support adding a rust feature or config option (e.g. a with_sorted_listing) to the LocalFileSystem implementation?

[tustvold]

With S3 and the other object stores, I can do paginated listing with such a token.

This is not guaranteed behaviour and will misbehave on some S3 compatible stores including S3 express. PaginatedListStore was added for this use case.

Is there a reason you don't wish to use an actual object store like minio or similar?

Would you support adding a rust feature or config option (e.g. a with_sorted_listing) to the LocalFileSystem implementation?

I am pretty lukewarm on adding this for the reasons already articulated.