`DatabricksFileSystem.ls()` returns empty list for directories in some scenarios.

[mkoistinen]

I'm using fsspec to access a number of filesystems including file, fs, hdfs, webhdfs and dbfs. Of these, only dbfs is returning an empty list in some conditions where I request the contents of a Parquet dataset directory. Here's a repro case:

from __future__ import annotations

import os
from typing import Any
from types import MethodType

import fsspec

if __name__ == "__main__":

    fs = fsspec.filesystem(
        "dbfs",
        instance=os.environ["DATABRICKS_HOST"],
        token=os.environ["DATABRICKS_TOKEN"],
    )

    for path in [
        "/mkoistinen@host.com/parquet_files",          # PATH A
        "/mkoistinen@host.com/parquet_files/bank/",    # PATH B
    ]:
        listing = fs.ls(path=path, detail=False)  # NOTE: the value for `detail` makes no difference
        if listing and isinstance(listing[0], dict):
            print(f"ls({path})\n{'\n'.join(f.get('name') for f in listing)}", end="\n\n")  # noqa
        else:
            print(f"ls({path})\n{'\n'.join(listing)}", end="\n\n")

When run with 2025.5.1 (and earlier versions, I tried all released 2025 versions), the output of this run is:

ls(/mkoistinen@howso.com/parquet_files)
/mkoistinen@host.com/parquet_files/adult
/mkoistinen@host.com/parquet_files/adult.parquet
/mkoistinen@host.com/parquet_files/bank
/mkoistinen@host.com/parquet_files/bank.parquet
/mkoistinen@host.com/parquet_files/iris
/mkoistinen@host.com/parquet_files/iris.parquet
/mkoistinen@host.com/parquet_files/tmp

ls(/mkoistinen@host.com/parquet_files/adult)
/mkoistinen@host.com/parquet_files/adult

If I disable PATH A in the list, I get:

ls(/mkoistinen@host.com/parquet_files/adult)
/mkoistinen@host.com/parquet_files/adult/part_18aacbf9-6844-4038-8929-c45028ef870c.parquet
/mkoistinen@host.com/parquet_files/adult/part_4950eed3-56d3-4f28-962e-46c9f728f5c4.parquet
/mkoistinen@host.com/parquet_files/adult/part_8e7e0a04-bc42-4275-bbb8-b95b5ef39ab7.parquet
/mkoistinen@host.com/parquet_files/adult/part_d6d5e107-77a3-41c2-96c4-18af726d5ef6.parquet
/mkoistinen@host.com/parquet_files/adult/part_dc302b0b-6f18-4093-be95-271554c4ca21.parquet

The correct output when both paths are enabled should be:

ls(/mkoistinen@host.com/parquet_files)
/mkoistinen@host.com/parquet_files/adult
/mkoistinen@host.com/parquet_files/adult.parquet
/mkoistinen@host.com/parquet_files/bank
/mkoistinen@host.com/parquet_files/bank.parquet
/mkoistinen@host.com/parquet_files/iris
/mkoistinen@host.com/parquet_files/iris.parquet
/mkoistinen@host.com/parquet_files/tmp

ls(/mkoistinen@host.com/parquet_files/adult)
/mkoistinen@host.com/parquet_files/adult/part_18aacbf9-6844-4038-8929-c45028ef870c.parquet
/mkoistinen@host.com/parquet_files/adult/part_4950eed3-56d3-4f28-962e-46c9f728f5c4.parquet
/mkoistinen@host.com/parquet_files/adult/part_8e7e0a04-bc42-4275-bbb8-b95b5ef39ab7.parquet
/mkoistinen@host.com/parquet_files/adult/part_d6d5e107-77a3-41c2-96c4-18af726d5ef6.parquet
/mkoistinen@host.com/parquet_files/adult/part_dc302b0b-6f18-4093-be95-271554c4ca21.parquet

I've traced this down to an issue where the cache self._ls_from_cache used in the ls() method is returning objects where, in this case, .../adult was seen as a plain file even though we're requesting the contents of that path in a subsequent call to ls().

(Note that this behavior is unaffected by the presence of a trailing slash in the path to denote a directory.)

I'm able to get the correct behavior by inserting one line:

src: fsspec/implementations/dbfs.py:60-75

    def ls(self, path, detail=True, **kwargs):
        """
        List the contents of the given path.
        ...
        """
        out = self._ls_from_cache(path)
        out = [o for o in out if o.get("name") == path and o.get("type") == "file"] if out else []   # <--- SINGLE LINE FIX
        if not out:
            ...

I can submit this as a PR, but, I thought I'd check with the maintainers first. Please advise.

[martindurant]

Of course, separating out files and directories where they can have the same name is always a problem (s3fs and gcsfs know this too).

Your fix may well be right, but I hope some of the people more familiar with dbfs can confirm: @nils-braun , @tomerm-iguazio , @andersy005 , @a24lorie maybe

[mkoistinen]

I went ahead and produced a PR. On closer analysis, it was clear that the correct solution is slightly more involved than my one-liner mentioned above.

The issue actually stems from logic in AbstractFileSystems._ls_from_cache() to reach into the parent path's cache, if available, to return the entry there for path when a native cached entry for path is not present. Unfortunately, I think that logic is flawed in the case of directories (to the affect reported in this issue).

It looks like from this codebase that only DatabricksFileSystem actually uses _ls_from_cache, but, to be on the safe side, I overrode that method directly in the DBFS class, leaving the one in AbstractFileSystem as-is. If the maintainers wish, I can also delete the non-public method from AbstractFileSystems in this PR. Let me know.

The new logic checks if there is a cache for the key path and returns it if found. If it is not found, we check to see if there is an entry in the cache for the parent path. If there is, and the file at path is a "file", it is returned as an optimization to avoid an additional API call for information already cached. However, if it is there but a "directory", nothing is returned so that ls can make the API call to get the contents of path and then cache the results per usual.

If path is not found in the parent path's cache, then a FileNotFoundError is raised and strongly indicates that path is a new file since it's parent path was cached. I also modified ls to detect this, and invalidate the parent cache entry before directly fetching the path's contents.

[martindurant]

So it is the case that the parent directory has an entry, but it is a directory type - did I understand correctly? That would be the correct thing to return in the case of an info() call, but not ls() as you say.

_ls_from_directory does explicitly check that the returned entry is a file here.

It looks like from this codebase that only DatabricksFileSystem actually uses _ls_from_cache

I don't think that can be true - at least s3fs and gcsfs do, and in this repo I see it in ReferenceFS.

[mkoistinen]

In the case of an info() call, it would be appropriate to return what the parent has cached in self.dircache... looks like info() doesn't use _ls_from_cache() directly, but does via ls(). Looking at the logic, it should be fine since the logic in info() looks at the output of ls(path.parent) first, which seems correct.

It looks like from this codebase that only DatabricksFileSystem actually uses _ls_from_cache

I don't think that can be true - at least s3fs and gcsfs do, and in this repo I see it in ReferenceFS.

Hmm, VS Code doesn't turn that up when I did "Find all references". Regardless, I won't delete the base implementation.

Great project, btw!