[Discussion] isDirectory()

[stefanofornari]

Hello,
currently isDirectory() is delegated to the implementation of PosixLikePathRepresentation.isDirectory() which basically checks if the given argument contains references (., ..) or separators (/).
Is there a way to actually detect if a S3 key is a real file object or not?
The use case I am trying to cover is the following: imagine we want to implement the ls command. If I issue

ls path

in linux, if path is a directory I get the content of the directory (i.e. I do not have to use path/).
With the current implementation of the NIO provider I would have to issue:

ls path/

thoughts?

[stefanofornari]

Taking inspiration from https://stackoverflow.com/questions/54444600/determine-if-folder-or-file-key-boto, we now actually have that information while listing the content of a key (e.g. in newDirectoryStream). Can we have maybe a lazy initialized field that can be set when a directory is scanned, or if not initialized, list the parent and check if it goes in the content or prefix sessions?

[markjschreiber]

I think the problem is that in S3 both of these are valid objects in the same bucket

s3://mybucket/path
s3://mybucket/path/foo

So s3://mybucket/path is an object and s3://mybucket/path/ is a prefix (s3 equivalent of a directory)

You cannot do this in Unix because you can't have a file called path when there is a directory also called path in the same location.

[stefanofornari]

Yep, that is what I was thinking as well. On the other side, if I am not mistaking, a not existing Path s3://mybucket/path2/ would return true on isDirectory(); this is not a problem by itself, it could be valid semantic, but it is not consistent with the FileSystem semantic:

System.out.println(Files.isDirectory(new File("doesnotexists").toPath())); -> false

Given that in S3 a folder exists if and only it is a prefix for other keys, maybe we should check for that. It is indeed much less efficient and we need to figure out how to handle the case where we have both a file and a dir (I guess we can just returns true to both isDirectory() and isRegularFile()), but I am inclined to think it would be better implementation.

[markjschreiber]

You are correct, we should check that s3://mybucket/path2/ actually exists before saying it is a directory.

You can have an empty prefix so there doesn't have to be objects under s3://mybucket/path2/ for it to exist. It wouldn't be a regular file unless if was s3://mybucket/path2 and that object exists.