Empty segment in path error

[ttomasz]

When listing (obstore.list) and object path contains double delimiter exception is thrown.
At least in Azure those paths - while weird - are actually readable. One can even have two different paths like: container/l1/l2/file.csv, container/l1/l2//file.csv.

As far as I can tell there's no way to ignore the error since the exception would stop the iterator but maybe it could be possible to have a flag that would print warning and ignore such paths? Or if the behaviour was dependent on implementation e.g. if it's not supported in s3 or something it would throw error but if it works in azure it returns those paths?

I know we shouldn't have such a mess with the file paths but unfortunately sometimes it's other people writing for us and we have no influence over that 😅

[kylebarron]

Thanks for the issue. obstore wraps the underlying Rust library object_store. If object_store errors on these empty path segments, then there's no way for obstore to recover from that exception. So we should open up an issue on the object_store
repo: https://github.com/apache/arrow-rs-object-store

Would you be able to do that?

[ttomasz]

Thanks for quick reply Kyle. Sorry for delay from my side but I did create the issue (enhancement request) in arrow-rs-object-store repo (linked above).

[kylebarron]

Thanks for creating the upstream issue. I'm pretty slammed and don't think I'll be able to make the upstream change soon, but it shouldn't be too complex if you want to try it out?