Support for arbitrary cloud storage services

[timblaktu]

Hi @hairyhenderson! I have selected your repo as a starting point for a component of a personal project I'm working on - essentially a multicloud personal digital media analyzer/organizer. I have decided to abstract each supported storage backend as a fs.FS, which is of course what brought me here. :-)

My question for you is whether you'd be interested in pull requests here supporting fs.FS impls for:

1. SmugMug (and perhaps other photo/video-centric sites)
   • These all have a tree-like structure, where galleries are like folders and can be nested.
2. Google Drive (and perhaps other user-focused cloud storage products like Dropbox or Box..)

??

I really like what you've done here and would love to contribute to my needs here, and also get the benefit of the cloud storage support you've already added.

Cheers,
Tim

[timblaktu]

I've read up on gomplate and see that your intent here is to provide gomplate support for reading gomplate data sources from a variety of cloud storage locations. FWIW, the goals of this project appear aligned with those of mine (neither are negatively affected by fs.FS's read-only constraint), so I'll go ahead and pursue developing fsimpl wrappers for some providers.

Let me know if you've got any feedback or guidance other than what's in your README. I've also been looking at hackpadfs and considering borrowing from their approach as well. CHeers.

[hairyhenderson]

Hi @timblaktu! 👋

This sounds like a pretty cool project! I'm not familiar with either the SmugMug or Google Drive APIs, so I can't comment on whether or not it's a good idea to access them like filesystems, but I can certainly see the appeal!

I've played with the Flickr API a bit, and while the thought did cross my mind, I figured the way that the same images can be stored in collections and/or albums makes shoehorning them into a directory structure feel awkward. But, I had no real use for that at the time, and maybe the awkwardness isn't too restrictive...

I'd recommend trying first to create go-fsimpl-compatible filesystems in separate modules, and if they prove to be super-useful and don't add a lot of extra complication, I could consider including them in go-fsimpl.

But do keep me posted on how the project goes - it sounds very interesting!

BTW - I really like the hackpadfs project - I use some of their writeable interfaces in my day job. I've been contemplating adding write support to some go-fsimpl filesystems, but the lack of standard interface is somewhat preventing me.

[timblaktu]

Re: cloud photo service as filesystem, i think every one that ive seen maps well to this model. The collection idea appears in most of them, and i believe this should be implemented as a symlink, or perhaps just not supported at all. Collections are just different views into backend storage "paths", and for my apps needs, i really am only concerned with a single, most primary view.

Behind collections, SmugMug has a tree of galleries which can be treated like folders in an FS. The galleries are the primary view.

If Flickr or any another service doesnt have anything exactly "folder-like", and instead only has a single, associative structural mechanism like collections, its probably cheap blob storage on the backend and collections represent folders containing symlinks.

It all aligns well in my mind, but we'll see how it goes. My app will be scanning entire FS trees and doing mostly r/o operations for now. So there will probably be bazillions of goroutines blocked on network or disk I/O most of the time...

[hairyhenderson]

Sounds good! I took a quick look at SmugMug's API and it looks fairly straightforward. Flickr's is a bit more complex - one interesting quirk is you can download an image in a range of sizes. I think for a filesystem the "original" size is probably always what you want.

Anyway - I'm very interested to see how this progresses 😉

[timblaktu]

Yeah, i coincidentally saw your post there on fs.FS write support the other day. Sometimes its a small world when you get deep into a nook or cranny of tech. I reached out to hackpadfs's creator on IBM internal slack yesterday after noticing how much we had in common. Not that im anywhere near as prolific in open-source as he...

Thanks for the tips. At this point, im likely to start with forks of hackpadfs and go-fsimpl, and will do some experimentation and keep you and John posted. I know he shares your concerns about the lack of write interface support and diverging feom the standard lib too much..

[hairyhenderson]

Small world indeed! I'm an IBM alumnus - though I left back in 2015 😉

[...] im likely to start with forks of hackpadfs and go-fsimpl [...]

I don't think you need a fork at all... go-fsimpl really only has a small amount of stuff on top of the standard fs.FS (and related) interfaces, and it's all completely optional.

From the top of my head, though, to be "go-fsimpl compatible" you could:

1. implement the FSProvider interface, so the filesystem can be registered with an FSMux

• this is optional as you can also just use WrappedFSProvider for ad-hoc filesystems

2. implement the internal WithContexter/WithHTTPClienter/WithHeaderer interfaces (https://pkg.go.dev/github.com/hairyhenderson/go-fsimpl/internal#WithContexter) to support context cancellation, custom HTTP client support, or custom header support

• again, this is optional, though when the filesystem is hitting an HTTP API I'd highly recommend at least adding the WithContext method so cancellation propagates properly

3. have your file struct's Stat return a type that implements the internal ContentTypeFileInfo interface so a content type can be discovered with ContentType

I'm thinking in your use-case - especially for the SmugMug wrapper - it might be handy to have the FileInfo type also have some way to store an image's EXIF data.