Option to save user avatars / image files to S3 instead of local disk

[lvwrence]

Feature Request

Is your feature request related to a problem? Please describe.
It might be good to not have to depend on the local filesystem to save files, since some platforms like Heroku (that we use) wipe local files on reboot. It would also allow Flarum to scale horizontally, since then the server would be stateless and you can spin up as many as required, since they would just be a broker to MySQL / S3. This could be an option on setup that writes to config.php. Let me know what you think.

Describe the solution you'd like
I have a solution (although badly written) that saves the logo, favicon, and user avatars to S3 instead of to the local file system, thanks to Flysystem's S3 adapter. It also uses redis to store the sessions since those are stored on disk as well. We'd need to make it optional and create and read the fields in config.php and initialize the correct Flysystem adapter if enabled.

Justify why this feature belongs in Flarum's core, rather than in a third-party extension
This deals with image saving in core.

Describe alternatives you've considered
We could also store the images as base64 in the MySQL database, but that would not be ideal, as forums would not benefit from CDNs.

[matteocontrini]

Related to #822

[franzliedke]

Right, the session driver still needs to either be improved or be made pluggable. I currently favor switching to a database-based one.

[askvortsov1]

The way I understand this, there are effectively 2 parts to this:

• Make sure the appropriate filesystem disk is used anytime assets or avatars are modified
• Provide some way to configure s3. Probably through config.php?

Let's start with the first one. A few concerns about our current system:

Firstly, at times, we use contextual binding to provide relevant disk drivers to classes. Namely:

• https://github.com/flarum/core/blob/298f6c39f2f56d6ac8c043426c583773d04df8e2/src/Settings/SettingsServiceProvider.php#L49-L49
• https://github.com/flarum/core/blob/298f6c39f2f56d6ac8c043426c583773d04df8e2/src/User/UserServiceProvider.php#L88-L88

While that's technically speaking the correct way to do it, it doesn't scale automatically to extensions, as we need to specify every single class that should receive these interfaces. This leaves us with several options:

1. Add an extender to make it easier to register which of an extension's classes needs.
2. Inject the whole filesystem manager instance, and let extensions get the disks they need.

Secondly (and much more crucially), there's a lot of places where we are assuming a local filesystem. In particular:

• https://github.com/flarum/core/blob/298f6c39f2f56d6ac8c043426c583773d04df8e2/src/Database/Console/MigrateCommand.php#L96-L96
• https://github.com/flarum/core/blob/298f6c39f2f56d6ac8c043426c583773d04df8e2/src/Extension/ExtensionManager.php#L258-L258
• https://github.com/flarum/core/blob/298f6c39f2f56d6ac8c043426c583773d04df8e2/src/Extension/ExtensionManager.php#L270-L270
• https://github.com/flarum/core/blob/298f6c39f2f56d6ac8c043426c583773d04df8e2/src/Extension/ExtensionManager.php#L282-L282

Here, we could also just inject a filesystem manager and set the needed disks in the consumer class's constructor. With that in place, the main thing we'd need to support is some form of util to easily copy entire directories over at once (we could make https://github.com/flarum/core/blob/298f6c39f2f56d6ac8c043426c583773d04df8e2/src/Extension/Extension.php#L433-L433 a bit more flexible).

Finally, we have a case of filesystem assumption in the installation script:

• https://github.com/flarum/core/blob/298f6c39f2f56d6ac8c043426c583773d04df8e2/src/Install/Installation.php#L168-L168

I'm not as sure what to do with this one: prompting for S3 info during installation feels like a bit much; I'm assuming that would be set up later (and updated with php flarum migrate)? Probably fine to keep as is.

[tankerkiller125]

Is there any reason we couldn't use something like https://github.com/thephpleague/flysystem as that would make it very easy to retrieve files, upload files, etc. regardless of driver. Not to mention it already has a ton of drivers pre-written (including S3, Azure, DigitalOcean, etc.)

It's the same library that Laravel uses under the hood I think it'd be smart to use a library that already has file system drivers and interfaces rather than re-writing the wheel.

Edit: Small note, if we do go with this library I recommend we go with v1 for now until v2 gets more adapters.

[askvortsov1]

It's the same library that Laravel uses under the hood I think it'd be smart to use a library that already has file system drivers and interfaces rather than re-writing the wheel.

If Laravel is just an abstraction around Flysystem, then why not take advantage of Laravel's implementation-independent API? We could allow overriding the default disk drivers via config.php (or through settings): https://laravel.com/docs/8.x/filesystem.

The only disadvantage to using Laravel's layer is that we can't use MountManager, but that's deprecated. Besides, in all Flarum-usage instances I've found, the only usage of MountManager is copying stuff from the local filesystem to whichever disk we want to modify. But we could just as easily write those files directly via the FilesystemInterface's put and replace methods, which frankly might result in even simpler code.

[askvortsov1]

The 3 PRs linked above close this issue.

[mmm8955405]

So far, I can't find a way to upload my avatar to other places (such as AWS)

[askvortsov1]

So far, I can't find a way to upload my avatar to other places (such as AWS)

This isn't (and won't) be supported out of the box, but an extension could add a filesystem driver for AWS, azure, etc. See https://docs.flarum.org/extend/filesystem.html

[mmm8955405]

到目前为止，我找不到将我的头像上传到其他地方（例如 AWS）的方法

这不是（也不会）开箱即用的支持，但扩展可以为 AWS、azure 等添加文件系统驱动程序。请参阅https://docs.flarum.org/extend/filesystem.html

Without a specific plug-in implementation, do you want others to modify the source code?
Why should avatars be separated from uploaded files? It's set to upload files to AWS S3 and deal with the path problem of avatars. Isn't it a crash? You just send a link to explain. Is it useful after reading it?

[mmm8955405]

You should reopen the problem, not turn it off

[mmm8955405]

Only primary school students and vegetable chickens can upload files to their own website directory

[askvortsov1]

Without a specific plug-in implementation, do you want others to modify the source code?

No. For situations like this, we recommend a custom extension. Core source code generally shouldn't be modified.

[mmm8955405]

Without a specific plug-in implementation, do you want others to modify the source code?

No. For situations like this, we recommend a custom extension. Core source code generally shouldn't be modified.

The problem is that this plug-in is not found!!! You write the source code, or you implement a??? Uploading to AWS S3 is a common requirement. You're avoiding the problem by turning off the problem directly. Think about a flarum website, migration and update; The picture is one in AWS and one in local. What a disgusting thing

[mmm8955405]

You're not really a student, are you? It makes me feel too weak

[askvortsov1]

The problem is that this plug-in is not found!!!

Right. This functionality will not be included directly in Flarum core. Instead, community members that need it are encouraged to make their own extension implementing a search driver, and providing an admin UI to configure settings.

This approach of a minimalist core is central to our development philosophy. See https://discuss.flarum.org/d/28869-flarum-philosophy-and-values and https://docs.flarum.org/extend/#core-vs-extensions for more information.

[mmm8955405]

Since all upload plug-ins can be uploaded to AWS, avatars cannot. If you wrote this function, why not implement one? It's painful to ask people in other languages to implement a PHP plug-in. And do you know the bug of your full-text search for special characters?