Support more filetypes / arbitrary file import

[glael]

Some screenshots from the discord

Since PSD files are already allowed, there is probably no reason not to allow these:



Same, .wav files are not very different from mp3 files:

You get the idea:

The actual issue

Since this gets requested a lot, it is probably worth discussing arbitrary file imports (i.e. allowing any file to be imported, without checking if it's "allowed" first):

This is potentially dangerous though, since the act of opening a file sometimes changes the hash of the file. (for example: when opening a .epub file with calibre, it adds/changes the META-INF/calibre-bookmarks.txt file inside the epub/zip.
This would break hydrus' hash-based system.

So: please discuss: "should hydrus prevent users from importing such files?"

[Zweibach]

In my opinion Hydrus should allow all "supported" formats through with no problem but give an explicit warning about how things can go pearshaped with formats likely to be changed by being used on at least the first import of such. If things do go pearshaped afterwards then it's explicitly on the user's head. And probably also a warning for all formats it doesn't recognise or know how to group (see system:filetype and how it groups image formats together).

[CuddleBear92]

Noting some 3D formats as people have requested it in the past:

.fbx
.obj
.daz
.stl
.3ds
.mdl
.sdl

There is many more formats for 3D use.... a 3D viewer is prob needed if one or more of these are added.

[ShadowJonathan]

Maybe MIME type could be tried to be inferred from file content/import details/etc, and users getting a warning/amend dialog if a MIME type cannot be asserted?

If the whole range (or at least a large subset) of standard MIME can be recognized and imported right off the bat, it'd instantly support a wide range, and later on make it easy to add more types.

[imtbl]

Maybe MIME type could be tried to be inferred from file content/import details/etc, and users getting a warning/amend dialog if a MIME type cannot be asserted?

If the whole range (or at least a large subset) of standard MIME can be recognized and imported right off the bat, it'd instantly support a wide range, and later on make it easy to add more types.

That's already the case, Hydrus uses magic numbers to determine file types. The limited support is more of an intended limitation atm and not really a technical one afaik.

[walramb]

This is potentially dangerous though, since the act of opening a file sometimes changes the hash of the file. (for example: when opening a .epub file with calibre, it adds/changes the META-INF/calibre-bookmarks.txt file inside the epub/zip.
This would break hydrus' hash-based system.

I'm not familiar with that software in particular, but couldn't a lot of mangling of this sort be prevented by having hydrus set all its media file permissions to read-only?

[bbappserver]

My understanding of file identification is that it was currently relying on some mix of magic number filetype detection and ffmpeg.
https://pypi.org/project/filetype/

For weird filetypes it is usually the responsibility of http to provide a Content-Type

I think marking file which could not be identified by either of these means just becoming application/octect-stream is fine, and you can manually adjust on your end. MIMEs basically exist because extensions are unreliable, but you could let the user whitelist the problematic extensions and assign them a mime, so they don't have to go in and manually reassign things.

[pozieuto]

One of those Discord screenshots implies that filetypes without visual media viewer support can't have thumbnails. I don't see why that would need to be the case. Some applications generate their own thumbnails for their own filetypes. For example, LibreOffice has a Windows Explorer extension that generates thumbnails for documents - could Hydrus just grab those thumbnails for those files?

Audio files could have thumbnails generated based on their waveform. ffmpeg could be used for this.

[floogulinc]

Hydrus does currently generate thumbnails for a file type it can't view, namely swf.

The issue of hashes changing is interesting. What is the behavior currently if I say edit an image form Hydrus in paint or Photoshop and save it. Does Hydrus recompute the hash or just keep thinking the file has its original hash?

[imtbl]

Does Hydrus recompute the hash or just keep thinking the file has its original hash?

The latter at first, but once file maintenance runs and Hydrus checks the hashes, it will think the file is invalid, see screenshot.

That means, editing files that have been added to Hydrus is currently not really a thing. Keeping track of potentially of millions of files changing is also not that trivial afaik. While Hydrus is running, you could watch for filesystem events. But if changes are made to a file while it's closed, Hydrus would need to potentially recheck every file on startup (which is not feasible I think).

Of course, you could do things like calculating hashes for groups of files and then checking those first (and if a hash is wrong, then check each file of the group, basically a chunked approach). I'm not really an expert on this. The way Git does it might also be feasible, but I don't know how well that would perform with this many (and potentially huge) binary files.

I proposed a simpler workaround in Discord that could also work, but surely isn't as ideal or as desirable:

[bb010g]

If the backing filesystem supports them, snapshots could be used, but that's a weird complexity. Implementation-wise, Windows's Volume Shadow Copy APIs can be not so fun too. (Directing VSS to only copy a specific directory isn't obvious. ZFS or btrfs should be far easier, though.) https://docs.microsoft.com/en-us/windows/win32/vss/using-the-volume-shadow-copy-service

[Zweibach]

Open Office XML document support as per #362 , see this comment in that issue for more information.