Video deduplication support

[HASJ]

[AeliusSaionji]

I think that's what he does for gif already, so I'd say yes.

[imtbl]

Hey, issues here have been reenabled and will be used for bug reports/feature requests in the future.

We are currently clearing out all the existing issues from 2017 and prior.
If your bug report or feature request is still relevant, please create a new issue (or use one of the other channels; see https://hydrusnetwork.github.io/hydrus/).

[bbappserver]

@CuddleBear92
Hydrus can't pdiff image sequences, this is still an issue.

[imtbl]

@bbappserver Reopened.

[pozieuto]

I think there is a way that exact duplicates could be detected very easily: hashing the raw audio and video bitstreams. This would detect files with identical audio and video streams that are just in different containers. It would also detect files that share at least one audio or video stream while still having other streams that differ or are absent. (I have had cases of this, usually when a video had audio added to it later.)

[CuddleBear92]

Well the current HyDev plans IIRC is to pull "interesting frames" from the video and compare them.
Not sure what that means to be exact. Its not just i-frames.

Pulling i-frames is prob also a decent option.

Going for video and frames first makes the most sense as not everything has audio.

[imtbl]

I think there is a way that exact duplicates could be detected very easily: hashing the raw audio and video bitstreams.

As you said, this would only work in a very limited way for exact duplicates; but we also want to be able to detect if, e.g., a 10 second GIF is part of a 3 minute music video encoded in H.264.

Implementing a generalized solution that should work for animated image sequences of any type and length (like the one HyDev is already planning, as Cuddle mentioned) imo makes the most sense here.

See also https://link.springer.com/article/10.1186/s13640-019-0442-7 (similar concept to what HyDev is planning, I think; though in a distributed approach).

[ShadowJonathan]

Well the current HyDev plans IIRC is to pull "interesting frames" from the video and compare them.

Keyframes?