Visual Segments: Blocking out social media icons and other intrusive visual content

[LoganDark]

SponsorBlock is invaluable when a portion of a video is entirely unwanted. However, some creators prefer to sprinkle subscribe buttons and social media icons throughout their video, during otherwise unrelated speech or visual activity.

Trinkets like those prefer to collect in the most inconvenient places, in the middle of useful or engaging information. Viewers generally still want that information, so skipping the section completely would not be very helpful. Example of a video utilizing trinkets: the server advertisement is a dedicated section that can be cut out with a self-promotion segment, which I have already submitted. However, the twitter trinket, which is followed by an instagram trinket, occurs in the middle of speech/gameplay that should not just be cut out.

I would like to propose a new feature for SponsorBlock: visual segments. Visual segments are not to be skipped like other segments are, but they represent a time window for a particular trinket - defined as a piece of intrusive visual content, like a subscribe button or social media icon, present during an otherwise engaging or important part of the video.

Each visual segment would contain exactly one obstruction, which would be overlaid on top of the video content for the duration of the segment. Of course, it's not nearly as seamless as completely skipping an obnoxious section, and it's pretty obvious that there's something beneath the obstruction - but viewers won't have to see what's behind it if they don't want to, and they will still not miss out on an important section of the video.

Obstructions are solid shapes (consisting of a bounding rectangle and the name of the primitive to draw, such as 'box' or 'circle'). The submitter of the obstruction can define a solid color, which defaults to black, but that's it. Obstruction bounds would be defined using percentages.

Viewers can enable or disable obstructions just like annotations. Viewers can also, in the extension settings, select if they would like to overlay the defined obstruction color, or blur obstructed parts instead (achievable with backdrop-filter).

The only obstruction shapes to start would be box (square/rectangle) and circle (circle/ellipse). There's no need for fancy shapes like stars or arbitrary polygons or even rounded rectangles.

To define a visual segment, you would first select its type as 'visual segment', then you are given a basic plotting UI to define one (and only one) obstruction. It's probably up for debate how to plot multiple overlapping visual segments at once; you'd likely only do one at a time by selecting which one the plotter is using. It sounds like it would be rare to see multiple social media icons on different parts of the screen at the same time... but it's actually not so rare to see, for example, one icon in each corner, during outros and such.

Resolved problems:

• Little trinket problem - Little subscribe buttons or social media icons can be blocked out

Unresolved problems:

• Moderation - what's to stop someone from using these to draw out text? Or completely block out a video? The voting system, but someone will have to suffer and know how to downvote the segment. There is no neural network to determine if there is currently a social network icon on the screen or not. It's not even that simple because a trinket could just be text that says "remember to subscribe".
• Large trinkets - Skeppy is known for this. They have a gigantic, elaborate subscribe animation that they paste smack dab in the center of their video content while doing other things. Obstructing this would be hugely detrimental because it would actually completely ruin visibility. Usually these sections can be cut out but not always. They can be pretty long and cut deeply into the backstory of the video.
• Many aspects of the plotter - it is mostly undefined
• What will this mean for SponsorBlock on non-Chrome platforms? YouTube Vanced?
• If SponsorBlock should allow segments to block out a portion of the video but not the audio, what about vice versa? I don't think there should be audio blocking segments... but that is definitely a debate that might happen (also would be nice to block out obnoxious music during timelapses and such without having to manually operate the mute button - multiple categories of audio blocking? heck)
• This extends SponsorBlock in a potentially non-backwards-compatible way and violates its current guarantee of only providing occasional skips (by adding visual obstructions to that as well)

Right now the questions largely outweigh the answers, so this is mostly a request for comment at the moment. I'd love to work out a more complete architecture for this in a way that most benefits the SponsorBlock project. I'd love for the extension developer(s) to weigh in on this and provide feedback, and I'd love to see a feature like this added to SponsorBlock eventually. Thank you for reading :D

[ajayyy]

Semi-related: #96

[LoganDark]

Yeah, probably. This is a bit more fleshed out and dedicated to one feature. A good format for a GitHub issue, I'd imagine.

[ajayyy]

Yep, just want to link previous discussion

[LoganDark]

Well, what do you think?

[LoganDark]

:(

[ajayyy]

I definitely think this is a cool idea, but it may probably is a little out of scope right now. Maybe as a long term goal.

Now that SponsorBlock is old, it does have many people with high reputation, so "dangerous" features like this and #409 can be limited to only those with very high reputation

[ajayyy]

To work best, they would probably have to be replaced with an image, not just a shape, but that might be too much work to be viable.

Some way to have the "annotations" move smoothly across the screen over time might be needed as well.

[LoganDark]

So far I've submitted 156 segments and saved others about 60 hours. Does that count as high reputation? "Very high reputation" worries me, I wouldn't set the bar too high. I've been submitting segments for about a month so far, on pretty much every video I watch.

To work best, they would probably have to be replaced with an image, not just a shape, but that might be too much work to be viable.

Instead of specifying a solid color, you could have the extension take a screenshot of that portion of the video, and display it for the duration of the visual segment. It's easily something that could happen in an update, rather than on first release.

Some way to have the "annotations" move smoothly across the screen over time might be needed as well.

Careful with this. I intentionally kept this as primitive as possible, to hopefully prevent revealing more than necessary about the underlying distraction.

This is one of the only extensions to the idea I would theoretically approve of (if I were the one calling the shots, which I'm not), but again like other aspects of the 'plotter' the way this would be presented to the editor is completely undefined. You could implement an entire keyframe system, or just have a start and end point and an animation curve (probably the way I would do it). Allow the curve to be edited manually, but provide a couple presets. This adds additional complexity, beware.

Of course, this obstruction system doesn't even touch things like sound effects... but it would be much more trouble than it's worth to try to hammer out a solution to somehow lessen that without affecting the surrounding audio in a negative way. It would also require SponsorBlock to start processing all audio which comes out of the player which is probably even more complicated than visual segments would be. Mute segments are mildly less complex than that, but still not optimal.

Finding the optimal solution is most likely not possible, and like the rest of SponsorBlock, a compromise needs to be found.

[LoganDark]

Keep in mind that once most of the concerns are worked out, since I'm a programmer myself I could look into implementing this all as a pull request. I just don't want to put work into a plotter that you don't like, or an animation system that you don't like.

[ajayyy]

I agree with your comment about audio, I don't think that is something feasible, especially with just a browser extension.

[ajayyy]

I feel like this may be useful getting the opinions of others on this. Should I make a channel/room on Discord/Matrix to discuss this?

[ajayyy]

Also, on the example link you showed in the first post, I'm not sure just covering it with a screenshot of the video will help much.

[LoganDark]

I feel like this may be useful getting the opinions of others on this. Should I make a channel/room on Discord/Matrix to discuss this?

Sure.

Also, on the example link you showed in the first post, I'm not sure just covering it with a screenshot of the video will help much.

It also won't help much to ask users to invent their own image to overlay. Not to mention the abuse that could bring.

You'd also either have to store every image that they upload, or have their computer make requests to an arbitrary url (BAD BAD BAD BAD), or proxy it through your backend (less bad but still dangerous)

[ajayyy]

You'd also either have to store every image that they upload, or have their computer make requests to an arbitrary url (BAD BAD BAD BAD), or proxy it through your backend (less bad but still dangerous)

While anonymous image upload is scary, I'm not sure if there is another viable way. We've already experienced severe ratelimiting just downloading captions for use with https://github.com/andrewzlee/NeuralBlock and had to shut it down until we find another solution.

I doubt we would be able to download videos on the server to be able to take screenshots server-side, they would have to be done client side.

Could you send an example of what you think the example video should look like to hide the obstructions?

[LoganDark]

While anonymous image upload is scary, I'm not sure if another viable way.

You could not have users provide custom images? That's kind of what I was hinting at in my comment.

I doubt we would be able to download videos on the server to be able to take screenshots server-side, they would have to be done client side.

That's not at all what I mean. The client is already playing the video, so when a visual segment is encountered they could just display a portion of the frame that it started on, on top of the video until the segment is over.

It's hard to explain over text, but let me try to show what I mean.

Control
Solid color
Persist

This is not a job for the server anyway. The server should only store data about the shape of the obstruction.

This could be what you were thinking of when you said "custom images". As a plus, the server does not have to do any extra work, and you don't have to think about image uploads. You do have to work out how to display that, though... possibly problems when you skip to the middle of a visual segment and the client does not have that frame in memory. You might just have to use black in that case.