Player seekbar thumbnail preview

[litetex]

What is it?

• Bugfix (user facing)
• Feature (user facing)
• Codebase improvement (dev facing)
• Meta improvement to the project (dev facing)

Description of the changes in your PR

Click here to show the demo (screenshots + video) data
(it's a lot so it's collapsed by default)

• Full demo video:

demo.compressed.mp4

• Screenshots:
  • Settings:
    
  • Video seekbar thumbnail preview (high quality):
    
    
  • Video seekbar thumbnail preview (low quality):
    
    
  • Video seekbar thumbnail preview (off):
    

Implementation details (click to open)

These are the major parts of this implementation:

(Meta)-Data
The metadata of a video contains a new field (the extractor had to be modified for this → TeamNewPipe/NewPipeExtractor#647 )

• This field consists of a list of framesets (yt currently returns 2)
• The framesets have different preview thumbnail sizes (that's why there exists the option low and high quality in the settings)
• A frameset has urls to the preview thumbnails; a url returns an ("base") image with multiple preview thumbnails, e.g.
  

How it get's into the seekbar

1. Preparing the data

The metadata is changed in the player, when a new video is played/started.

When this happens the urls get asynchronously fetched and processed, which happens mostly in SeekbarPreviewThumbnailHolder

2. Seekbar positioning

Note: The images from the url are not "pre-processed". The extraction of the corresponding preview thumbnail and resizing happens when the seekbar is changed.

When the seekbar is moved the closest image to the current position (in ms) is searched and returned.
If none is found or an error occurs the thumbnail will simply be hidden.

After thumbnail was extracted from/cut out of the "base" image (from the url of the frameset) it will also be resized:
The size of the final thumbnail is calculated as follows:

• absolute min-width: 100dp
• (preferred) width: player-width / 4
• absolute max-width: 2.5 times the original width (else it becomes too pixelated)

Settings
New option"seekbar thumbnail preview" (under Settings>Video and audio>Player)
Can contain the following values:

• "High quality (larger)" - default
• "Low quality (smaller)"
• "Don't show" (a thumbnail)

The settings will be applied when the next video is started / opened or more specific when the metadata of the player is changed

Fixes the following issue(s)

• Fixes Thumbnails preview on seek bar #3207

Relies on the following changes

• Code changes to enable player thumbnail seekbar preview in NewPipe NewPipeExtractor#647

APK testing

On the website the APK can be found by going to the "Checks" tab below the title and then on "artifacts" on the right.

You may also build/test it with this manual (click to open)

• Prepare a new folder
• Checkout https://github.com/litetex/NewPipe/tree/playerSeekbarPreview
• Checkout https://github.com/litetex/NewPipeExtractor/tree/playerSeekbarPreview
• Make sure both of these projects have the same parent folder
• Uncomment

  includeBuild('../NewPipeExtractor') {
      dependencySubstitution {
          substitute module('com.github.TeamNewPipe:NewPipeExtractor') with project(':extractor')
      }
  }
  

  in settings.gradle
• Launch the app using Android Studio or build the apk

Due diligence

• I read the contribution guidelines.

Things that should be checked 👁️‍🗨️

Technical

I implemented these changes as good as i could think of.
However there the following topics/parts of the code should be checked carefully by the reviewers, because if it fails it may cause excessive problems:

• thread management (a relatively simple ExecutorService is currently used, maybe there are better ways)
• handling of the bitmaps (check for memory leaks)
• Check the thumbnail showing logic / calculation where the seekbar preview container (thumbnail + currentTime) should be

Other behaviors and implementations that should be looked upon

• The resizing of the thumbnail logic - I tested that on "normal" devices (Google Pixel 3a emulated), maybe some devices like tablets or TVs respond differently and the thumbnail is too small or large

[skyGtm]

since it shows thumbnail previews, would it be possible to implement this #6383 ?

[opusforlife2]

Thank you so much, @litetex! This looks awesome! And it is also one of those big features that people miss when they shift to Newpipe.

Feedback: Why are the menu options called High and Low Quality when it is actually the Preview Size that is changing? They should be called Large and Small Preview or something instead. Both with High and Low Quality, the preview images are pixelated. At least on a 1080p screen. So as a result it looks like the Large and Small Previews actually have the same quality in proportion, but for different sizes.

[litetex]

@skyGtm

since it shows thumbnail previews, would it be possible to implement this #6383 ?

For now this pr will only fix #3207 and - to not mix things up and create a monolith/"all-in-one pr monster" - #6383 should be handled as a separate feature and therefore be done in a separate pr 😄

[litetex]

@opusforlife2

Why are the menu options called High and Low Quality when it is actually the Preview Size that is changing? They should be called Large and Small Preview or something instead. Both with High and Low Quality, the preview images are pixelated. At least on a 1080p screen. So as a result it looks like the Large and Small Previews actually have the same quality in proportion, but for different sizes.

Thank you for the feedback 😄

I called them high and low quality, because yt returns 2 storyboard/seekbar thumnail sizes - high and low quality.
However I'm not quite sure how to implement these the best way and through that I used that formula:

The size of the final thumbnail is calculated as follows:

• absolute min-width: 100px
• (preferred) width: player-width / 4
• absolute max-width: 2.5 times the original width (else it becomes too pixelated)

I may be revising this to make it work better with different devices (tablets, TVs, smartphones).

Currently this way it looks (at least) with "high quality" somewhat similar to the current YT behavior (on the website + app).

I think renaming these would be a good idea, I will do that when I have time again 😄

[opusforlife2]

Ohkay. I don't know what it looks like on Youtube, but if that's the highest possible quality, then maybe try making the preview a little smaller? The smaller preview is already quite small, though. Hmmm. Maybe leave the smaller one as it is.

Well, we'll need to try a few things, probably.

[litetex]

A few updates:

• Changed the settings to:
  
• The min-width of the thumbnail is now 100dp (so that it's more consistent)
• Made sure that it works on tablets and tvs:
  
  

[opusforlife2]

Well, I wanted to test with the latest changes, but the CI job failed.

[litetex]

Well, I wanted to test with the latest changes, but the CI job failed.

Occurs because underlying extractor pr is missing / not merged:
TeamNewPipe/NewPipeExtractor#647

I built a new apk:
debug.zip

Note: The manual build guide as stated in the PR description above also works :)

[opusforlife2]

@litetex Are you not supposed to update the Extractor version to the one that corresponds to your Extractor PR's latest commit? I may be wrong, but I think that's the general procedure.

[litetex]

@litetex Are you not supposed to update the Extractor version to the one that corresponds to your Extractor PR's latest commit? I may be wrong, but I think that's the general procedure.

I can do that but, this will point to my repo not NewPipe's so guess the person who merges that has to fix it 😆
Will also update the branch from develop so that the conflicts are gone

[Stypox]

I did not review carefully, but from a first look the code is a beauty! Thank you :-D

[litetex]

@Stypox
Thank you for the review 😄

I rebased the complete branch and applied the requested changes.

[Stypox]

Code looks good to me, but I would also like someone else to review. Thank you @litetex for the hard work! I tested and everything seems to work well on API 29 (real phone), API 29 (emulated TV), API 30 (emulated tablet). I was unable to test on API 19 since in the emulator the codecs to open videos are missing.

[opusforlife2]

I just noticed that this replaces the seek time in the centre of the player with one below the preview thumbnail. I think we should try keeping the seek time in the centre like before. Below the thumbnail, it might be too tiny for some people to read.

[opusforlife2]

so guess the person who merges that has to fix it 😆

That's standard procedure, don't worry.

[litetex]

@opusforlife2

I just noticed that this replaces the seek time in the centre of the player with one below the preview thumbnail. I think we should try keeping the seek time in the centre like before. Below the thumbnail, it might be too tiny for some people to read.

I wouldn't do that, as both the YT app and the website use the same positioning as introduced with the PR:

This could also cause problems on e.g. smaller screens, because the "seek time in the centre" and the thumbnail could maybe overlap.
It also breaks the visual unity of the both elements.

Note: The start- and endtime also have the same size and nobody seems to have problems with them

[opusforlife2]

People don't really need to care about the current time and end time. But when seeking, they may want to focus on the seek time. How about putting it above the thumbnail instead of below, and making it bigger?

[litetex]

People don't really need to care about the current time and end time.

I wouldn't say that.

But when seeking, they may want to focus on the seek time.
How about putting it above the thumbnail instead of below, and making it bigger?

Okay let's come to an agreement here: I will make the text a bit larger.

Note: The only video streaming providers implementing a seekbar preview are currently YT and media ccc

Both show the seek time below (or below inside) the preview thumbnail.
I will keep therefore that standard behavior.

[Mhowser]

I think the seek time should be in the lower left corner instead of the center.

[Stypox]

I think we should not reinvent the wheel and just keep the standard behaviour. Making the text a little bit bigger is a good compromise, but I would not make anything different.

[litetex]

Enlarged the currentDisplaySeek-text

Demo (click to open)

[Stypox]

@opusforlife2 @litetex Placing the seek time below the thumbnail looks more natural in my opinion, maybe just because that's the usual way. On the other hand placing the seek time above has multiple advantages: making the text more readable and having the thumbnail overlay cover a smaller portion of the player below. So I'd go with seek time above, at the end of the day.

Though I do think it would look better with that shadow extended to the width of the thumbnail.

I'd rather not do this as it would cover more of the player below. It looks just fine as it already is imo.

[litetex]

@Stypox
@opusforlife2

Guess the majority wins.
Then let's move the time to the top 😄

Demo pictures (click to expand)

Update:
Also increased the bottom padding a bit, so people with bigger fingers can see the thumbnail 😉

[Stypox]

The behavior in the latest apk looks polished and works well in my opinion

[opusforlife2]

Yay! Looks awesome! Thanks, @litetex!

[Stypox]

@litetex please rebase one last time, then I will do one last round of review and merge this :-)

[litetex]

@Stypox
Rebasing done

[Stypox]

Code looks good to me, thank you for the dedication! I tested on my API 29 phone and on emulated API 19, API 21, API 29 (tv), API 30, API 30 (tablet). I tried both fullscreen and not-fullscreen. (on API 19 and 21, though, I could not find any playable video with a frameset: newer videos do not play because of missing decoders and "me at the zoo" has no frameset)

[litetex]

(on API 19 and 21, though, I could not find any playable video with a frameset: newer videos do not play because of missing decoders and "me at the zoo" has no frameset)

If I have time, I will test the current develop on my old Android 4.4 device. Last time I tested it (about 2 weeks ago) everything was looking fine.