[YouTube] Use the new internal API in NewPipe Extractor #604

AudricV · 2021-04-09T14:28:59Z

I carefully read the contribution guidelines and agree to them.
I have tested the API against NewPipe.
I agree to create a pull request for NewPipe as soon as possible to make it compatible with the changed API. - Nothing should be changed in the app side with these changes in the extractor.

Use the new internal API called innertube or youtubei (https://www.youtube.com/youtubei/v1/endpoint?key=INNERTUBE_API_KEY) to fetch informations of YouTube contents instead of using the pbj JSON (https://www.youtube.com/webpage_endpoint?pbj=1; for YouTube Music, nothing were changed because the search already uses the InnerTube API). Responses are pretty similar (most of time, the order of the objects is just changed), so this should not be a big work like it was in 2020 for the migration from the old HTML YouTube pages to the desktop polymer version and its pbj JSON. ~~This pbj seems to be deprecated, the desktop website if YouTube is only using this API for video comments right now (there are A/B tests with the next endpoint right now).~~

The changes in this PR needs testing for exceptions due to a big traffic and if the API returns 429/Too Many Requests, support of this in the extractor needs to be check (maybe sending cookies generated by a captcha to this request on high network traffics should bypass this error code).

Extraction of comments is fixed with this PR, like the extraction of embeddable age-restricted videos.

Improvements made in this PR:

before this PR, in order to get continuations of a channel, the initial page was always fetched. That's not the case anymore with a workaround using the ids field of the Page class, see e0011de
use a lightweight request to check if hardcoded versions are valid:
- for YouTube, use the guide endpoint, which returns the menu items of the website
- for YouTube music, use the get_search_suggestions endpoint with an empty string suggestion, used by the website when loading it for the first time of a session.

Screenshot of a 403/Forbidden error message (this should only happen if your IP was banned by Google):

Endpoints changed:

channels (continuations improved, see above)
playlists
searches (for YouTube)
videos
mixes
comments

TO DO:

fix the fetch of the JS player at each time the extractor falls back to the desktop version (it should be normally cached).
~~add a better spoofing of the mobile API (by analyzing the requests made by the Android client, which uses protobuf)~~ It should be made in a separate PR.
add a method to reset debofuscationCode, playerCode, playerJsUrl and signatureTimestamp strings.
~~update mocks and client version when PR is approved.~~
readd the deleted code for views because it breaks the number of views in livestreams.

I also reformatted some code to be in the 100 characters line limit and used final where possible, in the files that I changed.

This will close #568 (even if that I still use the desktop version of the new internal API instead of the mobile version, excepted for the videos when I use to the Android API if a video is protected by signatureCiphers).
Thanks to @FireMasterK for his findings.

APK for testing

See AudricV/NewPipe#1 for an up to date debug APK.

(Sorry in advance for my English.)

XiangRongLin · 2021-04-09T16:36:29Z

@TiA4f8R Can you merge in the change more incrementally instead of all at once. Basically finish up the current channels, search, playlist extractor and get those merged or even split those up into seperate PR.

opusforlife2 · 2021-04-10T08:56:52Z

Then later, all the related PRs for this change can be linked to each other in the OP in a bullet list to make it easier to navigate for future devs.

extractor/src/main/java/org/schabi/newpipe/extractor/services/youtube/YoutubeParsingHelper.java

AudricV · 2021-04-21T18:45:45Z

@XiangRongLin I think I fixed all tests because the CI passed. What do you think?

extractor/src/main/java/org/schabi/newpipe/extractor/services/youtube/YoutubeParsingHelper.java

XiangRongLin · 2021-04-22T08:57:41Z

From the mock standpoint, if the tests are passing, then it should be fine.

…ocks

…deos + update clients version Here is now the requests which will be made by the `onFetchPage` method of `YoutubeStreamExtractor`: - the desktop API is fetched. If there is no streaming data, the desktop player API with the embed client screen will be fetched (and also the player code), then the Android mobile API. - if there is no streaming data, a `ContentNotAvailableException` will be thrown by using the message provided in playability status If the video is age restricted, a request to the next endpoint of the desktop player with the embed client screen will be sent. Otherwise, the next endpoint will be fetched normally, if the content is available. If the video is not age-restricted, a request to the player endpoint of the Android mobile API will be made. We can get more streams by using the Android mobile API but some streams may be not available on this API, so the streaming data of the Android mobile API will be first used to get itags and then the streaming data of the desktop internal API will be used. If the parsing of the Android mobile API went wrong, only the streams of the desktop API will be used. Other code changes: - `prepareJsonBuilder` in `YoutubeParsingHelper` was renamed to `prepareDesktopJsonBuilder` - `prepareMobileJsonBuilder` in `YoutubeParsingHelper` was renamed to `prepareAndroidMobileJsonBuilder` - two new methods in `YoutubeParsingHelper` were added: `prepareDesktopEmbedVideoJsonBuilder` and `prepareAndroidMobileEmbedVideoJsonBuilder` - `createPlayerBodyWithSts` is now public and was moved to `YoutubeParsingHelper` - a new method in `YoutubeJavaScriptExtractor` was added: `resetJavaScriptCode`, which was needed for the method `resetDebofuscationCode` of `YoutubeStreamExtractor` - `areHardcodedClientVersionAndKeyValid` in `YoutubeParsingHelper` returns now a `boolean` instead of an `Optional<Boolean>` - the `fetchVideoInfoPage` method of `YoutubeStreamExtractor` was removed because YouTube returns now 404 for every client with the `get_video_info` page - some unused objects and some warnings in `YoutubeStreamExtractor` were removed and fixed Co-authored-by: TiA4f8R <74829229+TiA4f8R@users.noreply.github.com>

Migrate YouTube comments to the desktop version by using the `next` endpoint of the InnerTube internal API. With the desktop version, we are able to get the exact like count of YouTube comments (by parsing the accessibility data) (the current extraction is used as a fallback). We are also now able to get if the uploader of the comment is verified or not. Co-authored-by: TiA4f8R <74829229+TiA4f8R@users.noreply.github.com>

…ions of YouTube Music search results The clickTrackingParams of YouTube Music search results are not needed to get continuations. This commit removes their use, which may improve privacy.

…ctorTest Without removing RunWith and SuiteClasses annotations (and the corresponding imports) in YoutubePlaylistExtractorTest and YoutubeMixPlaylistExtractorTest, some mocks cannot be generated, so the CI fails because of the missing mocks. Mocks of workings tests have been also updated.

Outdated review

Stypox

Looks good to me, finally! I'm gonna test some things and if I see nothing wrong I'll merge this and open a PR for the hotfix. Are you all ok with this?

Stypox · 2021-08-03T16:14:56Z

.../java/org/schabi/newpipe/extractor/services/youtube/extractors/YoutubePlaylistExtractor.java

+        final byte[] body = JsonWriter.string(prepareDesktopJsonBuilder(localization,
+                getExtractorContentCountry())
+                .value("browseId", "VL" + getId())
+                .value("params", "wgYCCAA%3D") // Show unavailable videos


Are we sure about this? What is the purpose of showing unavailable videos in playlists? YouTube does not show them normally, and in NewPipe they would just create problems. Anyway, we'll think about this later.

Before a YouTube update, they were shown every time, so I thought it may be useful for some users (premieres, temporary georestrictions (think to music releases), ...).

There was a button on YouTube to show them manually if you were logged in iirc.

On a technical side, this should be just base64 URL-encoded protobuf.

You don't have to be logged in to show them manually: it also works if you are a guest.

Stypox · 2021-08-03T17:01:10Z

Ok, I tested as much as I could all of the features I could think of. It works well. Thank you @TiA4f8R and @FireMasterK for your hard work :-)

AudricV added enhancement youtube service, https://www.youtube.com/ labels Apr 9, 2021

This comment has been minimized.

Sign in to view

AudricV force-pushed the youtubei-api branch 4 times, most recently from 0e69831 to 8022275 Compare April 11, 2021 14:26

AudricV changed the title ~~Use the youtubei API in NewPipe Extractor~~ Use the youtubei API in NewPipe Extractor for channels, playlists and searches Apr 11, 2021

AudricV force-pushed the youtubei-api branch from 9bd07d6 to 1563468 Compare April 11, 2021 16:20

AudricV marked this pull request as ready for review April 11, 2021 16:22

AudricV force-pushed the youtubei-api branch from 82dade0 to dc4b14d Compare April 13, 2021 10:55

AudricV marked this pull request as draft April 15, 2021 10:31

AudricV force-pushed the youtubei-api branch from dc4b14d to 9ae8161 Compare April 15, 2021 17:01

B0pol reviewed Apr 15, 2021

View reviewed changes

extractor/src/main/java/org/schabi/newpipe/extractor/services/youtube/YoutubeParsingHelper.java Outdated Show resolved Hide resolved

AudricV force-pushed the youtubei-api branch 3 times, most recently from b61d54f to 59e20e0 Compare April 19, 2021 17:16

AudricV marked this pull request as ready for review April 19, 2021 17:27

AudricV force-pushed the youtubei-api branch 2 times, most recently from 3eb2e9a to d607fe9 Compare April 21, 2021 17:48

XiangRongLin reviewed Apr 22, 2021

View reviewed changes

extractor/src/main/java/org/schabi/newpipe/extractor/services/youtube/YoutubeParsingHelper.java Outdated Show resolved Hide resolved

AudricV force-pushed the youtubei-api branch 3 times, most recently from 47dd511 to f7bca36 Compare April 29, 2021 10:28

AudricV and others added 15 commits August 1, 2021 12:39

Adress again reviews, fix some rebase issues

609919d

Update mocks

27bd797

Update clients version

8aa60d7

Workaround for rate limits: always use the Android mobile API

3adac6a

Ignore some broken tests

629dcd6

Update mocks

2320aec

Adress the last requested changes + update YoutubeCommentsExtractor m…

7753556

…ocks

Fix YoutubeStreamExtractorAgeRestrictedTest

d551092

Ignore a broken test of YoutubeMixPlaylistExtractorTest

693f654

Update mocks (except mocks of YoutubeCommentsExtractorTest)

286d839

Update YouTube comments and YouTube playlist continuations mocks

f8197da

Reformat some code and don't use the clickTrackingParams in continuat…

b74a39c

…ions of YouTube Music search results The clickTrackingParams of YouTube Music search results are not needed to get continuations. This commit removes their use, which may improve privacy.

AudricV force-pushed the youtubei-api branch from 7a6ff78 to 79cc5c8 Compare August 1, 2021 15:34

AudricV requested a review from Stypox August 2, 2021 10:30

Stypox approved these changes Aug 3, 2021

View reviewed changes

Stypox merged commit 5a88263 into TeamNewPipe:dev Aug 3, 2021

Stypox mentioned this pull request Aug 3, 2021

Hotfix release 0.21.8 TeamNewPipe/NewPipe#6834

Merged

AudricV mentioned this pull request Aug 3, 2021

java.lang.NullPointerException when opening an age restricted video TeamNewPipe/NewPipe#6726

Closed

4 tasks

AudricV deleted the youtubei-api branch August 4, 2021 11:07

triallax mentioned this pull request Aug 4, 2021

Trending page not loading TeamNewPipe/NewPipe#6839

Closed

SamantazFox mentioned this pull request Aug 5, 2021

[Enhancement] Show unavailable videos in playlists iv-org/invidious#2303

Closed

litetex mentioned this pull request Aug 24, 2021

Add ucbcb=1 parameter #669

Closed

3 tasks

litetex mentioned this pull request Oct 3, 2021

Pass additional regexes to youtube extractor #297

Closed

3 tasks

AudricV mentioned this pull request Feb 4, 2022

Remove deciphering alltogether #790

Closed

3 tasks

opusforlife2 mentioned this pull request Apr 19, 2022

Suggestions / recommendation tab (like YouTube) TeamNewPipe/NewPipe#2256

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[YouTube] Use the new internal API in NewPipe Extractor #604

[YouTube] Use the new internal API in NewPipe Extractor #604

AudricV commented Apr 9, 2021 •

edited

Loading

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

XiangRongLin commented Apr 9, 2021 •

edited

Loading

opusforlife2 commented Apr 10, 2021

AudricV commented Apr 21, 2021

XiangRongLin commented Apr 22, 2021

Stypox left a comment

Stypox Aug 3, 2021

AudricV Aug 3, 2021

FireMasterK Aug 3, 2021

FireMasterK Aug 3, 2021

AudricV Aug 3, 2021

Stypox commented Aug 3, 2021

[YouTube] Use the new internal API in NewPipe Extractor #604

[YouTube] Use the new internal API in NewPipe Extractor #604

Conversation

AudricV commented Apr 9, 2021 • edited Loading

APK for testing

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

XiangRongLin commented Apr 9, 2021 • edited Loading

opusforlife2 commented Apr 10, 2021

AudricV commented Apr 21, 2021

XiangRongLin commented Apr 22, 2021

Stypox left a comment

Choose a reason for hiding this comment

Stypox Aug 3, 2021

Choose a reason for hiding this comment

AudricV Aug 3, 2021

Choose a reason for hiding this comment

FireMasterK Aug 3, 2021

Choose a reason for hiding this comment

FireMasterK Aug 3, 2021

Choose a reason for hiding this comment

AudricV Aug 3, 2021

Choose a reason for hiding this comment

Stypox commented Aug 3, 2021

AudricV commented Apr 9, 2021 •

edited

Loading

XiangRongLin commented Apr 9, 2021 •

edited

Loading