Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[YouTube] Workaround again 403 HTTP issues on the ANDROID InnerTube client and fix stream tests #1084

Conversation

AudricV
Copy link
Member

@AudricV AudricV commented Jul 24, 2023

  • I carefully read the contribution guidelines and agree to them.
  • I have tested the API against NewPipe. No, but I tested my changes with a visitorData on which the 403 HTTP issues was present before and they didn't occur anymore with the new parameters.

This PR works around again 403 HTTP issues when getting streams from the ANDROID InnerTube client, by using new parameters to bypass this client's integrity checks.

These parameters are the only ones currently known to bypass the 403 HTTP issues, as the ones of stories (and the base of the shorts ones) do not work anymore (currently A/B tested), which may be related to end of this format on the service.

I also fixed stream test failures, which were not present in the existing mocks but were present when I updated stream mocks (required due to the player parameters update of the ANDROID client).

I had especially to change the video used in YoutubeStreamExtractorDefaultTest.PublicBroadcasterTest test class, as the channel Dinge Erklärt – Kurzgesagt is not affiliated with a public broadcast channel anymore.

I chose the 3 million subscribers thanks video from ARTE French YouTube channel (cJ9to6EmElQ) as a replacement.

Fixes TeamNewPipe/NewPipe#10174.

…sing new player parameters

These parameters are the only ones currently known to bypass 403 HTTP issues
related to failure of passing Android client integrity checks, as the ones of
stories (and the base of the shorts ones) do not work anymore, which may be
related to end of this format on the service.
- Fix testCheckAudioStreams test of
YoutubeStreamExtractorDefaultTest.AudioTrackLanguage test class, by updating
the excepted audio track name test to use the updated English audio track name
(audio track type info has been added on the video tested);
- Fix YoutubeStreamExtractorDefaultTest.PublicBroadcasterTest test class by
using a different video from a French and German public broadcast channel, as
the channel Dinge Erklärt – Kurzgesagt is not affiliated with a public
broadcast channel anymore;
- Fix YoutubeStreamExtractorLivestreamTest test class, by updating the excepted
name of the livestream to the current one.
@AudricV AudricV added bug Issue is related to a bug youtube service, https://www.youtube.com/ labels Jul 24, 2023
@SavSanta
Copy link

Is there a samp apk while waiting for checks?

Also if this is A/B related how would you suggest testing this. I've been getting the errors lately but I also have 2 other older previous debug Newpipe versions (for various issues). When the 403 pops up I can share the video to one of those and generally the issue doesn't appear.

Cheers

Copy link
Member

@Stypox Stypox left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If this works then LGTM, it's only a small change that makes sense

@TobiGr TobiGr merged commit 39a911d into TeamNewPipe:dev Jul 31, 2023
1 check failed
@AudricV AudricV deleted the yt_android-403s-workaround-and-streams-tests-fixes branch July 31, 2023 22:58
syeopite added a commit to syeopite/invidious that referenced this pull request Aug 7, 2023
YouTube appears to be A/B testing some new integrity checks. Adding the
parameter "CgIQBg" to InnerTube player requests appears to workaround
the problem

See TeamNewPipe/NewPipeExtractor#1084
@joshuabengal
Copy link

This PR works around again 403 HTTP issues when getting streams from the ANDROID InnerTube client, by using new parameters to bypass this client's integrity checks.

These parameters are the only ones currently known to bypass the 403 HTTP issues, as the ones of stories (and the base of the shorts ones) do not work anymore (currently A/B tested), which may be related to end of this format on the service.

Works great, my question is how/where was the new param setting ("CgIQBg") found? I could not find any list of valid param settings in the YouTube API documentation.

Basically if at some point CgIQBg stops working, where do we look for another valid param value?

@Bashiraljomma

This comment was marked as off-topic.

@Bashiraljomma

This comment was marked as off-topic.

@opusforlife2
Copy link
Collaborator

@Bashiraljomma English only. Use Google Translate.

@Pheb62

This comment was marked as off-topic.

@TeamNewPipe TeamNewPipe locked as resolved and limited conversation to collaborators Sep 25, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
bug Issue is related to a bug youtube service, https://www.youtube.com/
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[YouTube] 403 HTTP errors while playing/downloading videos (workaround broken)
8 participants