[Youtube] Download from playlist limited to the first 100 movies

[CWaterSon]

• I've verified that I'm running youtube-dl version 2021.03.03
• I've checked that all provided URLs are alive and playable in a browser
• I've checked that all URLs and arguments with special characters are properly quoted or escaped
• I've searched the bugtracker for similar bug reports including closed ones
• I've read bugs section in FAQ

Verbose log

command:
youtube-dl.exe -v --no-cache-dir -i --cookies youtube.com_cookies.txt --yes-playlist --playlist-start 101 --playlist-end 102 https://www.youtube.com/playlist?list=LL

[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['-v', '--no-cache-dir', '-i', '--cookies', 'youtube.com_cookies.txt', '--yes-playlist', '--playlist-start', '101', '--playlist-end', '102', 'https://www.youtube.com/playlist?list=LL']
[debug] Encodings: locale cp1250, fs mbcs, out cp852, pref cp1250
[debug] youtube-dl version 2021.03.03
[debug] Python version 3.4.4 (CPython) - Windows-10-10.0.19041
[debug] exe versions: none
[debug] Proxy map: {}
[youtube:tab] LL: Downloading webpage
[download] Downloading playlist: Filmy, które mi się podobają
[youtube:tab] Downloading page 1
[youtube:tab] playlist Filmy, które mi się podobają: Downloading 0 videos
[download] Finished downloading playlist: Filmy, które mi się podobają

Description

I have almost 300 movies in my playlist. Youtube only shows 100 in the list and dynamically loads the next part (next 100) as you scroll the page. Probably youtube-dl can't load more than the first 100.

Can you fix it?

P.S. [debug] Encodings: locale cp1250, fs mbcs, out cp852, pref cp1250
Why OUT is cp852? If locale is cp1250 and pref cp1250 too? I don't use any switch to change it.

[CHJ85]

No. That "load more" thing is just CSS. There must be a deleted video or something. Have you tried adding --ignore-errors?
And you can continue where you left off by using --no-post-overwrites to prevent re-downloading everything you just downloaded.
Or continue where you left off in the playlist with --playlist-start.

[CWaterSon]

Yes. Just look on parameters => "-i" and "--playlist-start 101"

[CHJ85]

Oh sorry. Must have overlooked that.

[formtapez]

Problem confirmed when trying to download the "watch later" playlist.
yt-dl loads only the first 100 entries.

[alcuin2k]

I am also having the 1 page 100 video issue with the youtube watch later playlist.

[CHJ85]

Here's a nice workaround for you guys.
There's this really neat extension/addon for both Chrome and Firefox called Link Gopher.
This addon will extract all the website links.
After you've installed the addon, just load the entire youtube playlist and click the addon icon, then click "extract links by filter", then type in "watch" (because all the video urls contain the word "watch"). This will give you a list of all the playlist links. Now just copy/past all these video links into an empty document, call it "links.txt". Then simply run "youtube-dl -a links.txt".

[coletdjnz]

The continuation pages for playlists/feeds that require auth are failing. I've been needing to report this issue since the 404 fix, but here's the TL;DR:
It seems like the browse API endpoint is requiring the following header:

• Authorization header containing a SAPISIDHASH (generated from SAPISID in the cookies files, this stackoverflow post describes how to generate it)
• I believe X-Origin header is required for this too

There is also a couple headers required when using a secondary channel account (otherwise the continuation tries to get the page under the main channels account!):

• X-Goog-PageId - first part of dataSyncId in yt initial data - if and only if there is 2 or more "ids" present. This is also present in ytcfg (either as DATASYNC_ID in the same format as or DELEGATED_SESSION_ID which is only present if on a secondary channel)
• X-Goog-AuthUser - edit: this is the SESSION_INDEX in ytcfg, it is related to what account is logged in when many accounts are logged in.

[Metacinnabar]

it would be great for this to be fixed :)

[SamHall14]

I tried the -i and the --playlist-start 101 would return the Error 410 and would start downloading 0 videos.
Edit: this is on the 4.1.2021 build.

[BBaoVanC]

This is happening for me too, I am trying to download my liked videos, and I am using cookies (since you can't make liked videos public anymore)

[Starwalker98]

Issue is still present :( with latest build :(

[valdas]

For me it downloads 1st page consisting of 121 videos. I have 5000 in them. What so special about that "Watch Later" playlist that google overcomplicates the matter and does not provide at least similar "Add all to" another playlist... Billion dollar business cannot implement simple function. Sorry for emotions.

[e-d-n-a]

Post with solution in #28788 ! :)

[schmitmd]

I must be missing something. I popped the following in there, but still see just 100 vids. I am using a download archive txt file, maybe that's the problem...? I'd really like to keep that though, since the playlist in question has thousands of vids in it... Python 3.7.9

        data = {
            'context': context,
        }

        origin = 'https://www.youtube.com'
        cookies = self._get_cookies(origin)
        SAPISID = cookies.get('SAPISID').value
        headers['origin'] = origin
        headers['x-origin'] = origin

        url_extr = re.compile(r'<link rel="canonical" href="(.+?)">')
        match = url_extr.search(webpage)
        referer = match.group(1)
        headers['referer'] = referer

        for page_num in itertools.count(1):
            if not continuation:
                break

            ts = str(round(dt.now().timestamp()))
            hash = SHA1(bytes(' '.join((ts, SAPISID, origin)),'utf8'))
            auth = f'SAPISIDHASH {ts}_{hash.hexdigest()}'
            headers['authorization'] = auth

            if visitor_data:
                headers['x-goog-authuser'] = 0
                headers['x-goog-visitor-id'] = visitor_data
            data['continuation'] = continuation['continuation']
            data['clickTracking'] = {
                'clickTrackingParams': continuation['itct']
            }

#        for page_num in itertools.count(1):
#            if not continuation:
#                break
#            if visitor_data:
#                headers['x-goog-visitor-id'] = visitor_data
#            data['continuation'] = continuation['continuation']
#            data['clickTracking'] = {
#                'clickTrackingParams': continuation['itct']
#            }
            count = 0
            retries = 3
            while count <= retries:
...

[e-d-n-a]

So you also inserted the additional imports at the top!? (well, should throw errors otherwise)
And you use cookies from an inactive logged-in browser session, right? (well, you get 100 entries of a private PL)

I also double-checked, that I didn't miss any other relevant changes!
If you have a regular playlist-URL it should definitely work, you could also try another 100+ playlist, like WL or something.

Otherwise you should be a bit more specific, what command and what type of URL you used.
Processing steps could be different!

P.S.: Also those commented lines in your code below the change seem very suspicious!
They seem to break the indentation level and might end the function there!?
Try cutting them out and make sure the function is continuous.

[schmitmd]

I did indeed add the imports and I grabbed the latest cookies from a logged-in session right before running the command. It is a private playlist, but does that mean I'm always stuck w/ 100 vids? The comments are just the original code, to show what I did/did not add. They're comments, so I don't think the spacing matters... Regardless, I tried taking them out with no change. Command I ran was:
/usr/local/bin/youtube-dl --verbose --config-location /media/Nextcloud/.config/youtube-dl/config --cookies /media/Nextcloud/.cookies.txt --download-archive /media/Nextcloud/files/tracking_files/songs_n_such2021_tracking_archive.txt "https://www.youtube.com/playlist?list=XXX"

[e-d-n-a]

Well, it's meant to make private playlists work beyond 100 entries!
Did you remove those commented lines?

You can try my modded files (use "with debug outputs") to see more of what's going on.

It has all variants with and without debug output and also original files for youtube-dl version 2020.06.06.

[schmitmd]

Ended up cut & running with yt-dlp instead. Solved by circumvention I guess. Thanks for looking and for the modded files. I'll hang onto them.

[dirkf]

Continued in #28788.