Vrtnu running with username and password: age restriction not handled correctly

[tisik]

Checklist

• [ x] I'm reporting a broken site support
• [ x] I've verified that I'm running youtube-dl version 2021.04.26
• [ x] I've checked that all provided URLs are alive and playable in a browser
• [ x] I've checked that all URLs and arguments with special characters are properly quoted or escaped
• [ x] I've searched the bugtracker for similar issues including closed ones

Verbose log

[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['-v', '-u', 'PRIVATE', '-p', 'PRIVATE', 'https://www.vrt.be/vrtnu/a-z/petra/1/petra-s1a2/']
[debug] Encodings: locale UTF-8, fs utf-8, out utf-8, pref UTF-8
[debug] youtube-dl version 2021.04.26
[debug] Python version 3.8.5 (CPython) - Linux-5.8.0-53-generic-x86_64-with-glibc2.29
[debug] exe versions: ffmpeg 4.2.4, ffprobe 4.2.4, rtmpdump 2.4
[debug] Proxy map: {}
[VrtNU] Logging in
[VrtNU] Requesting a token
[VrtNU] petra-s1a2: Downloading webpage
[Canvas] pbs-pub-e3cd125e-b92d-4142-a860-dc819cdd9424$vid-c358d4e5-e805-4db1-8c92-ccc087ead5ca: Downloading token
[Canvas] pbs-pub-e3cd125e-b92d-4142-a860-dc819cdd9424$vid-c358d4e5-e805-4db1-8c92-ccc087ead5ca: Downloading video JSON
ERROR: Age category 0+ is not allowed to access 12+ content
Traceback (most recent call last):
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 806, in wrapper
    return func(self, *args, **kwargs)
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 827, in __extract_info
    ie_result = ie.extract(url)
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/common.py", line 534, in extract
    ie_result = self._real_extract(url)
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/canvas.py", line 80, in _real_extract
    raise ExtractorError(data.get('message') or code, expected=True)
youtube_dl.utils.ExtractorError: Age category 0+ is not allowed to access 12+ content

Description

Can not download age restricted video from https://www.vrt.be/vrtnu/ (e.g. https://www.vrt.be/vrtnu/a-z/petra/1/petra-s1a2/), although my account allows playing it.
An account can be made from the site: https://www.vrt.be/vrtnu/

[Nick4-1]

Same problem, even with v2021.05.16

[MatthiasCoppens]

It's got something to do with the vrtPlayerToken.
When watching https://www.vrt.be/vrtnu/a-z/bloodlands/1/bloodlands-s1a1/ on Firefox, the vrtPlayerToken is different from youtube-dl's vrtPlayerToken.
When resending the GET request in FF with youtube-dl's vrtPlayerToken, it results in the same error. When overwriting the token in youtube-dl with FF's token, it works perfectly.

[MatthiasCoppens]

To get a valid vrtPlayerToken to watch age-restricted videos, an identityToken needs to be passed with the POST request at

youtube-dl/youtube_dl/extractor/canvas.py

Line 65 in dfbbe29

token = self._download_json(

This identityToken can be found in the X-VRT-Token cookie, however for me this cookie differs between youtube-dl and my browser.

[Dre62]

Is it possible to replace the incorrect token by the right one in the cmd line request?

[sharethewisdom]

dear @MatthiasCoppens thanks for your clear description of the problem. I don't quite understand how to add an identityToken to the POST request. I naively tried to add a Firefox cookie to test in canvas.py, for example

headers.update({'Content-Type': 'application/json'})
headers.update({'X-VRT-Token': 'eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJodHRwczovL3Rva2VuLnZydC5iZSIsInN1YiI6IjYyOWE0ZThjZWQ1MzQxNDdhNTE3ZGI1OGM1YWE3NWZiIiwibmV3c3ViIjoiYjllOWY1ZGEtMDUwYy00NWQ0LWI4NzAtZjE5ZDFhOWZiZWY2IiwiY291IjoiQkUiLCJhZ2UiOjMzLCJhdWQiOiIvdnJ0bnUiLCJpYXQiOjE2MjE3NzUxNzgsImV4cCI6MTYyMTc3ODc3OCwiYWMiOiIxNisifQ.VNZoO0ZwgOezUt60So-j0cDkS8mtcg_2Kxf4ZtCeVCmw-Z5tsQfR5Ji0HVEWhWKz8ejN4B20d5XmD59KZCYkZgdGmqvhwA1_8vedi9xbFakXKDNHOYmqEjBrPOYa3Og5D7VYJHqnas2HXg-MrhPfKu0ECKqO9aS64Df0QMDUxXN9Y1ulIvNfiA0lX3cbSeBMoWZ5HUSKA7POLLWLgmnhiFKic1Lm7w4Pi7RNDw0Ri8oNaWr9ZYtVWKdrz3Y1STZRSsDmemnvFIz1-gLQJek73jBnbei20iG_TNvkuHCfWtiXIKBcBFv_Sih0S426Bsgr4pb4Io4yw8O3dGLwjLa7zfS_BGce8qwTno3cevkg5aeJBQA-8q3Uc9LR2deM0AxjHOfiME6ds9r9zqS7D7DKo2DtcrA-xJmhPs7e7aLkG5h32dEqoSpFkQmU_bvWU43TRzZ7tvNEgHTSV-HaZvCe8y877ckoBkec2rBtIpeeuL-YMsEA68xPUqOaPN8tIS5xy1NUPXcVs7MNaJbXRCIb2peHL15Dl2rQimPTWSsSM4o1HjjXM-f_TDC4jlHsjnQYMB1OBrkYwP6sKBMwwkd-C6lyvAeBwYKngXkP8w6bBpY5-JQhARVN1hKXMNgSBcT1_QmdMB3Jw9qhMDY-uPTfz2FvIbzwGbB_cvAGUEAyNa8'})

but I can't get it to work.

[MatthiasCoppens]

@sharethewisdom: This enables me to download https://www.vrt.be/vrtnu/a-z/before-we-die/2/before-we-die-s2a1/:

headers.update({'Content-Type': 'application/json'})
identityToken = "eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJodHRwczovL3Rva2VuLnZydC5iZSIsInN1YiI6IjY3NmJjMTk1MDAwNzQ1MWFiMTU2NGNjYjI5NTdmZjJjIiwibmV3c3ViIjoiMTQ1ODRjZmUtMjllNC00NzZhLWI1NDEtYjM5MWJhOTc3M2U4IiwiY291IjoiQkUiLCJhZ2UiOjI0LCJhdWQiOiIvdnJ0bnUiLCJpYXQiOjE2MjE3Nzg2MDgsImV4cCI6MTYyMTc4MjIwOCwiYWMiOiIxNisifQ.g8lM8aQX9tVacP9OUz8EAplGac76v5P9wBT8DlXGoDwQvVLncNcafnZiI5OYwVsClcMJBScAsRVKdgSXPXWT9DuPhlfyUs49zpgz-Xb05NHDAIFuFxv0DTNOB9FJEqcMGho4ehl5_524ZoIa9PavlzmPRee-Y9HKw1f-E3o3rkws73GFYAkOhTjXqo55Gg3_HCOqfJEqgKrYzvF-ILMmY9XRqVmn4c1Fw5tTcBspt77h6nKkpb6S231BQELTurJtF7-zwixuKSwZZbdhnruTFUUYlPMK4Y71ZP8PmIapappGXuf7r162FqHEVMgX1UmxWgVdHBPGzaCrNsTZtM3jWzgSlsaopN9u78cKs1N7MKxqXmD2har2a1FJBMDT0XK-oqlMRF8afTnaNEMlHixpz8M-Y4xOcDiUCPmhSCPCiLxJVmcApMM-h_jixwE04eHMYAEndseWp5cZAkz7R3p0oGmrTnYne5xTEJ5-myq3HLtzGfsAkq873e0PepEqv4mDvEEQOUWzmOL44TUQboPv8Ju267Jk3stTBhUcXvKLoFDHmvFJxj_a77QSrnLtaSvLpIqQKjvigC6JejVjwHfe3TQuBDMdeZoH3maW0s9dyM3cK6fMLTI6eS0lqQ3Oct2YM3kvV3XHGO70QD_pLpCY1YbE2YpteWZbfC-2PY-5dP0"
post_data = json.dumps({"identityToken": identityToken}).encode()
token = self._download_json(
    '%s/tokens' % self._REST_API_BASE, video_id,
    'Downloading token',
    data=post_data,
    headers=headers)['vrtPlayerToken']

[MatthiasCoppens]

I can get a working X-VRT-Token with

curl 'https://token.vrt.be/refreshtoken' -H 'Cookie: vrtlogin-rt=...' -I

[sharethewisdom]

thanks, you're so clever, that works 🥳 (but now, how to fix canvas.py...)

I initially tried data=urlencode_postdata({ 'identityToken': '...' }) from utils, but got a 400 bad request error. (FYI I get my Netscape HTTP Cookie File with https://github.com/lennonhill/cookies-txt but a curl command is much better, thanks)

[MatthiasCoppens]

When you don't have the vrtlogin-rt cookie, you can get the X-VRT-Token from the regular login sequence of requests:

youtube-dl/youtube_dl/extractor/canvas.py

Line 283 in dfbbe29

# Sometimes authentication fails for no good reason, retry

# Sometimes authentication fails for no good reason, retry
login_attempt = 1
while login_attempt <= 3:
    try:
        # This request should set the cookie OIDCXSRF
        self._request_webpage(
            'https://token.vrt.be/vrtnuinitlogin?provider=site&destination=https://www.vrt.be/vrtnu/',
            None,
            note='Initializing login',
            errnote='Could not initialize login',
        )
        # This request should set the cookie X-VRT-Token
        post_data = {
            'UID': auth_info['UID'],
            'UIDSignature': auth_info['UIDSignature'],
            'signatureTimestamp': auth_info['signatureTimestamp'],
            'client_id': 'vrtnu-site',
        }
        for cookie in self._downloader.cookiejar:
            if cookie.name == 'OIDCXSRF':
                post_data['_csrf'] = cookie.value
                break
        self._request_webpage(
            'https://login.vrt.be/perform_login?client_id=vrtnu-site',
            None,
            note='Requesting a token',
            errnote='Could not get a token',
            headers={
                'Content-Type': 'application/x-www-form-urlencoded',
            },
            data=json.dumps(post_data).encode('utf-8'),
        )
    except ExtractorError as e:
        if isinstance(e.cause, compat_HTTPError) and e.cause.code == 401:

The second request fails (403: the server understood the request, but is refusing to authorize it). In browser this request sends some extra cookies to the server, not sure where they come from or which ones are important.

[chaff02]

Hi,
I'm also having the problem with the age restriction on vrtnu.be.
I tried to fix the canvas.py file with the above code of MatthiasCoppens, but no luck so far.
Always have the error: "ERROR: Could not get a token: HTTP Error 403: "

Is there a fix available for the canvas.py file. Many thanks

[ghen2]

See also Kodi's VRT NU add-on bug add-ons/plugin.video.vrt.nu#882 and fix add-ons/plugin.video.vrt.nu#885; apparantly age restricted content now requires a "user" X-VRT-Token instead of the legacy token (recent change on VRT side).

Matthias' #29044 (comment) works for me.

[chaff02]

Works for me too now. Was a little bit of a search for the correct cookie (vrtnu-site_profile_vt).
Thanks to ghen2 and Matthias

[MatthiasCoppens]

I'm in the middle of my exams, so I can't really do a lot of troubleshooting right now, but the code at #29044 (comment) seems very similar to add-ons/plugin.video.vrt.nu#885 so this shouldn't take a lot of tweaking to get it to work.

[chaff02]

No problem Matthias. Your code at #29044 (comment) works for me. I only have to replace the value of the identityToken by the actual value of the 'vrtnu-site_profile_vt' cookie from www.vrt.be. However this cookie has a short lifetime, as a result many 'make youtube-dl' commands.
Success with your exams.

[pgaig]

I wrote a fix #29614 heavily inspired from add-ons/plugin.video.vrt.nu#885 and the comments above.

Seems to work for me + fixes the authentication issue #29534 .

[BelgianNoise]

Until the linked PR is merged (and released) you can always download from vrtNU this way: #29534 (comment)