[ITV] ERROR: Unable to extract params ('Unable to extract %s' % _name)

[Qiangong2]

Checklist

• I'm reporting a broken site support
• I've verified that I'm running youtube-dl version 2021.12.17
• I've checked that all provided URLs are alive and playable in a browser
• I've checked that all URLs and arguments with special characters are properly quoted or escaped
• I've searched the bugtracker for similar issues including closed ones

Verbose log

python3 -m youtube_dl --user-agent "Mozilla/5.0" -f mp4/best http://www.itv.com/watch/the-chase/1a7842/1a7842a1903 --hls-prefer-ffmpeg --verbose
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['--user-agent', 'Mozilla/5.0', '-f', 'mp4/best', 'http://www.itv.com/watch/the-chase/1a7842/1a7842a1903', '--hls-prefer-ffmpeg', '--verbose']
[debug] Encodings: locale UTF-8, fs utf-8, out utf-8, pref UTF-8
[debug] youtube-dl version 2021.12.17
[debug] Git HEAD: f24bc9272
[debug] Python 3.11.4 (CPython arm64 64bit) - macOS-14.0-arm64-arm-64bit - OpenSSL 3.1.1 30 May 2023
[debug] exe versions: ffmpeg 6.0, ffprobe 6.0, rtmpdump 2.4
[debug] Proxy map: {}
[ITV] 1a7842a1903: Downloading webpage
ERROR: Unable to extract params; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
Traceback (most recent call last):
  File "/Users/Orca/Downloads/youtube-dl/youtube_dl/YoutubeDL.py", line 829, in wrapper
    return func(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/Orca/Downloads/youtube-dl/youtube_dl/YoutubeDL.py", line 850, in __extract_info
    ie_result = ie.extract(url)
                ^^^^^^^^^^^^^^^
  File "/Users/Orca/Downloads/youtube-dl/youtube_dl/extractor/common.py", line 535, in extract
    ie_result = self._real_extract(url)
                ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/Orca/Downloads/youtube-dl/youtube_dl/extractor/itv.py", line 193, in _real_extract
    params = extract_attributes(self._search_regex(
                                ^^^^^^^^^^^^^^^^^^^
  File "/Users/Orca/Downloads/youtube-dl/youtube_dl/extractor/common.py", line 1013, in _search_regex
    raise RegexNotFoundError('Unable to extract %s' % _name)
youtube_dl.utils.RegexNotFoundError: Unable to extract params; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.

Description

I'm trying to download videos from ITV, however, each time I try (regardless of the video), I get an "unable to extract params" error. I am using the latest master. I can watch any of the shows through the browser without issue.

Only similar issue I've been able to find is #22638 which does not give a fix

[dirkf]

@xaoscience, no: read #30839 as asked.

#22638 hasn't got anything to do with ITV. Unfortunately the ITV[x] page structure has changed. The desired playlist URL can now be found with this:

--- old/youtube_dl/extractor/itv.py
+++ new/youtube_dl/extractor/itv.py
@@ -189,14 +189,19 @@
 
         webpage = self._download_webpage(url, video_id)
 
-        # now quite different params!
-        params = extract_attributes(self._search_regex(
-            r'''(<[^>]+\b(?:class|data-testid)\s*=\s*("|')genie-container\2[^>]*>)''',
-            webpage, 'params'))
-
-        ios_playlist_url = traverse_obj(
-            params, 'data-video-id', 'data-video-playlist',
-            get_all=False, expected_type=url_or_none)
+        ios_playlist_url = url_or_none(self._search_regex(
+            r'''\bplaylistUrl['"]\s*:\s*(?P<q>"|')(?P<url>https?://magni\.itv\.com/playlist/(?:(?!(?=q)).)+)''',
+            webpage, 'playlist URL', group='url', fatal=False))
+
+        if not ios_playlist_url:
+            # legacy: does it ever happen?
+            params = extract_attributes(self._search_regex(
+                r'''(<[^>]+\b(?:class|data-testid)\s*=\s*("|')genie-container\2[^>]*>)''',
+                webpage, 'params'))
+
+            ios_playlist_url = traverse_obj(
+                params, 'data-video-id', 'data-video-playlist',
+                get_all=False, expected_type=url_or_none)
 
         headers = self.geo_verification_headers()
         headers.update({

Again unfortunately, then we get 404

...
[ITV] 1a7842a1903: Downloading webpage
[ITV] 1a7842a1903: Downloading JSON metadata
ERROR: Unable to download JSON metadata: HTTP Error 404: Not Found ...

when fetching the data with the URL.

Perhaps @Vangelis66 can advise on how, or if at all, the POST to the playlist URL can be modified so as to return valid data.

[ta264]

I ran into this yesterday. I believe the issue is the variantAvailability parameter. Currently it is sending:

'variantAvailability': {
                    'player': 'hls',
                    'featureset': {
                        'min': ['hls', 'aes', 'outband-webvtt'],
                        'max': ['hls', 'aes', 'outband-webvtt']
                    },
                    'platformTag': 'mobile'
                }

This returns 404 and the following message:

{
	"Message": "something needed to service this request was missing",
	"TransactionId": "oas-xxx"
}

The desktop web client sends:

  "variantAvailability": {
    "player": "hls",
    "featureset": {
      "min": [
        "mpeg-dash",
        "widevine",
        "outband-webvtt",
        "hd",
        "single-track"
      ],
      "max": [
        "mpeg-dash",
        "widevine",
        "outband-webvtt",
        "hd",
        "single-track"
      ]
    },
    "platformTag": "dotcom"
  }

But the response from this doesn't allow the download to work.

FWIW I think this 404 might only apply to newer shows. It fails for
https://www.itv.com/watch/tour-de-france/34067/1a7000a0353

But the original POST params work for the built in test case:
https://www.itv.com/watch/vera/1a7314/1a7314a0014

[dirkf]

I ran into this yesterday. I believe the issue is the variantAvailability parameter. ...

The desktop client is tied into the DRM stuff. We pretend to be an old mobile, hence 'platformTag': 'mobile'. Maybe the 404 response is indicating that now there are no media links matching the featureset that we want.