Previously supported videos now unsupported after site update UKTV UKTVPLAY

[Rencroft]

Hi,

---

• I've verified and I assure that I'm running youtube-dl 2018.10.05
• Searched the bugtracker for similar issues including closed ones
• Checked that provided video/audio/playlist URLs (if any) are alive and playable in a browser
• Site support request (request for adding support for a new site)

---

C:\Program Files (x86)\youtubedl>youtube-dl --version
2018.10.05

C:\Program Files (x86)\youtubedl>youtube-dl.exe -o -v "https://uktvplay.uktv.co.
uk/shows/casualty/watch-online/5839038888001"
[generic] 5839038888001: Requesting header
WARNING: Falling back on generic information extractor.
[generic] 5839038888001: Downloading webpage
[generic] 5839038888001: Extracting information
ERROR: Unsupported URL: https://uktvplay.uktv.co.uk/shows/casualty/watch-online/
5839038888001

C:\Program Files (x86)\youtubedl>
...
<end of log>

---

• Single video: https://uktvplay.uktv.co.uk/shows/casualty/watch-online/5839038888001

---

Support for downloading these videos was working but the site hosting them was updated and now the unsupported url error is given. The URLs for the videos have not changed format so hopefully it will be possible/easy to fix support for these videos. Note a free account and login is needed to play the videos in a browser but youtube-dl was still previously able to download them without login before the site update.

Kind Regards
Rencroft

[darthhaggis]

To expand on the previous post, URL formatting has changed slightly. For example (for same video page):
URL before site update (still works with youtube-dl): https://uktvplay.uktv.co.uk/shows/secrets-of-the-castle/watch-online/?video=5307684600001
URL after site update (doesn't work with youtube-dl): https://uktvplay.uktv.co.uk/shows/secrets-of-the-castle/watch-online/5818502556001
The old URL however is not viewable in a browser; it just defaults to the series landing page whereas the new URL will direct you to the specific episode within the browser.
Looking at the page source for the new URL, the old video_id can not be found within the html.

[Rencroft]

darthhaggis, thanks for the input your quite right.

[thrnz]

Note that some videos still fail to download, even having changed to the old URL format. For instance:

[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['--verbose', 'https://uktvplay.uktv.co.uk/shows/taskmaster/watch-online/?video=5827355680001']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2018.10.05
[debug] Python version 3.6.6 (CPython) - Linux-4.15.0-36-generic-x86_64-with
[debug] exe versions: ffmpeg 3.4.4, ffprobe 3.4.4
[debug] Proxy map: {}
[brightcove:new] 5827355680001: Downloading webpage
[brightcove:new] 5827355680001: Downloading JSON metadata
ERROR: No video formats found; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/youtube_dl/YoutubeDL.py", line 792, in extract_info
    ie_result = ie.extract(url)
  File "/usr/lib/python3.6/site-packages/youtube_dl/extractor/common.py", line 507, in extract
    ie_result = self._real_extract(url)
  File "/usr/lib/python3.6/site-packages/youtube_dl/extractor/brightcove.py", line 743, in _real_extract
    json_data, video_id, headers=headers)
  File "/usr/lib/python3.6/site-packages/youtube_dl/extractor/brightcove.py", line 648, in _parse_brightcove_metadata
    self._sort_formats(formats)
  File "/usr/lib/python3.6/site-packages/youtube_dl/extractor/common.py", line 1286, in _sort_formats
    raise ExtractorError('No video formats found')
youtube_dl.utils.ExtractorError: No video formats found; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.

[darthhaggis]

@thrnz:
Notice in the example given, the video_id associated with the old format is not the same video_id associated with the new format URL, even though they are for the same video. Using the video_id associated with the other format does not work and produces errors. I have not found reference to the old video_id within the new URL source code, so I don't see a simple way of extracting the old video_id to be able to continue using the old format.

EDIT: If there were a quick and easy way of extracting the old video_id from the new URL, we could continue with 'business as usual' but such a way has so far eluded us. If anyone can prove us wrong, please enlighten us.

In situations like this, it would be so much simpler if we could just throw the Blob file address at the program...

[dankargo]

Just a very amateur guess but could the value after "/fairplay/' in this example be a potential VideoID?

src="https://manifest.prod.boltdns.net/manifest/v1/hls/v5/fairplay/1242911124001/436b2bb0-3519-421c-aebb-eb1b1be02534/10s/master.m3u8?...

For URL
https://uktvplay.uktv.co.uk/shows/classic-eastenders/watch-online/5844243441001

[darthhaggis]

Just a very amateur guess but could the value after "/fairplay/' in this example be a potential VideoID?

src="https://manifest.prod.boltdns.net/manifest/v1/hls/v5/fairplay/1242911124001/436b2bb0-3519-421c-aebb-eb1b1be02534/10s/master.m3u8?...

For URL
https://uktvplay.uktv.co.uk/shows/classic-eastenders/watch-online/5844243441001

I have tried using that number with no luck. It also appears elsewhere in the source code as:
data-account="1242911124001"

[Vangelis66]

after "/fairplay/"

Well, if they're now using Apple Fairplay DRM for their mobile streams, then it's GAME OVER (it hasn't been cracked) 💢 . Prior to their site overhaul, they used the brightcove service and AppleHLS encrypted (HLSe) streams for mobile devices (which use AES-128 encryption supported by yt-dl) ...

I have kept records on disk and for a show in the old site format

http://uktvplay.uktv.co.uk/shows/narrow-escapes-of-world-war-ii/watch-online/?video=2776248288001

the HLSe master playlist would have been

http://c.brightcove.com/services/mobile/streaming/index/master.m3u8?videoId=2776248288001

The master playlist posted by @dankargo suggests a completely new playlist API...
BTW, it would help other people troubleshoot this further if valid login credentials were shared here, even if temporarily... 😉

[Rencroft]

Here's a login :)

cemud@oranek.com
devtempy1#

[Vangelis66]

@Rencroft

Many thanks for the login credentials 👍
HOWEVER, my worst fears have been realised 😞

Trying, for instance,

https://uktvplay.uktv.co.uk/shows/classic-eastenders/watch-online/5844243441001

in a desktop browser and inspecting network activity via Web Console, one can identify, even before login-in, the following GET XHR (XMLHttpRequest) request to

https://edge.api.brightcove.com/playback/v1/accounts/1242911124001/videos/5844243441001

The raw response from that request is the following JSON file:

{
 "ad_keys": null,
 "sources": [
  {
   "src": "http://manifest.prod.boltdns.net/manifest/v1/hls/v5/fairplay/1242911124001/436b2bb0-3519-421c-aebb-eb1b1be02534/10s/master.m3u8?fastly_token=NWJlZTE2N2RfZjg1NTNmYzZiNGU2YjA1OWE4ZWMyMjg2ZTFkYjc1MWM4OWU3MmE4ZjVhY2U4ZGZiOTJiNGNhMWFhMTkyMTQyNA%3D%3D",
   "key_systems": {
    "com.apple.fps.1_0": {
     "certificate_url": "https://manifest.prod.boltdns.net/license/v1/fairplay_app_cert/1242911124001",
     "key_request_url": "https://manifest.prod.boltdns.net/license/v1/fairplay/1242911124001/436b2bb0-3519-421c-aebb-eb1b1be02534/6a19b118-ad29-4a53-8fa5-8abbdea10fca?fastly_token=NWJlZTE2N2RfZmU1MWQ5OTFhODc3MTcwYjAxMTQxZTAzMmE0MmQ2YTUzMzgyNTEwYTkyNmYxODM4ZDc4N2VhZDkyNGE2ZjljNw%3D%3D"
    }
   },
   "ext_x_version": "5",
   "type": "application/x-mpegURL"
  },{
   "src": "https://manifest.prod.boltdns.net/manifest/v1/hls/v5/fairplay/1242911124001/436b2bb0-3519-421c-aebb-eb1b1be02534/10s/master.m3u8?fastly_token=NWJlZTE2N2RfZjg1NTNmYzZiNGU2YjA1OWE4ZWMyMjg2ZTFkYjc1MWM4OWU3MmE4ZjVhY2U4ZGZiOTJiNGNhMWFhMTkyMTQyNA%3D%3D",
   "key_systems": {
    "com.apple.fps.1_0": {
     "certificate_url": "https://manifest.prod.boltdns.net/license/v1/fairplay_app_cert/1242911124001",
     "key_request_url": "https://manifest.prod.boltdns.net/license/v1/fairplay/1242911124001/436b2bb0-3519-421c-aebb-eb1b1be02534/6a19b118-ad29-4a53-8fa5-8abbdea10fca?fastly_token=NWJlZTE2N2RfZmU1MWQ5OTFhODc3MTcwYjAxMTQxZTAzMmE0MmQ2YTUzMzgyNTEwYTkyNmYxODM4ZDc4N2VhZDkyNGE2ZjljNw%3D%3D"
    }
   },
   "ext_x_version": "5",
   "type": "application/x-mpegURL"
  },{
   "src": "http://manifest.prod.boltdns.net/manifest/v1/dash/live-baseurl/bccenc/1242911124001/436b2bb0-3519-421c-aebb-eb1b1be02534/6s/manifest.mpd?fastly_token=NWJlZTE2N2RfOTlkMTAwNDg0NTQ2YTdjNDgyZTViODc3YThjYzgxYzM0MWRmNGM5OTY5ZTEyYzZjZTI2YmFhYzhmZTNlMzE5ZA%3D%3D",
   "key_systems": {
    "com.widevine.alpha": {
     "license_url": "https://manifest.prod.boltdns.net/license/v1/cenc/widevine/1242911124001/436b2bb0-3519-421c-aebb-eb1b1be02534/ecc64650-eade-4f4a-89d5-51b0505ea674?fastly_token=NWJlZTE2N2RfMDE1OWUwY2U2ZmY5ZDk5OTYxYmFhMGU2MWEzNzhkNjE4ZWEyYTllNzhiMmI1NTMwNmQ1MGYwYTZhNjgzMDJmOA%3D%3D"
    },
    "com.microsoft.playready": {
     "license_url": "https://manifest.prod.boltdns.net/license/v1/cenc/playready/1242911124001/436b2bb0-3519-421c-aebb-eb1b1be02534/ecc64650-eade-4f4a-89d5-51b0505ea674?fastly_token=NWJlZTE2N2RfNjhlOWU1YzY3OWE2Yjk1OWU2MTYxZjcwZmIwMzczZDRlYThjNGM5MjM2NmYzNzUyYmY1ODhjY2I0NjM4MGFhNg%3D%3D"
    }
   },
   "profiles": "urn:mpeg:dash:profile:isoff-live:2011",
   "type": "application/dash+xml"
  },{
   "src": "https://manifest.prod.boltdns.net/manifest/v1/dash/live-baseurl/bccenc/1242911124001/436b2bb0-3519-421c-aebb-eb1b1be02534/6s/manifest.mpd?fastly_token=NWJlZTE2N2RfOTlkMTAwNDg0NTQ2YTdjNDgyZTViODc3YThjYzgxYzM0MWRmNGM5OTY5ZTEyYzZjZTI2YmFhYzhmZTNlMzE5ZA%3D%3D",
   "key_systems": {
    "com.widevine.alpha": {
     "license_url": "https://manifest.prod.boltdns.net/license/v1/cenc/widevine/1242911124001/436b2bb0-3519-421c-aebb-eb1b1be02534/ecc64650-eade-4f4a-89d5-51b0505ea674?fastly_token=NWJlZTE2N2RfMDE1OWUwY2U2ZmY5ZDk5OTYxYmFhMGU2MWEzNzhkNjE4ZWEyYTllNzhiMmI1NTMwNmQ1MGYwYTZhNjgzMDJmOA%3D%3D"
    },
    "com.microsoft.playready": {
     "license_url": "https://manifest.prod.boltdns.net/license/v1/cenc/playready/1242911124001/436b2bb0-3519-421c-aebb-eb1b1be02534/ecc64650-eade-4f4a-89d5-51b0505ea674?fastly_token=NWJlZTE2N2RfNjhlOWU1YzY3OWE2Yjk1OWU2MTYxZjcwZmIwMzczZDRlYThjNGM5MjM2NmYzNzUyYmY1ODhjY2I0NjM4MGFhNg%3D%3D"
    }
   },
   "profiles": "urn:mpeg:dash:profile:isoff-live:2011",
   "type": "application/dash+xml"
  },{
   "src": "http://manifest.prod.boltdns.net/manifest/v1/dash/live-hbbtv15/playready/1242911124001/436b2bb0-3519-421c-aebb-eb1b1be02534/6s/manifest.mpd?fastly_token=NWJlZTE2N2RfM2NkYjAzZmY0NTExYTUzYzQ1ZGY4YTAxZjJjOTc5NGJhMDk5ODJkMDFmOWQ5YWE2YWFkYWFlZGM4ZDJiOGQ3Mg%3D%3D",
   "key_systems": {
    "com.microsoft.playready": {
     "license_url": "https://manifest.prod.boltdns.net/license/v1/playready/1242911124001/436b2bb0-3519-421c-aebb-eb1b1be02534/7ef1386e-c391-4256-ba13-069340a0345e?fastly_token=NWJlZTE2N2RfYjgyODk4NjZmNDRjZjE1YTYxODMzYWY5MmFmMGM5YmUxZWI4Y2M3OTU5MzhhNDBmYzAwN2E3ZjUyN2NkYTYyOQ%3D%3D"
    }
   },
   "profiles": "urn:hbbtv:dash:profile:isoff-live:2012,urn:mpeg:dash:profile:isoff-live:2011",
   "type": "application/dash+xml"
  },{
   "src": "https://manifest.prod.boltdns.net/manifest/v1/dash/live-hbbtv15/playready/1242911124001/436b2bb0-3519-421c-aebb-eb1b1be02534/6s/manifest.mpd?fastly_token=NWJlZTE2N2RfM2NkYjAzZmY0NTExYTUzYzQ1ZGY4YTAxZjJjOTc5NGJhMDk5ODJkMDFmOWQ5YWE2YWFkYWFlZGM4ZDJiOGQ3Mg%3D%3D",
   "key_systems": {
    "com.microsoft.playready": {
     "license_url": "https://manifest.prod.boltdns.net/license/v1/playready/1242911124001/436b2bb0-3519-421c-aebb-eb1b1be02534/7ef1386e-c391-4256-ba13-069340a0345e?fastly_token=NWJlZTE2N2RfYjgyODk4NjZmNDRjZjE1YTYxODMzYWY5MmFmMGM5YmUxZWI4Y2M3OTU5MzhhNDBmYzAwN2E3ZjUyN2NkYTYyOQ%3D%3D"
    }
   },
   "profiles": "urn:hbbtv:dash:profile:isoff-live:2012,urn:mpeg:dash:profile:isoff-live:2011",
   "type": "application/dash+xml"
  },{
   "src": "http://bcboltuktv-a.akamaihd.net/media/v1/smooth/simple/playready/1242911124001/436b2bb0-3519-421c-aebb-eb1b1be02534/7ef1386e-c391-4256-ba13-069340a0345e/manifest.ism/Manifest?akamai_token=exp=1542329981~acl=/media/v1/smooth/simple/playready/1242911124001/436b2bb0-3519-421c-aebb-eb1b1be02534/7ef1386e-c391-4256-ba13-069340a0345e/manifest.ism/Manifest*~hmac=e3e9c3053b1043edd71c6c0a8aeaf8b90a35fe1e90890821dad8ee27893ebf3e",
   "key_systems": {
    "com.microsoft.playready": {
     "license_url": "https://manifest.prod.boltdns.net/license/v1/playready/1242911124001/436b2bb0-3519-421c-aebb-eb1b1be02534/7ef1386e-c391-4256-ba13-069340a0345e?fastly_token=NWJlZTE2N2RfYjgyODk4NjZmNDRjZjE1YTYxODMzYWY5MmFmMGM5YmUxZWI4Y2M3OTU5MzhhNDBmYzAwN2E3ZjUyN2NkYTYyOQ%3D%3D"
    }
   },
   "type": "application/vnd.ms-sstr+xml"
  },{
   "src": "https://bcboltuktv-a.akamaihd.net/media/v1/smooth/simple/playready/1242911124001/436b2bb0-3519-421c-aebb-eb1b1be02534/7ef1386e-c391-4256-ba13-069340a0345e/manifest.ism/Manifest?akamai_token=exp=1542329981~acl=/media/v1/smooth/simple/playready/1242911124001/436b2bb0-3519-421c-aebb-eb1b1be02534/7ef1386e-c391-4256-ba13-069340a0345e/manifest.ism/Manifest*~hmac=e3e9c3053b1043edd71c6c0a8aeaf8b90a35fe1e90890821dad8ee27893ebf3e",
   "key_systems": {
    "com.microsoft.playready": {
     "license_url": "https://manifest.prod.boltdns.net/license/v1/playready/1242911124001/436b2bb0-3519-421c-aebb-eb1b1be02534/7ef1386e-c391-4256-ba13-069340a0345e?fastly_token=NWJlZTE2N2RfYjgyODk4NjZmNDRjZjE1YTYxODMzYWY5MmFmMGM5YmUxZWI4Y2M3OTU5MzhhNDBmYzAwN2E3ZjUyN2NkYTYyOQ%3D%3D"
    }
   },
   "type": "application/vnd.ms-sstr+xml"
  }],
 "updated_at": "2018-10-05T05:21:19.086Z",
 "poster_sources": [
  {
   "src": "https://cf-images.eu-west-1.prod.boltdns.net/v1/jit/1242911124001/436b2bb0-3519-421c-aebb-eb1b1be02534/main/1024x576/14m46s48ms/match/image.jpg"
  }],
 "reference_id": null,
 "offline_enabled": false,
 "text_tracks": [],
 "economics": "AD_SUPPORTED",
 "description": null,
 "poster": "https://cf-images.eu-west-1.prod.boltdns.net/v1/jit/1242911124001/436b2bb0-3519-421c-aebb-eb1b1be02534/main/1024x576/14m46s48ms/match/image.jpg",
 "long_description": null,
 "created_at": "2018-10-04T09:41:43.440Z",
 "id": "5844243441001",
 "custom_fields": {
  "springstream": "OD",
  "housenum": "CTBT132W82-dd",
  "springcq": "CTBT132W"
 },
 "duration": 1772096,
 "account_id": "1242911124001",
 "tags": [],
 "name": "CTBT132W82-dd",
 "link": null,
 "thumbnail": "https://cf-images.eu-west-1.prod.boltdns.net/v1/jit/1242911124001/436b2bb0-3519-421c-aebb-eb1b1be02534/main/320x180/14m46s48ms/match/image.jpg",
 "cue_points": [
  {
   "metadata": null,
   "force_stop": false,
   "id": "5844706303001",
   "type": "AD",
   "time": 673.64,
   "name": "cp_16243983"
  },{
   "metadata": null,
   "force_stop": false,
   "id": "5844706304001",
   "type": "AD",
   "time": 1772,
   "name": "cp_16243984"
  }],
 "thumbnail_sources": [
  {
   "src": "https://cf-images.eu-west-1.prod.boltdns.net/v1/jit/1242911124001/436b2bb0-3519-421c-aebb-eb1b1be02534/main/320x180/14m46s48ms/match/image.jpg"
  }],
 "published_at": "2018-10-04T09:41:43.440Z"
}

As anyone can see, the place reeks of DRM: AppleHLS streams (mobile devices, mobile apps) use FairPlay, MPEG-DASH streams (desktop browsers) use Common Encryption (cenc) with Widevine, plus there exist Microsoft friendly "Smooth Streaming" streams which use PlayReady DRM!

All mentioned DRM schemes have not been reverse-engineered (if someone has, he/she'd be wise enough to not disclose his/her feat...), so it is currently unrealistic to expect yt-dl to support the new site of UKTVplay...

Unless someone figures out an algorithm to convert the "new" video_ids to the "old" value that would still work (for how long?) with the previous site template, as I said, it's GAME OVER! If, like me, you abhor any form of DRM, you'd better stay away from the new UKTVPlay iteration 👎

[rautamiekka]

All mentioned DRM schemes have not been reverse-engineered (if someone has, he/she'd be wise enough to not disclose his/her feat...), so it is currently unrealistic to expect yt-dl to support the new site of UKTVplay...

Why ?

[Rencroft]

Well that's a shame, they were better quality than recording of TV.

Thanks to everyone who looked into it.

[darthhaggis]

Let me see if I understand this correctly. Long story short, even though we can view an episode without issue in a browser, due to the change in how their website is structured, there is now currently no way we will be able to use yt-dl to save that same file for offline viewing, correct?

[dankargo]

I just discovered this Kodi plugin which has recently been updated to allow streaming of UKTV Play's new DRM content in Kodi. Maybe it may give us some hope?

https://github.com/Catch-up-TV-and-More/plugin.video.catchuptvandmoretestdrmkodi18

[Vangelis66]

@dankargo

Their UKTV Play plugin can be found here:
https://github.com/Catch-up-TV-and-More/plugin.video.catchuptvandmoretestdrmkodi18/blob/master/resources/lib/channels/uk/uktvplay.py
but, as I said previously, it's useless for yt-dl purposes!

Widevine DRM hasn't been cracked; in fact, Kodi does not circumvent that DRM scheme but relies upon a system installed Chrome browser (with a supported and working WidevineCDM, currently at version >= 1.4.8.984) for the actual decryption of the streams; you can read more at:
https://kodi.tv/article/dev-journal-kodi-and-drm

As another example, in Windows, there isn't a very good 10-foot interface for Netflix,
but if you have installed Chrome, you can watch Netflix from your desktop.
Supporting low-level DRM means that Kodi could hook into the binary blob inside Chrome
and use it purely for handling the DRM while video playback and control stays with Kodi.