Bandwidth throttling mitigation

[Serkora]

Before submitting a pull request make sure you have:

• At least skimmed through adding new extractor tutorial and youtube-dl coding conventions sections
• Searched the bugtracker for similar pull requests

In order to be accepted and merged into youtube-dl each piece of code must be in public domain or released under Unlicense. Check one of the following options:

• I am the original author of this code and I am willing to release it under Unlicense
• I am not the original author of this code but it is in public domain or released under Unlicense (provide reliable evidence)

What is the purpose of your pull request?

• Bug fix
• Improvement
• New extractor
• New feature

---

Description of your pull request and other information

#6923
Pretty much self explanatory.
Some websites only let you download a little bit at maximum speed and then slow it down to some value (may be media bitrate, may be just an arbitrary limit they decided upon)
If this website also supports Range header, a new request can be made to get full speed again. Repeat until ready.

In a nutshell, that's all this code does: monitors the overall (to get the peak) and separate block transfer rates and makes a new request when the block rate falls below a certain threshold.
And a new option is added, of course.
A few unnecessary reconnections here and there, but mostly stable chunks are downloaded before each new request is made, so I'm rather pleased with it.

Important notes:

1. When disabled, downloader behaves exactly as before (other than a few excessive calcuations, perhaps), so it should be safe to replace the original downloader with it (instead of a making a separate one).
2. I can't (easily) simulate various network conditions, so more testing is required to see if the strategies used are actually good and robust enough.
3. Tested mainly in clear network conditions and on two websites only, with high and not so high peak rates (5-6MiB/s and ~1.2MiB/s) and similar throttling rates (50-100KiB/s), so all of the constant values and dynamic formulae, while thought out, are approximate and obtained partially empirically to maximaize the overall mean download rate on my computer. ymmv, as always. Any suggestions are welcomed, testing of various different websites is greatly appreciated,
4. Mean download rate of around 50-80% of the reported peak can be achieved,
5. It behaves quite well in absence of throttling, only a few spurious reconnections might occur due to temporary network issues. Not with a bad internet, however.

Boring explanations of the strategies, can be easily seen in code but just so that it is documented:
First of all, no way to tell if throttling started or your network became bad (or downloading lots of porn in parallel, for instance), so if you start a second network-intesive task, well, things will break.
The main thing with throttling is the block size. Too small — slow overall download and/or unreliable instantaneous rate measurements. Too large — you'll become old before throttling is detected. 512KiB seemed like a reasonable enough compromise to set as a starting point.

1. First second of the download is ignored, nothing can happen there,
2. Peak rate is only set if block size is 64KiB or more, because if it's less, block rate measurements will be very inaccurate (orders of magnitute) and this also means your connections is not probably not good enough to suffer from throttling,
3. When peak rate is set and current block rate falls below 70% of that peak, timer starts,
4. After each consequent block with rate below 70%, a check is made to see if 3 seconds have passed, at which point it is assessed if the download rate starting from time recorded in point 2 is still below 70%. This helps deal with false positives when those blocks are a good amount of time apart from each other,
5. If throttling, the block size is set to the power of two nearest to the download rate during throttling, to download as little as possible during each subsequent throttle (i.e. detect it fast),
6. Threshold is set to the first quarter between throttling and peak rates (e.g. 400K peak and 50K throttling wil result in a 137.5K, or ~0.35 threshold) to cactch it early (e.g. if it was enabled in the middle of the block). However, unless you have shitty internet, throttling rate will be almost negligible, leading to the threshold of ~0.25-7
7. When this threshold is acquired, after each block a check is made to see if the current block rate fell below this threshold, and a new request is issued if so, replacing the data variable.
   Voila, grab some pizza and enjoy the thing you've just downloaded!

[Serkora]

Some verbose logs of downloads.

Youtube.com (no throttle):

$ youtube-dl --avoid-throttling -f 22 -v "https://www.youtube.com/watch?v=NYVjIjBMx6o"
[debug] System config: []
[debug] User config: []
[debug] Command-line args: ['--avoid-throttling', '-f', '22', '-v', 'https://www.youtube.com/watch?v=NYVjIjBMx6o']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2016.12.09
[debug] Python version 3.5.2 - Darwin-15.4.0-x86_64-i386-64bit
[debug] exe versions: none
[debug] Proxy map: {}
[youtube] NYVjIjBMx6o: Downloading webpage
[youtube] NYVjIjBMx6o: Downloading video info webpage
[youtube] NYVjIjBMx6o: Extracting video information
[youtube] NYVjIjBMx6o: Downloading MPD manifest
[debug] Invoking downloader on 'https://r1---sn-gvnuxaxjvh-c35z.googlevideo.com/videoplayback?id=o-APYSVSXXi8KzfLVPFpoSMrZb6l-IA65DkTzGDxd3MUVN&ei=VDlWWJTuKtHLYsbcotgF&ms=au&mt=1482045426&mv=m&pl=36&ratebypass=yes&ip=2a02%3A2168%3A1840%3Af00%3A345a%3A70f9%3Adea9%3Ac916&source=youtube&mm=31&mn=sn-gvnuxaxjvh-c35z&sparams=dur%2Cei%2Cgcr%2Cid%2Cinitcwndbps%2Cip%2Cipbits%2Citag%2Clmt%2Cmime%2Cmm%2Cmn%2Cms%2Cmv%2Cpl%2Cratebypass%2Crequiressl%2Csource%2Cupn%2Cexpire&expire=1482067380&lmt=1478674677018311&key=yt6&initcwndbps=2705000&signature=DA81FF8898D65F071D91942FFF18C4D1C4800C00.D3BEC1269D8B9B44392B3557A282AE557911B220&ipbits=0&upn=hPqNnbnTLzI&itag=22&gcr=ru&mime=video%2Fmp4&dur=4798.403&requiressl=yes'
[download] Destination: New Macbook Pro can't walk & chew gum at same time(watch wifi + USB = fail)  PART 1-NYVjIjBMx6o.mp4
[download] 100% of 693.10MiB in 02:01

Vk.com:

$ youtube-dl -f url480 -u PRIVATE -o healer.mp4 -v --avoid-throttling "https://vk.com/video-41931658_171186378"
[debug] System config: []
[debug] User config: []
[debug] Command-line args: ['-f', 'url480', '-u', 'PRIVATE', '-o', 'healer.mp4', '-v', '--avoid-throttling', 'https://vk.com/video-41931658_171186378']
Type account password and press [Return]: 
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2016.12.09
[debug] Python version 3.5.2 - Darwin-15.4.0-x86_64-i386-64bit
[debug] exe versions: none
[debug] Proxy map: {}
[vk] Downloading login page
[vk] Logging in as PRIVATE
[vk] -41931658_171186378: Downloading webpage
[debug] Invoking downloader on 'https://cs543604.vk.me/u3184474/videos/fed523eff7.480.mp4?extra=LMTmdjep5zkOloEtxfqFY5j2dGoE7cwQ_9BJB3GXBpOod0Q8-nascU18WK7mFymeqj6E5CzWzC2aX9VtTkrpOAJ72DfiN50C8X6IXly2UxYoqcUKhHsMdFB9DfhFoo7K9PRBUQLk6KgzHmgQnZUKyYm3HkhSvozEA3SJLzBBpgOiEw'
[download] Destination: healer.mp4
[download]   6.8% of 440.98MiB at  5.50MiB/s ETA 01:14
[throttling] Throttling started or is continuing, block rate = 803.15KiB/s, peak rate = 5628.14KiB/s
[download]   6.9% of 440.98MiB at  5.00MiB/s ETA 01:22
[throttling] Throttling started or is continuing, block rate = 234.91KiB/s, peak rate = 5628.14KiB/s
[download]   7.1% of 440.98MiB at  3.76MiB/s ETA 01:48
[throttling] Throttling started or is continuing, block rate = 223.10KiB/s, peak rate = 5628.14KiB/s
[throttling] Throttling detected! peak rate = 5628.14KiB/s, current rate = 348.02KiB/s, setting threshold to 0.30 and block size limit to 256KiB
[throttling] Bandwidth throttling detected, making a new request. (peak rate = 5628.14KiB/s, block rate = 223.10KiB/s, downloaded 32511KiB before throttling)
[download]  14.0% of 440.98MiB at  3.75MiB/s ETA 01:41
[throttling] Bandwidth throttling detected, making a new request. (peak rate = 5628.14KiB/s, block rate = 633.30KiB/s, downloaded 30976KiB before throttling)
[download]  20.9% of 440.98MiB at  4.13MiB/s ETA 01:24
[throttling] Bandwidth throttling detected, making a new request. (peak rate = 5628.14KiB/s, block rate = 652.57KiB/s, downloaded 30976KiB before throttling)
[download]  27.7% of 440.98MiB at  4.37MiB/s ETA 01:13
[throttling] Bandwidth throttling detected, making a new request. (peak rate = 5628.14KiB/s, block rate = 322.69KiB/s, downloaded 30976KiB before throttling)
[download]  34.6% of 440.98MiB at  4.40MiB/s ETA 01:05
[throttling] Bandwidth throttling detected, making a new request. (peak rate = 5628.14KiB/s, block rate = 788.02KiB/s, downloaded 30976KiB before throttling)
[download]  41.4% of 440.98MiB at  4.54MiB/s ETA 00:56
[throttling] Bandwidth throttling detected, making a new request. (peak rate = 5628.14KiB/s, block rate = 325.08KiB/s, downloaded 30976KiB before throttling)
[download]  48.3% of 440.98MiB at  4.44MiB/s ETA 00:51
[throttling] Bandwidth throttling detected, making a new request. (peak rate = 5628.14KiB/s, block rate = 370.88KiB/s, downloaded 30976KiB before throttling)
[download]  55.2% of 440.98MiB at  4.47MiB/s ETA 00:44
[throttling] Bandwidth throttling detected, making a new request. (peak rate = 5628.14KiB/s, block rate = 797.73KiB/s, downloaded 30976KiB before throttling)
[download]  62.0% of 440.98MiB at  4.55MiB/s ETA 00:36
[throttling] Bandwidth throttling detected, making a new request. (peak rate = 5628.14KiB/s, block rate = 283.98KiB/s, downloaded 30976KiB before throttling)
[download]  68.9% of 440.98MiB at  4.58MiB/s ETA 00:29
[throttling] Bandwidth throttling detected, making a new request. (peak rate = 5628.14KiB/s, block rate = 440.35KiB/s, downloaded 30976KiB before throttling)
[download]  75.7% of 440.98MiB at  4.62MiB/s ETA 00:23
[throttling] Bandwidth throttling detected, making a new request. (peak rate = 5628.14KiB/s, block rate = 576.62KiB/s, downloaded 30976KiB before throttling)
[download]  82.7% of 440.98MiB at  4.67MiB/s ETA 00:16
[throttling] Bandwidth throttling detected, making a new request. (peak rate = 5628.14KiB/s, block rate = 110.74KiB/s, downloaded 31488KiB before throttling)
[download]  89.6% of 440.98MiB at  4.59MiB/s ETA 00:09
[throttling] Bandwidth throttling detected, making a new request. (peak rate = 5628.14KiB/s, block rate = 319.12KiB/s, downloaded 31104KiB before throttling)
[download]  96.5% of 440.98MiB at  4.57MiB/s ETA 00:03
[throttling] Bandwidth throttling detected, making a new request. (peak rate = 5628.14KiB/s, block rate = 663.63KiB/s, downloaded 30976KiB before throttling)
[download] 100% of 440.98MiB in 01:36

pornhub.com:

$ youtube-dl --avoid-throttling -f 480P-600K -o a.mp4 -v "ADULT_CONTENT"
[debug] System config: []
[debug] User config: []
[debug] Command-line args: ['--avoid-throttling', '-f', '480P-600K', '-o', 'a.mp4', '-v', 'ADULT_CONTENT']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2016.12.09
[debug] Python version 3.5.2 - Darwin-15.4.0-x86_64-i386-64bit
[debug] exe versions: none
[debug] Proxy map: {}
[PornHub] ph574abb7e735a1: Downloading webpage
WARNING: unable to extract uploader; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
WARNING: unable to extract view count; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
WARNING: unable to extract comment count; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
[debug] Invoking downloader on 'ADULT_CONTENT'
[download] Destination: a.mp4
[download]   3.5% of 71.05MiB at  1.91MiB/s ETA 00:35
[throttling] Throttling started or is continuing, block rate = 92.54KiB/s, peak rate = 1952.35KiB/s
[throttling] Throttling detected! peak rate = 1952.35KiB/s, current rate = 46.26KiB/s, setting threshold to 0.27 and block size limit to 32KiB
[throttling] Bandwidth throttling detected, making a new request. (peak rate = 1952.35KiB/s, block rate = 92.54KiB/s, downloaded 3071KiB before throttling)
[download]   4.6% of 71.05MiB at 462.62KiB/s ETA 02:30
[throttling] Bandwidth throttling detected, making a new request. (peak rate = 1952.35KiB/s, block rate = 242.84KiB/s, downloaded 320KiB before throttling)
[download]   8.5% of 71.05MiB at 658.93KiB/s ETA 01:41
[throttling] Bandwidth throttling detected, making a new request. (peak rate = 1952.35KiB/s, block rate = 60.86KiB/s, downloaded 2848KiB before throttling)
[download]  12.4% of 71.05MiB at 740.34KiB/s ETA 01:26
[throttling] Bandwidth throttling detected, making a new request. (peak rate = 1952.35KiB/s, block rate = 126.56KiB/s, downloaded 2848KiB before throttling)
[download]  16.3% of 71.05MiB at 818.05KiB/s ETA 01:14
[throttling] Bandwidth throttling detected, making a new request. (peak rate = 1952.35KiB/s, block rate = 103.47KiB/s, downloaded 2816KiB before throttling)
[download]  20.2% of 71.05MiB at 858.01KiB/s ETA 01:07
[throttling] Bandwidth throttling detected, making a new request. (peak rate = 1952.35KiB/s, block rate = 152.97KiB/s, downloaded 2848KiB before throttling)
[download]  24.1% of 71.05MiB at 907.23KiB/s ETA 01:00
[throttling] Bandwidth throttling detected, making a new request. (peak rate = 1952.35KiB/s, block rate = 55.12KiB/s, downloaded 2848KiB before throttling)
[download]  28.1% of 71.05MiB at 928.58KiB/s ETA 00:56
[throttling] Bandwidth throttling detected, making a new request. (peak rate = 1952.35KiB/s, block rate = 59.97KiB/s, downloaded 2848KiB before throttling)
[download]  32.0% of 71.05MiB at 948.01KiB/s ETA 00:52
[throttling] Bandwidth throttling detected, making a new request. (peak rate = 1952.35KiB/s, block rate = 58.40KiB/s, downloaded 2848KiB before throttling)
[download]  35.9% of 71.05MiB at 955.26KiB/s ETA 00:48
[throttling] Bandwidth throttling detected, making a new request. (peak rate = 1952.35KiB/s, block rate = 104.96KiB/s, downloaded 2848KiB before throttling)
[download]  39.8% of 71.05MiB at 969.22KiB/s ETA 00:45
[throttling] Bandwidth throttling detected, making a new request. (peak rate = 1952.35KiB/s, block rate = 112.48KiB/s, downloaded 2848KiB before throttling)
[download]  43.7% of 71.05MiB at 982.12KiB/s ETA 00:41
[throttling] Bandwidth throttling detected, making a new request. (peak rate = 1952.35KiB/s, block rate = 99.67KiB/s, downloaded 2848KiB before throttling)
[download]  47.6% of 71.05MiB at 999.42KiB/s ETA 00:38
[throttling] Bandwidth throttling detected, making a new request. (peak rate = 1952.35KiB/s, block rate = 54.76KiB/s, downloaded 2848KiB before throttling)
[download]  51.5% of 71.05MiB at 1000.74KiB/s ETA 00:35
[throttling] Bandwidth throttling detected, making a new request. (peak rate = 1952.35KiB/s, block rate = 105.12KiB/s, downloaded 2848KiB before throttling)
[download]  55.5% of 71.05MiB at 1009.74KiB/s ETA 00:32
[throttling] Bandwidth throttling detected, making a new request. (peak rate = 1952.35KiB/s, block rate = 90.60KiB/s, downloaded 2848KiB before throttling)
[download]  59.3% of 71.05MiB at 1021.74KiB/s ETA 00:28
[throttling] Bandwidth throttling detected, making a new request. (peak rate = 1952.35KiB/s, block rate = 122.39KiB/s, downloaded 2816KiB before throttling)
[download]  63.2% of 71.05MiB at  1.01MiB/s ETA 00:25
[throttling] Bandwidth throttling detected, making a new request. (peak rate = 1952.35KiB/s, block rate = 75.48KiB/s, downloaded 2848KiB before throttling)
[download]  65.3% of 71.05MiB at  1.00MiB/s ETA 00:24
[throttling] Bandwidth throttling detected, making a new request. (peak rate = 1952.35KiB/s, block rate = 381.17KiB/s, downloaded 1472KiB before throttling)
[download]  69.1% of 71.05MiB at  1.02MiB/s ETA 00:21
[throttling] Bandwidth throttling detected, making a new request. (peak rate = 1952.35KiB/s, block rate = 316.73KiB/s, downloaded 2816KiB before throttling)
[download]  70.9% of 71.05MiB at  1.02MiB/s ETA 00:20
[throttling] Bandwidth throttling detected, making a new request. (peak rate = 1952.35KiB/s, block rate = 332.67KiB/s, downloaded 1248KiB before throttling)
[download]  70.9% of 71.05MiB at  1.01MiB/s ETA 00:20
[throttling] Bandwidth throttling detected, making a new request. (peak rate = 1952.35KiB/s, block rate = 361.57KiB/s, downloaded 64KiB before throttling)
[download]  74.9% of 71.05MiB at  1.01MiB/s ETA 00:17
[throttling] Bandwidth throttling detected, making a new request. (peak rate = 1952.35KiB/s, block rate = 51.19KiB/s, downloaded 2880KiB before throttling)
[download]  75.4% of 71.05MiB at 1017.96KiB/s ETA 00:17
[throttling] Bandwidth throttling detected, making a new request. (peak rate = 1952.35KiB/s, block rate = 378.02KiB/s, downloaded 384KiB before throttling)
[download]  79.4% of 71.05MiB at 1015.14KiB/s ETA 00:14
[throttling] Bandwidth throttling detected, making a new request. (peak rate = 1952.35KiB/s, block rate = 118.29KiB/s, downloaded 2880KiB before throttling)
[download]  79.7% of 71.05MiB at 1006.10KiB/s ETA 00:14
[throttling] Bandwidth throttling detected, making a new request. (peak rate = 1952.35KiB/s, block rate = 386.13KiB/s, downloaded 224KiB before throttling)
[download]  83.7% of 71.05MiB at 999.06KiB/s ETA 00:11
[throttling] Bandwidth throttling detected, making a new request. (peak rate = 1952.35KiB/s, block rate = 50.03KiB/s, downloaded 2912KiB before throttling)
[download]  87.7% of 71.05MiB at 978.31KiB/s ETA 00:09
[throttling] Bandwidth throttling detected, making a new request. (peak rate = 1952.35KiB/s, block rate = 51.73KiB/s, downloaded 2912KiB before throttling)
[download]  91.6% of 71.05MiB at 966.09KiB/s ETA 00:06
[throttling] Bandwidth throttling detected, making a new request. (peak rate = 1952.35KiB/s, block rate = 403.69KiB/s, downloaded 2816KiB before throttling)
[download]  95.5% of 71.05MiB at 964.43KiB/s ETA 00:03
[throttling] Bandwidth throttling detected, making a new request. (peak rate = 1952.35KiB/s, block rate = 240.86KiB/s, downloaded 2880KiB before throttling)
[download]  99.4% of 71.05MiB at 964.62KiB/s ETA 00:00
[throttling] Bandwidth throttling detected, making a new request. (peak rate = 1952.35KiB/s, block rate = 152.70KiB/s, downloaded 2848KiB before throttling)
[download] 100% of 71.05MiB in 01:15

[dstftw]

Breaks downloading completely:

PS C:\dev\youtube-dl\master> py -2.7 .\youtube_dl\__main__.py -v BaW_jenozKc
[debug] System config: []
[debug] User config: []
[debug] Command-line args: [u'-v', u'BaW_jenozKc']
[debug] Encodings: locale cp1251, fs mbcs, out cp866, pref cp1251
[debug] youtube-dl version 2016.12.15
[debug] Git HEAD: 4fee44d
[debug] Python version 2.7.12 - Windows-10-10.0.10240
[debug] exe versions: ffmpeg N-81192-g04da20e, ffprobe N-81192-g04da20e, rtmpdump 2.4
[debug] Proxy map: {}
[youtube] BaW_jenozKc: Downloading webpage
[youtube] BaW_jenozKc: Downloading video info webpage
[youtube] BaW_jenozKc: Extracting video information
[youtube] BaW_jenozKc: Downloading MPD manifest
WARNING: Requested formats are incompatible for merge and will be merged into mkv.
[debug] Invoking downloader on u'https://r2---sn-ug5onuxaxjvh-c5me.googlevideo.com/videoplayback?id=05a5bf8de9e8cca7&itag=137&source=youtube&requiressl=yes&mv=m&ms=au&pl=21&initcwndbps=3367500&mn=sn-ug5onuxaxjvh-c5me&mm=31&ratebypass=yes&mime=video/mp4&gir=yes&clen=2208750&lmt=1387961822987808&dur=9.800&key=dg_yt0&mt=1482062833&upn=0iq45Go0Fi0&signature=78D766DDE5C76DAFDA3174E7E3765B59F2149A7F.58FDA990D2FA0D507EE4AF9AD0E1AD5CBFA5B554&ip=<snip>&ipbits=0&expire=1482084789&sparams=ip,ipbits,expire,id,itag,source,requiressl,mv,ms,pl,initcwndbps,mn,mm,ratebypass,mime,gir,clen,lmt,dur'
[download] Destination: youtube-dl test video ''_-BaW_jenozKc.f137.mp4
[download]   0.0% of 2.11MiB at 111.11KiB/s ETA 00:19Traceback (most recent call last):
  File ".\youtube_dl\__main__.py", line 19, in <module>
    youtube_dl.main()
  File "C:\dev\youtube-dl\master\youtube_dl\__init__.py", line 445, in main
    _real_main(argv)
  File "C:\dev\youtube-dl\master\youtube_dl\__init__.py", line 435, in _real_main
    retcode = ydl.download(all_urls)
  File "C:\dev\youtube-dl\master\youtube_dl\YoutubeDL.py", line 1791, in download
    url, force_generic_extractor=self.params.get('force_generic_extractor', False))
  File "C:\dev\youtube-dl\master\youtube_dl\YoutubeDL.py", line 705, in extract_info
    return self.process_ie_result(ie_result, download, extra_info)
  File "C:\dev\youtube-dl\master\youtube_dl\YoutubeDL.py", line 751, in process_ie_result
    return self.process_video_result(ie_result, download=download)
  File "C:\dev\youtube-dl\master\youtube_dl\YoutubeDL.py", line 1435, in process_video_result
    self.process_info(new_info)
  File "C:\dev\youtube-dl\master\youtube_dl\YoutubeDL.py", line 1693, in process_info
    partial_success = dl(fname, new_info)
  File "C:\dev\youtube-dl\master\youtube_dl\YoutubeDL.py", line 1641, in dl
    return fd.download(name, info)
  File "C:\dev\youtube-dl\master\youtube_dl\downloader\common.py", line 353, in download
    return self.real_download(filename, info_dict)
  File "C:\dev\youtube-dl\master\youtube_dl\downloader\http.py", line 236, in real_download
    block_rate = block_size / (time.time() - block_start)
ZeroDivisionError: float division by zero

[Serkora]

Right, all that time thinking about how to deal with slow internet and got rekt by a too fast one.
Fixed, you can try again.

[dstftw]

> py -2.7 .\youtube_dl\__main__.py -v http://www.pornhub.com/view_video.php?viewkey=648719015 --proxy socks5://127.0.0.1:9050 --avoid-throttling
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'-v', u'http://www.pornhub.com/view_video.php?viewkey=648719015', u'--proxy', u'socks5://127.0.0.1:9050', u'--avoid-throttling']
[debug] Encodings: locale cp1251, fs mbcs, out cp866, pref cp1251
[debug] youtube-dl version 2016.12.31
[debug] Git HEAD: ab70f89
[debug] Python version 2.7.12 - Windows-10-10.0.10240
[debug] exe versions: ffmpeg N-81192-g04da20e, ffprobe N-81192-g04da20e, rtmpdump 2.4
[debug] Proxy map: {u'http': u'socks5://127.0.0.1:9050', u'https': u'socks5://127.0.0.1:9050'}
[PornHub] 648719015: Downloading webpage
[debug] Invoking downloader on u'http://cdn-d-vid-public.pornhub.com/videos/201306/28/14084201/vl_720_1073k_14084201.mp4?ttl=1483361684&ri=1228800&rs=1344&hash=cbb18edf1250c8f2e87993d87aca75f2'
[download] Resuming download at byte 20780290
[download] Destination: Seductive Indian beauty strips down and fingers her pink pussy-648719015.mp4
[download]  45.0% of 47.31MiB at 663.43KiB/s ETA 00:40
[throttling] Throttling started or is continuing, block rate = 160.45KiB/s, peak rate = 663.43KiB/s
[throttling] Throttling detected! peak rate = 663.43KiB/s, current rate = 80.20KiB/s, setting threshold to 0.34 and block size limit to 64KiB
[throttling] Bandwidth throttling detected, making a new request. (peak rate = 663.43KiB/s, block rate = 160.45KiB/s, downloaded 2004KiB before throttling)
Traceback (most recent call last):
  File ".\youtube_dl\__main__.py", line 19, in <module>
    youtube_dl.main()
  File "C:\dev\youtube-dl\master\youtube_dl\__init__.py", line 445, in main
    _real_main(argv)
  File "C:\dev\youtube-dl\master\youtube_dl\__init__.py", line 435, in _real_main
    retcode = ydl.download(all_urls)
  File "C:\dev\youtube-dl\master\youtube_dl\YoutubeDL.py", line 1791, in download
    url, force_generic_extractor=self.params.get('force_generic_extractor', False))
  File "C:\dev\youtube-dl\master\youtube_dl\YoutubeDL.py", line 705, in extract_info
    return self.process_ie_result(ie_result, download, extra_info)
  File "C:\dev\youtube-dl\master\youtube_dl\YoutubeDL.py", line 751, in process_ie_result
    return self.process_video_result(ie_result, download=download)
  File "C:\dev\youtube-dl\master\youtube_dl\YoutubeDL.py", line 1435, in process_video_result
    self.process_info(new_info)
  File "C:\dev\youtube-dl\master\youtube_dl\YoutubeDL.py", line 1699, in process_info
    success = dl(filename, info_dict)
  File "C:\dev\youtube-dl\master\youtube_dl\YoutubeDL.py", line 1641, in dl
    return fd.download(name, info)
  File "C:\dev\youtube-dl\master\youtube_dl\downloader\common.py", line 353, in download
    return self.real_download(filename, info_dict)
  File "C:\dev\youtube-dl\master\youtube_dl\downloader\http.py", line 334, in real_download
    data, throttled = self.speed_up(data, request, peak_rate, block_rate, byte_counter, throttling_threshold)
  File "C:\dev\youtube-dl\master\youtube_dl\downloader\http.py", line 43, in speed_up
    request = sanitized_Request(request.full_url, None, request.headers)
  File "C:\Python\Python27\lib\urllib2.py", line 254, in __getattr__
    raise AttributeError, attr
AttributeError: full_url

[Serkora]

Sorry, really inconvenient for me to test on various python versions right now. (Though, for changes in such a core file I probably should have).

[rodcul]

Hey @Serkora any chance you could get this working? I'm trying to download over 125 tracks from Mixcloud and it would be REALLY useful...

[Serkora]

@rodcul
Is it not working for you right now? What kind of errors are you seeing etc? I'm not aware of any other issues, but would gladly solve them. I'm just waiting for dstftw to get around this request again to decide what to do with it.

[Gunny123]

Any update on this issue?

[Serkora]

Hi @dstftw ,
Is this thing rejected and abandoned or there's a chance it might be integrated? I'm happy to fix/improve whatever's required, just need some feedback.

[zannix]

I'm eager to see this working, especially now that YouTube is throttling download speed. It's really important.

[mzso]

@zannix commented on 2018. jan. 18. 09:59 CET:

I'm eager to see this working, especially now that YouTube is throttling download speed. It's really important.

I noticed downloads being really slow 1-2 days ago. But coincidentally I upgraded yt-dl not much before it.
So it's youtube that started doing things differently and not yt-dl?

[zannix]

Yes, see the references a few posts above. This functionality would really be priceless right now.

[arunesh90]

@yan12125 bit weird that you refer people to here for a "possible solution" while the creator of the PR had asked a collaborator for feedback but got ignored

[dstftw]

Note to commenters: all comments that don't add anything useful to the discussion will be removed, if you have nothing valuable to add then just don't post.

[yan12125]

            new_data = self.ydl.urlopen(request) 

This looks like a recursive approach. I'm not sure if it will cause stack overflow on very long videos. This pull request needs rebase anyway.