RTMP Streams Won't Scrape (was: Yellowclaw won't download)

[Miserlou]

I have no idea why this URL won't work:

https://soundcloud.com/yellowclaw

[Miserlou]

Downloading: Yellow Claw & GRX & Cesqeaux - Psycho *FREE DOWNLOAD*
Problem downloading Yellow Claw & GRX & Cesqeaux - Psycho *FREE DOWNLOAD*
> /Users/tuttle/Projects/SoundScrape/soundscrape/soundscrape.py(58)main()
-> print e
(Pdb) e
HTTPError('404 Client Error: Not Found',)
(Pdb) print track.keys()
[u'attachments_uri', u'video_url', u'track_type', u'release_month', u'original_format', u'label_name', u'duration', u'id', u'streamable', u'user_id', u'title', u'favoritings_count', u'commentable', u'label_id', u'download_url', u'state', u'downloadable', u'waveform_url', u'sharing', u'description', u'release_day', u'purchase_url', u'permalink', u'purchase_title', u'stream_url', u'key_signature', u'user', u'genre', u'isrc', u'download_count', u'permalink_url', u'playback_count', u'kind', u'release_year', u'license', u'artwork_url', u'created_at', u'bpm', u'uri', u'original_content_size', u'comment_count', u'release', u'tag_list', u'embeddable_by']
(Pdb) track.embeddable_by
u'all'

What the fuck.

[Miserlou]

Okay, so this is mostly solved. There are a few tracks which 404 in the API, but are still playable in the web browser. This also will no download from the API, even with the MAGIC_CLIENT_ID, there is still a 404.

Turns out these are still using RTMP rather than HTTP. I don't know why!

Here is an example track:
https://soundcloud.com/yellowclaw/yellow-claw-lny-tnz-last-night

and the resulting RTMP URL:
rtmp://ec-rtmp-media.soundcloud.com:80/mp3:sECrW64oG8x2.128?9527d18f1063a01f059bf10590159adb10dea0996b8c0cdb6748972125188097b8af051dbd61b6a9a7304fc9b1b529cde0180b70a74d23c7aedf6d243bd13c6a2c99035d795f31d30dd30d9190756b03d4e0e225c9988f229441
Looks like the format is:
rtmp://ec-rtmp-media.soundcloud.com:80/mp3:SONG_ID.BITRATE?MAGIC_RTMP_KEY

Investigating further.. fucking terrible song too...

[Miserlou]

Okay, looks like there is actually a pretty decent python rtmp library emerging ( https://github.com/chrippa/python-librtmp by @chrippa ), but the dependencies are quite heavy and it might be overkill to include all of that just for the case of the few songs on SoundCloud which still require RTMP.

I'm closing this issue for now, if there is a good argument (read: a good song) for including this, I will reopen.

[Miserlou]

Reopening because this doesn't work for Pusha T. Goddammit.

https://soundcloud.com/pushat

[chrippa]

Hi, since I got pinged here I might aswell offer my brief opinion.

I also think using my library for this purpose is overkill, if you only need to download a RTMP stream without doing anything fancy (like using remote method calls), you're probably better off writing a small subprocess wrapper around the command line utility rtmpdump instead.

For example I was able to dump these streams via this command:

$ rtmpdump -r "rtmp://ec-rtmp-media.soundcloud.com:80" -y "mp3:sECrW64oG8x2.128?9527d18f1063a01f059bf10590159adb10dea0996b8c0cdb674897212518859c239dffcf91de47f1797a9b3d32fa3e979f526c21bde11161a73ce69f4e7e86d69e10d4262756827dd2a3f122c09c1f2b34b51eead57dc47f38fb" -o file.mp3

[Miserlou]

Thanks for the input @chrippa!

That was my next thought, but that still requires the use of an external program, which complicates the install process. The goal is really to be able to be able to 'pip install soundscrape' anywhere and have it work right out of the box.

I have found a pure python implementation of RTMP here, which may suit my needs: https://github.com/lumaedu/rtmplite/blob/master/rtmpclient.py

The "downside" is that it's GPL'd, but that doesn't really bother me, I'm fine with the GPL.

[Miserlou]

Or maybe not. Doesn't seem to work with the aforementioned streams.