Skip to content

Conversation

@atr0s
Copy link
Contributor

@atr0s atr0s commented Oct 23, 2016

For some sets, soundscrape will download all tracks available rather than downloading the number of tracks passed via -n on the command line.

It seems to manifest when tracks need to be fetched via get_soundcloud_api_playlist_data() rather than via resolved.tracks. Resizing the array and then setting aggressive to True so it won't try downloading the files further down seems to have fixed the issue.

…ine when tracks to download is retrieved via get_soundcloud_api_playlist_data()

Setting aggressive to True right after to make sure soundscrape won't try downloading the files again
@Miserlou
Copy link
Owner

Looks like Travis failed, but I'm not sure if it's related. Can you see if the tests pass locally for you? Could be DDoS related..

Shrinking the array to get around an issue whereby the number of files being scraped wasn't being respected
@atr0s
Copy link
Contributor Author

atr0s commented Oct 24, 2016

It looks like 2 tests have failed, even locally:

FAIL: test_mixcloud (test.TestSoundscrape)

Traceback (most recent call last):
File "/Users/atros/Documents/Projects/SoundScrape/tests/test.py", line 114, in test_mixcloud
self.assertTrue((new_mp3_count > mp3_count) or (new_m4a_count > m4a_count))
AssertionError: False is not true
-------------------- >> begin captured stdout << ---------------------
HTTPSConnectionPool(host='stream14.mixcloud.com', port=443): Max retries exceeded with url: /c/originals/3/9/f/4/f513-a72c-4741-9e4f-c021bb01058f.mp3?v=0.1 (Caused by NewConnectionError('<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x10e7e1150>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known',))

--------------------- >> end captured stdout << ----------------------

FAIL: test_soundcloud_hard (test.TestSoundscrape)

Traceback (most recent call last):
File "/Users/atros/Documents/Projects/SoundScrape/tests/test.py", line 54, in test_soundcloud_hard
self.assertTrue(new_mp3_count == 1) # This used to be 3, but is now 'Not available in United States.'
AssertionError: False is not true

The first one seems to be related to a DNS issue, it looks like stream14.mixcloud.com doesn't resolve any longer. I tried it on different DNS servers and the outcome was the same.

As for the second issue, it looks like it's the same sort of issue I'm trying to fix but it's getting hit for other reasons. I've committed new changes to my fork that should address it. As for the first one, I believe it the test needs to be updated.

@brachna
Copy link
Contributor

brachna commented Nov 4, 2016

@atr0s
soundscrape was updated, test_mixcloud should pass now.

@Miserlou
Copy link
Owner

Miserlou commented Nov 4, 2016

I had to rerun the job a few times but works now

@Miserlou Miserlou merged commit 680ff44 into Miserlou:master Nov 4, 2016
@Miserlou
Copy link
Owner

Miserlou commented Nov 4, 2016

Thanks for your contributions!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants