Skip to content

limiting the maximum number of download threads #41

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

linsalrob
Copy link

If you provide aria2c with more than 16 threads it dies rather rudely.

Here is an example, requesting 64 download threads:

04/14/2025 03:13:19 PM INFO: Kingfisher v0.4.1
04/14/2025 03:13:34 PM INFO: Attempting download method ena-ftp for run SRR8482173 ..
04/14/2025 03:13:34 PM INFO: Querying ENA for FTP paths for SRR8482173..
04/14/2025 03:13:36 PM INFO: Downloading ftp.sra.ebi.ac.uk/vol1/fastq/SRR848/003/SRR8482173/SRR8482173.fastq.gz ..
Exception: [AbstractOptionHandler.cc:69] errorCode=28 We encountered a problem while processing the option '--max-connection-per-server'.
  -> [OptionHandlerImpl.cc:184] errorCode=1 max-connection-per-server must be between 1 and 16.
Usage:
 -x, --max-connection-per-server=NUM The maximum number of connections to one
                              server for each download.

                              Possible Values: 1-16
                              Default: 1
                              Tags: #basic, #http, #ftp
04/14/2025 03:13:37 PM WARNING: Method ena-ftp failed, error was Command 'aria2c -x64 -o SRR8482173.fastq.gz 'ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR848/003/SRR8482173/SRR8482173.fastq.gz'' returned non-zero exit status 28.
04/14/2025 03:13:37 PM WARNING: Method ena-ftp failed
Traceback (most recent call last):
  File "/kingfisher/bin/kingfisher", line 323, in <module>
    main()
  File "/kingfisher/bin/kingfisher", line 266, in main
    kingfisher.download_and_extract(
  File "/kingfisher/bin/../kingfisher/__init__.py", line 72, in download_and_extract
    download_and_extract_one_run(run, **kwargs)
  File "/kingfisher/bin/../kingfisher/__init__.py", line 365, in download_and_extract_one_run
    raise Exception("No more specified download methods, cannot continue")
Exception: No more specified download methods, cannot continue

If num_threads is above 16, this PR just sets num_threads to 16 and provides a warning.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant