Skip to content

Change parallel processing for assemble_reads.py #31

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Jun 7, 2022

Conversation

bmichanderson
Copy link
Contributor

The previous version of this script would create corescores and coresmemory demands on the system. This change adds an argument --instances that hopefully makes this more explicit.

The previous version of this script would create cores*cores and cores*memory demands on the system. This change adds an argument `--instances` that hopefully makes this more explicit.
@bmichanderson
Copy link
Contributor Author

It seems like (based on my tests on my computer) that the current version of the script will actually create "cores" number of parallel assemblies, each using "cores" cores and "max_memory". This is not what I think makes sense, as it will go way beyond the actual "max_memory" setting and available cores. I've added an argument, "instances" that makes it more explicit. Now, the "cores" argument is per SPAdes assembly, the "max_memory" argument is per SPAdes assembly, and the "instances" determines how many the user wants to run in parallel. The actual demand on the system is "instances" * "cores" and "instances" * "max_memory".

It seems like this is what was perhaps going to be implemented, given "cores" and "max_memory" are passed to the spades_assembly function along with "args", but neither "cores" nor "max_memory" are used in the function (just "args.cores" and "args.max_memory"), leading to the weird behaviour.

There may be other ways to ideally fix this behaviour, so perhaps this pull request is not desired.

Cheers,
Ben

@tandermann tandermann merged commit b80c4bd into AntonelliLab:master Jun 7, 2022
@bmichanderson bmichanderson deleted the parallel_patch branch June 7, 2022 15:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants