Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added -q --sequential option for running the program single-threaded. #14

Closed
wants to merge 4 commits into from

Conversation

bkcnimbis
Copy link

This is a nice tool but wasn't working for me, so I made it work for me. The option to run single-threaded is very useful where memory usage is a concern and running time isn't. Additionally, this addresses issue #7 by cutting out the call to multiprocessing methods when the option is specified.

The managedblockchain commit was necessary for the tool to run on my machine, so it is included here as well.

This tool can be very memory intensive, enough so to cause the tool to fail to finish executing on less powerful hardware.  This modification allows the user to trade slower operation for less memory consumption, which is necessary for some uses.

In my case, this option takes about 3x longer to run, using about 10x less memory space.

Additionally, this is a suitable workaround for issue JohannesEbke#7.
@bkcnimbis bkcnimbis marked this pull request as ready for review May 15, 2019 18:14
@felixhuttmann
Copy link
Contributor

Your workaround for the managedblockchain error still leaves the tests failing:
https://travis-ci.org/JohannesEbke/aws_list_all/jobs/532946893

I think managedblockchain has to be added to the SERVICE_BLACKLIST and I have updated my pull request accordingly. #12

PR JohannesEbke#12 uses this solution and travis is happy, so I want a happy Travis too
yapf fails and complains about my linesplit, so here's the same thing as one line, like it asks for.
@bkcnimbis
Copy link
Author

bkcnimbis commented May 17, 2019

Passes tests now. To be clear, f0c7706 (and 6e04ba7 depending on what you're doing with yapf) is the sole commit containing a useful contribution, and I'll happily resubmit a PR rebased from any passing commit @JohannesEbke likes.

I'd like to see this option pulled in

JohannesEbke added a commit that referenced this pull request Jul 28, 2019
Inspired and generalized from @bcknimbis PR #14.
@JohannesEbke
Copy link
Owner

Hi @bkcnimbis ; thanks for the Idea and all the work! I've picked up the idea, but implemented it as a "--parallel N" flag, where N is the number of parallel requests in flight. I've left the pool in, since #7 has been fixed otherwise.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants