Open
Description
I think a nice additional option would be a minchunksize
that goes with nchunks
. Basically, the way I'd want it to work is
scheduler=StaticScheduler(nchunks=10, minchunksize=5)
tforeach(f, 1:3; scheduler) # does not spawn tasks (serial fallback)
tforeach(f, 1:10; scheduler) # spawns 2 tasks
tforeach(f, 1:20; scheduler) # spawns 4 tasks
tforeach(f, 1:100; scheduler) # spawns 10 tasks
tforeach(f, 1:10_000; scheduler) # still spawns 10 tasks
spawn 10 tasks. I think this is really useful for the majority of naively parallel operations on well behaved data when the mapping / reducing functions are cheap, because you can basically say "I want this parallelized, but don't create more tasks than necessary"