Configuring Max Current Items

[eddie3716]

Hi Daniel,

I opened a pull request #285 regarding adding the ability to configure the number of max current tasks items via WorkflowOptions using a multiplier. I'm in a situation, as I imagine other might also be in, where our legacy product architecture doesn't yet support scaling CPU cores or service nodes on the fly without a lot of red tape, and this is the only mechanism currently provided to increase the max number of concurrent tasks. In lieu of a better product architecture (this is legacy software I'm dealing with), this solution I proposed seems quite reasonable. What are your thoughts?

[danielgerlag]

I think setting the concurrency to the number of processors is a good heuristic.
Unless your workflows are doing a lot of intensive I/O, increasing this number won't really help much.

[eddie3716]

My workflow aren't particularly resource intensive, so this isn't to increase performance, but I do have one long running step that blocks while waiting for some other long-running process to complete. When this step is blocked, and if I have a lot of these steps blocking (from parallel workflows running), then all of my active tasks are consumed. So instead of having all of my active tasks monopolized by long-running steps that only wait for another process to complete, I'd like to be able to kick off other workflows. Does this make sense?

My proposed change is totally opt-in only. If someone wants to keep the existing heuristic, they can, but if they want flexibility, they have that option as well.

[eddie3716]

@danielgerlag thanks for the feedback. I saw your comment in the pull request. I created a new pull request for this change: #287. (I deleted and re-forked my repo, and implemented the suggested changes from scratch). I've closed the old pull request #285 as obsolete.

[SebastienLagrange]

Hello,
I am very happy to see that you implemented this change that I was just thinking on !
I implement workflows that contains lot of steps that actually are waiting for external resources. And I start hundreds of workflows at the same time. The current implementation limiting the number of tasks to the number of CPU is very limiting the performance to process all the hundreds of workflows... and I see my CPU is zero all the time because the steps actually call REST services that are on other server so the workflow engine is just waiting all the time. In that case it would possible to start 50 workflows at the same time even with my 2 CPU, and that would really improve the time to process all the hundreds of workflows...
daniel do you think this modification is good, will you accept the pullrequest and make this new option available from your master package ?
Thank you all !