Description
The scheduler plugin system is (to my knowledge) unused by any modern users of Dagger. However, the presence of the potential for multiple external schedulers makes changing the scheduler API technically a breaking change, which would slow Dagger's development were that actually a concern. Of course, we still want to support different kinds of scheduling algorithms and optimizations that best suit a user's use case and DAG structure, so we should make the Sch scheduler user-programmable by making the current worker pressure algorithm optional, and having schedule!
call a user-defined callable.
The option to change the internal scheduler algorithm is intended to only be used by adventurous users who understand that deadlocks/livelocks/hangs/etc. are all possible when changing the default scheduler, although we should expose "safe" semi-internal APIs that can perform common tasks correctly (which the default scheduler should also use whenever possible). Hopefully this change can spur developers and adventurous users to experiment with (and contribute) new scheduling algorithms which are better than the default scheduler for certain classes of workloads, with Dagger itself becoming the foundation for user-defined scheduling of distributed Julia code.