Skip to content

Remove scheduler plugin machinery, make Sch programmable #166

Open
@jpsamaroo

Description

@jpsamaroo

The scheduler plugin system is (to my knowledge) unused by any modern users of Dagger. However, the presence of the potential for multiple external schedulers makes changing the scheduler API technically a breaking change, which would slow Dagger's development were that actually a concern. Of course, we still want to support different kinds of scheduling algorithms and optimizations that best suit a user's use case and DAG structure, so we should make the Sch scheduler user-programmable by making the current worker pressure algorithm optional, and having schedule! call a user-defined callable.

The option to change the internal scheduler algorithm is intended to only be used by adventurous users who understand that deadlocks/livelocks/hangs/etc. are all possible when changing the default scheduler, although we should expose "safe" semi-internal APIs that can perform common tasks correctly (which the default scheduler should also use whenever possible). Hopefully this change can spur developers and adventurous users to experiment with (and contribute) new scheduling algorithms which are better than the default scheduler for certain classes of workloads, with Dagger itself becoming the foundation for user-defined scheduling of distributed Julia code.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions