Questions about the Flux simulator and how to writing new schedulers

[grondo]

@adfaure asked in #2561

If I want to use Flux simulation, do I need to use this branch or the simulation is also possible in the master branch ?

Thank you.

[grondo]

It depends on what you mean by simulation. What are you looking to do? For example, the mainline version of flux-core can simulate job execution when the attributes.system.exec.test.run_duration attribute is set.

[grondo]

From @adfaure:

I am interested to understand the simulation capabilities of flux to have a global picture of what it offers, especially about scheduling simulation.
Especially:

What is the simulation model for the jobs;
What is the simulation model of the platform;
How can I write a new scheduling algorithm.
Next year, I managed to make the simulator of this PR working, I will try to do the same with the current master branch.

Thank you for your quick answer.



[grondo]

The scheduler in Flux is an independent module. To develop a new scheduling algorithm you can either write a new scheduler module (using perhaps the extremely simple included scheduler as a starting point), or by developing new planner or matching plugins for the Fluxion graph based scheduler.

[adfaure]

Is it possible to feed a workload (like SWF) for instance to flux-core with attributes.system.exec.test.run_duration?

[grondo]

Ah, no I think you will want the simulator for that because attributes.system.exec.test.run_duration (so-called mock execution) only gives you fake execution (jobs are not actually executed), but everything still runs in real time, not simulated time. You could have a script that submitted jobs at the correct time as specified in the SWF input, but you'd have to wait for the real-time events to happen.

Mock execution also doesn't currently give you a way to set abnormal exit conditions (except for some specific job exceptions) on jobs which I believe would be necessary in a real simulator.

Thus, there is still a big role for the simulator and it sounds like that is what you really need here.

@SteVwonder may have additional comments.

[SteVwonder]

@grondo summarized things pretty well. For now, the only way to use the simulator is with the PR branch (which is a much older version of Flux). Right now it takes job traces in the sacct format, but it wouldn't be that hard to update the python code to read SWL (or to write a sacct -> SWL translator). The simulator was designed such that it would work with the main/production schedulers in flux (both sched-simple in flux-core and fluxion in flux-sched). So if you want to implement and test out a new scheduling algorithm, you can either implement a whole new scheduler OR you could implement a plugin for the Fluxion scheduler (much less work).

In terms of getting the simulator merged, the plan is indeed to eventually get it merged, but I'm short on cycles to do so right now. I'd be happy to accept help if you or anyone else wants to help update the PR. Since that PR was posted, Flux now has jobtap plugins which are a really good fit for the simulator code within job-manager, so we will probably want to pull that code out into a plugin.

[SteVwonder]

PS - I'd be happy to provide additional details/pointers to anything listed above (SWL format reader, new scheduler, Fluxion plugin, etc).

[dongahn]

We also have resource-query level simulation if you want to try your own resource representation and/or how jobspecs are matched with one or more match policies: https://github.com/flux-framework/flux-sched/blob/master/resource/utilities/README.md