[RISCV] running zephyr scheduler in user-/supervisor mode (instead of machine mode)?

[tswaehn]

Is your feature request related to a problem? Please describe.
It appears to me that on RISCV the expectation is, that zephyr runs in machine mode (guessing from here). I am wondering why this is needed. Lets say we have a supervisor running multiple applications, where zephyr is just one of them. Then zephyr should be able to run in USER mode.

Describe the solution you'd like
I think it could be accomplished when using a generic hook for the "yield" and a generic hook for a timer interrupt. Additionally the switching routine could use defined CSRs (currently fixed machine mode CSRs are expected). Speaking of that actually there are RISCV cpus, where not all CSRs are implemented ex: mscratch might not be available => want to redefine and use another CSR instead?

Describe alternatives you've considered
Copy paste the RISCV architecture, name it RISCV-IN-USER and replace the interrupt/ ECALL handling. Which might a bit messy - because actually only the interrupt handling need to be changed a little.

Did anybody already think about that - or is that completely non-sense?

[carlescufi]

@fkokosinski @carlocaione @npitre

[npitre]

Random notes:

• Zephyr's "user mode" is optional. When configured out, there are no
  scratch register usage.

• It should be possible for Zephyr to run in s-mode instead of m-mode and
  still support its own "user mode" in u-m,ode. This is already the case
  on Linux (configurable at build time).

• Having Zephyr running entirely in u-mode would be more tricky. It wouldn't
  be able to support its own "user mode" threads of course, and some
  special abstractions around IRQ manipulation and handling would be needed.

I think moving Zephyr from m-mode to s-mode when available should be rather
easy to do. More than that is probably doable given many restrictions, but
whether if it is worth it or not pretty much depends on your motivation.

[tswaehn]

Thank you for your notes. I mostly agree with it.

Having Zephyr running entirely in u-mode would be more tricky. It wouldn't
be able to support its own "user mode" threads of course, and some
special abstractions around IRQ manipulation and handling would be needed.

Regarding the zephyr fully in user mode. I think of an oportunity where user interrupts can be used exactly the same way like in machine mode. instead of MTVEC = > UTVEC; MSTATUS = > USTATUS; MRET => URET, ... and so on.

Additonally I was wondering, if I would propose changes in that way and create a PR, would that be in the range of acceptable, or rejected, because of any strategic / theoretical / conceptional reasons.

[npitre]

MTVEC = > UTVEC; MSTATUS = > USTATUS; MRET = > URET, ... and so on.

Right. If UTVEC and friends are implemented then MSTATUS/SSTATUS/USTATUS
etc. could be abstracted behind some
macros. Linux does this:

#ifdef CONFIG_RISCV_M_MODE
# define CSR_STATUS     CSR_MSTATUS
# define CSR_IE         CSR_MIE
# define CSR_TVEC       CSR_MTVEC
# define CSR_SCRATCH    CSR_MSCRATCH
# define CSR_EPC        CSR_MEPC
# define CSR_CAUSE      CSR_MCAUSE
# define CSR_TVAL       CSR_MTVAL
# define CSR_IP         CSR_MIP
[...]
#else /* CONFIG_RISCV_M_MODE */
# define CSR_STATUS     CSR_SSTATUS
# define CSR_IE         CSR_SIE
# define CSR_TVEC       CSR_STVEC
# define CSR_SCRATCH    CSR_SSCRATCH
# define CSR_EPC        CSR_SEPC
# define CSR_CAUSE      CSR_SCAUSE
# define CSR_TVAL       CSR_STVAL
# define CSR_IP         CSR_SIP
[...]
#endif

And then only a few corner cases are conditionally selected with
CONFIG_RISCV_M_MODE in the actual code. Applying this approach to Zephyr
and extending it to u-mode as well (for available U regs) would be perfectly
fine to me.

However, given this is deep architecture stuff, we'd need a way for CI to
make sure the u-mode build won't go broken i.e. this needs to work using
QEMU. I don't know if QEMU has e.g. UTVEC implemented but a quick grep
seems to indicate it is not.

[tswaehn]

However, given this is deep architecture stuff, we'd need a way for CI to
make sure the u-mode build won't go broken i.e. this needs to work using
QEMU. I don't know if QEMU has e.g. UTVEC implemented but a quick grep
seems to indicate it is not.

It may also need some m-mode/s-mode starting code to prepare the forwarding/delegating of interrupts from machine to user and also take care of actual timer setup (user may not have access to it directly). additionally I m wonding if syscall like openSBI could be used to communicate between user mode and machine mode supervisor.

[achech]

If you decide to use https://github.com/riscv-software-src/opensbi, it will takeover/hande the M-mode, leaving you with S and U mode for the OS.

[tswaehn]

If you decide to use https://github.com/riscv-software-src/opensbi, it will takeover/hande the M-mode, leaving you with S and U mode for the OS.

yes, then zephyr OS will run S or U mode then. ex. running a save device with secure/certified drivers and user zephyr OS for application.

additionally the user zephyr OS will be compatible to any host providing the opensbi interface.

[con-pax]

Just to note, I don't thing there is an equivalent to mhartid in lesser privileged modes (though, it could be passed in a0). There would be a chunk of work needed for SMP and the PLIC driver for instance. Also to note, I think Zephyr running in s-mode would be amazing! 😃

[achech]

The mhartid is transferred in
https://github.com/riscv-software-src/opensbi/blob/master/lib/sbi/sbi_hsm.c#L157
for both, the init_coldboot and init_warm_startup

[tswaehn]

checklist for changes

• custom version of z_soc_irq_lock()
• replace __initialize - reading of HART from mhartid => uhartid
• replace __soc_handle_irq()
• just to be on safe side, there should be a RISCV toolchain option where the M-extension is not included

... in progress

[con-pax]

checklist for changes

• custom version of z_soc_irq_lock()
• replace __initialize - reading of HART from mhartid => uhartid
• replace __soc_handle_irq()
• just to be on safe side, there should be a RISCV toolchain option where the M-extension is not included

... in progress

Really cool! Let me know if there is anything you want tested from my side. I just so happen to have a board with a heterogenous core-complex sitting on my desk 😃