Add support for lua yields and basic continuation support

[cheesycod]

This PR adds support for continuation functions in all Lua versions other than Lua 5.1 and LuaJIT with support for the Luau yieldable continuations fflag as well as the prerequisite basic support for Rust yielding in Lua (as basic yielding from Rust is needed to use continuations in practice). This may be expanded upon in the future.

Closes PR #518 as well

Some notes:

• This PR makes a new callback_error_ext_yieldable method which is a variant of callback_error_ext with support for yielding. This is required due to the return type for yielding threads needing to be a c_int and not a generic R (the value from lua_yield needs to be returned to actually yield in luau and is expected in other lua variants as well). With the R generic on callback_error_ext, it is not possible to do the yield and so a separate yieldable function is anyways hence required. Also, most operations using callback_error_ext right now do not support or need to support yielding. Lastly, on non-Luau versions, an additional function parameter, in_callback_with_continuation, is needed to determine whether to use lua_yieldk or just normal lua_yield
• Async functions currently do not support yielding right now due to safety concerns. Attempting to yield within an async function will be ignored
• lua_yield on luau requires a lua_pop to clear thread state as of right now due to yieldable continuations fflag preserving entire thread state (this is intended behavior) and leading to incorrect results in continuation. See luau continuation does not include yielded/resumed args luau-lang/luau#1867 for more info on why the lua_pop is needed in the yield code
• In non-Luau variants, a yield_continuation boolean in ExtraData is used to either call lua_yieldk or lua_yield. In Luau, lua_yield is always called as continuations are directly attached to the C closure
• Yielding in continuations is fully supported on all Lua versions supporting continuations (Luau needs fflag enabled to use this however)

[khvzak]

Could you provide some context please, what problem are you trying to solve?
Lua/C continuations are rather workaround to allow suspending C functions. In Rust world this is solved by asynchronous functions that are perfectly integrated with Lua.

[cheesycod]

Could you provide some context please, what problem are you trying to solve? Lua/C continuations are rather workaround to allow suspending C functions. In Rust world this is solved by asynchronous functions that are perfectly integrated with Lua.

In my case, my users expect Roblox-style behavior in things like async etc as that's kinda what I advertise in my product (of user scriptable Discord bot with Roblox Luau). mlua async has some quirks there where the behavior doesn't exactly match such as task.wait which must yield according to Roblox and then resume the thread with the slept for seconds etc). In a custom async system, yielding and continuations makes it nicer and more performant to actually run async functions. [While I haven't fully incorporated continuations yet, I do have plans to use them in porting my scheduler beyond luau (it’s needed there as resumeerror does not exist beyond luau).

Also, my custom async is able to have much better CPU and resource usage than just using mlua async across lots of threads because it's tailor made for my specific use cases and needed functionality (currently sits at 1.7% CPU on top even at peak load vs a lot more than 1.7% on top last time I checked) which basically makes mlua async a nonstarter for me (I have limited CPU to provide to users).

Finally, given that I anyways directly need yielding right now and very likely will be needing continuations in the near future, I decided to just add both in one single PR.

(In fact, it's so much nicer to have native yielding and continuations that I've already started using this in production [its noticeably faster and nicer ergonomics wise to yield from Rust than to have a Lua function just to yield with or without continuations when I tested it on user code])

[cheesycod]

I do understand if you arent ready to merge this however as most normal users can use mlua async. This is mostly for the niche use cases in which mlua async doesn't work out so well for whatever reason (semantics, resource usage, other quirks etc). In either case, I'll prob keep using this PR in my bot since its useful to me personally but yeah. @khvzak

[cheesycod]

Sorry for this last comment as I needed some time to actually try doing smth I’ve wanted to do for a while without continuations to see why I needed it back then.

One thing you cannot do without continuations is propagate an error across a yielding thread in non-luau (luau has resume error). This makes porting my scheduler beyond luau impossible (you need a continuation to propagate the error correctly)

[khvzak]

Thank you for the context and explanation.

mlua was never designed to support all (especially niche) use cases, and merging PRs with relatively complex code creates a significant long-term maintenance responsibility for me as the maintainer - ensuring the code remains sound, well-tested, and compatible across updates.
Given that this feature designed to solve problems for a small number of projects, I'd like to see some community interest before considering merging it into the main codebase.

I'm open to reasonable ideas and happy to incorporate a feedback when there's clear benefit for mlua.

Also, my custom async is able to have much better CPU and resource usage than just using mlua async across lots of threads because it's tailor made for my specific use cases and needed functionality (currently sits at 1.7% CPU on top even at peak load vs a lot more than 1.7% on top last time I checked) which basically makes mlua async a nonstarter for me (I have limited CPU to provide to users).

mlua async support is tuned for high performance and when integrated with tokio (or other runtimes) demonstrate an excellent performance.
For example, in my project it processes hundreds of thousands QPS with footprint comparable to a pure-Rust solution.

[cheesycod]

Thank you for the context and explanation.

mlua was never designed to support all (especially niche) use cases, and merging PRs with relatively complex code creates a significant long-term maintenance responsibility for me as the maintainer - ensuring the code remains sound, well-tested, and compatible across updates. Given that this feature designed to solve problems for a small number of projects, I'd like to see some community interest before considering merging it into the main codebase.

I'm open to reasonable ideas and happy to incorporate a feedback when there's clear benefit for mlua.

Also, my custom async is able to have much better CPU and resource usage than just using mlua async across lots of threads because it's tailor made for my specific use cases and needed functionality (currently sits at 1.7% CPU on top even at peak load vs a lot more than 1.7% on top last time I checked) which basically makes mlua async a nonstarter for me (I have limited CPU to provide to users).

mlua async support is tuned for high performance and when integrated with tokio (or other runtimes) demonstrate an excellent performance. For example, in my project it processes hundreds of thousands QPS with footprint comparable to a pure-Rust solution.

Yeah, I figured, I'll keep this PR open then in case other people run into this limitation and want to chime in.

In the meantime, I'll keep using a fork of mlua which supports my niche use cases (like list of aux threads, the thread optimizations (its hot path code for me) and this yielding/continuation PR) along with removing mlua features I don't use (like mlua's async)