Catalyast V16 with MTK10 planning

[isaacsas]

It is going to be a major overhaul to update for MTK10. Let's use this issue to track things we need to be aware of

• Switching to System from all the other system types. Here we probably want to replace all the convert(::AbstractSystem,...) calls and rely on physical_scales to decide at what level a reaction is modeled in the generated system. We could then have lightweight functions like make_ode_model, make_sde_model, make_jump_model that set the physical scale uniformly for every reaction, while make_hybrid_model requires a default be given for each reaction.
• Field updates: we should probably add support for user-provided Brownians and jumps that get merged with any we generate from reactions during conversion. Along those lines we need to check how MTK handles mixing noise_eqs and Brownians, or if they are mutually exclusive options.
• Allow users to select if they want a System with Brownians or noise_eqs when converting to SDE models.
• Update compositional modeling dispatches to just work with ReactionSystems and general Systems.
• Update tests for all the changes like the new merged u0 and p maps.
• Update docs appropriately...
• Use Pre in all callback related tests.
• How to handle updating for extensions. Here we will either need to drop some extensions till they update for MTK10, or be blocked till they are updated.
• Make sure to have a final review of the MTK changes before we release, so we don't miss anything like we did with MTK9 and parameter arrays.
• Make sure ReactionSystem metadata works with the same API as MTK now uses.
• Drop == for ReactionSystems per refactor: remove old == and hash implementation for System ModelingToolkit.jl#3683

[ChrisRackauckas]

Allow users to select if they want a System with Brownians or noise_eqs when converting to SDE models.

This doesn't really make sense because they are the same. One is just further down in the lowering process, but they both lower to noise_eqs which is just internal API.

[isaacsas]

Presumably a system with Brownians can still be used compositionally though. Can a system with noise_eqs?

[ChrisRackauckas]

@AayushSabharwal can you give a first pass?

Presumably a system with Brownians can still be used compositionally though. Can a system with noise_eqs?

To some extent. But that's my point: why would Catalyst do both? If you just always do the Brownians you'll end up at the same code as doing the noise_eqs manually, but it will be more robust in getting there.

[AayushSabharwal]

Along those lines we need to check how MTK handles mixing noise_eqs and Brownians, or if they are mutually exclusive options.

They are mutually exclusive. noise_eqs does not support composition. It is intended for directly specifying an SDE system in a form that can be codegen-ed directly. ModelingToolkit exports the noise_to_brownians function which allows converting noise_eqs to brownians.

[isaacsas]

Thanks for clarifying. Sounds like we should probably just target Brownians as you both suggest. However, in that regard for us the original Brownian order has a clear meaning in that it will generally correspond to the reaction order (giving a simple way to map between a reaction and its associated Brownian). My only hesitation about using Brownians is that I assume their order is not guaranteed to be preserved? I don't usually convert Catalyst models to SDEs, and perhaps it doesn't matter much, but @TorkelE are there places where we need to know the order of the noise terms / how they relate to the underlying reactions?

[AayushSabharwal]

Brownian order in the noise equations isn't particularly guaranteed. If you need a symbolic interface to noise i.e. "what is the noise associated with the ith equation and symbolic brownian b" I can make that work

[isaacsas]

Sounds good! Let us figure out what we actually need/use before you invest time in that though.

[TorkelE]

I think one of the main worries with the SDEs with beownian is that it requires structural simplification, which implies a large delay for certain larger models (however, not sure if people actually use SDEs for large models, personally I avoid it, so is likely no worries)

I'll try play around with the Brownian interface for Catalyst some time. It is nice because it would allow us to inject SDE terms in normal variable as well. I have some ideas for this.

[TorkelE]

(didn't mean to close the issue, was typing in my phone on a bus that suddenly went into a pothole...)

[isaacsas]

We could support Brownian as the default, and then for systems where users don't provide any extra Brownians provide a kwarg for us to directly generate the noise_eqs.

[ctessum]

Hi all, thanks for all your work on this.

We have some code that depends on catalyst, but also requires some bug fixes that are only in MTK v10. I was wondering if it would be possible to create a branch that at least has MTK v10 in the Project.toml? That way we can at least keep moving with forward with something that will pass the project compatibility checks, even if it doesn't necessarily compile or work correctly.

I think @AayushSabharwal has said that he may be willing to help with this.

Thanks for your consideration!

[isaacsas]

I made a branch named "v10_ctessum" and updated the Project.toml, but please keep in mind that in no way would I expect this branch to actually work with MTK10.

MTK9, and the updates to indexing and initialization in MTK ~9.60 and on, each took us around 6 months between releases to get Catalyst updated. I can't speak for @TorkelE, but I don't really have time to undertake a major Catalyst update this summer for V10, especially since we just spent an enormous amount of time earlier this year updating for MTK 9.60+. Given the past timescales to get updated, we will hopefully get V10 compatibility by the end of the Fall.