Enable reload of modified models

[blecourt-private]

Suggestion for loading models from definitions instead of files which would be helpful in custom loaders.

Without reloading models after modifying them, custom loaders are more or less limited to using pure SQL in added or modified expressions.

This MR showcases a way to load models from their rendered definitions. The PR currently only adds support for sql models but I imagine support for python models and seed models can be added in a similar fashion.

I suppose support for python models, seed models plus a bunch of tests need to be added to make this PR proper but before proceeding with this I would appreciate some feedback. It wouldn't make sense to put more work into this if I'm completely on the wrong track.

[izeigerman]

"I've read the description and the PR a few times, but I'm still a bit unclear on what this is fixing exactly. Could you share steps to reproduce the issue—or even better, a test case that fails without this fix?

[blecourt-private]

Hi @izeigerman

The problem description is in #4250.
I would like to setup a custom loader similar to what is described in Customizing SQLMesh but with the possibility to invoke macros in the model mutations.

For this to work the models' python_env needs to be updated with additional dependencies, as you also confirmed in your comment on forementioned issue.

In the mean time I have made an attempted to figure out how to in fact load the models once more including the mutations, thereby updating the python_env. This PR serves to describe the approach.

The (slightly contrived, I admit) example where the macro call @print_locals() is added to the post statements of the sql models in the custom loader actually works when I put the _load_models_from_definitions() to use in the return statement in _load_models().

I thought it could make sense to add a method _load_models_from_definitions() in this PR to the SqlmeshLoader class that users can invoke in their custom loaders in case they want to call macro functions in their model mutations. It would definitely be helpful for the use case that my team are working on to have such a method and perhaps it could also be of good use to others.

So far I have only looked into how to do this for sql models, so python models and seeds are pending. And I might have missed a thing or two of course. So I was hoping you (or one of the other experts) could provide some feedback. If you think the outlined approach is sensible I'll be happy to have a go at adding pytests and so on.

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
1 out of 2 committers have signed the CLA.

✅ blecourt-private
❌ blecourt
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[blecourt-private]

@sungchun12 this is the PR I mentioned in our meeting