Preemptively parsing t-string templates

[ianjosephwilson]

In this example I use a decorator to just preemptively run the t-strings through the html() function because it keeps that call close to the template function itself but it could also just be an explicit function that calls all the module functions like def run_templates() or something that could be run from somewhere else. This all seems like pretty weird hacks. Is there anything more pre-preemptive other than just waiting for the t-string to be executed during some run-time rendering of the application? Or could the AST be used somehow to just pull out ALL t-strings of functions starting with html_ and then pre-parse them to check for correctness and cache the parsed structure?

Thanks.

Example

from tdom import html


# @NOTE: Do we even need to save this off if we just want to trigger parsing?
tmpl_instances = {}


def tmpl(*args, **kwargs):
    def tmpl_wrapper(f):
        # Trigger the template so that the parsing happens.
        tmpl_instances[f.__name__] = str(html(f(*args, **kwargs)))
        return f
    return tmpl_wrapper


@tmpl(label='')
def html_selected_nav_item(label):
    return t"<li><b>{label}</b></li>"


@tmpl(label='', href='')
def html_nav_item(label, href):
    return t"<li><a href={href}>{label}</a></li>"


@tmpl(items=[])
def html_nav(items, selected_href=None):
    nav_items = [html_selected_nav_item(label) if href == selected_href else html_nav_item(label, href) for label, href in items]
    return t"<ul>{nav_items}</ul>"


@tmpl(title='', nav_ul='')
def html_header(title, nav_ul):
    return t'''
    <div class="heading-container"><h1>{title}</h1>
         <div class="nav">{nav_ul}</div>
     </div>
'''


def main():
    html_str = str(html(html_header("My Cool Site", html_nav([("Home", "/"), ("About", "/about"), ("News", "/news")], selected_href="/news"))))
    print (html_str)

    for k_v in tmpl_instances.items():
        print (k_v)



if __name__ == '__main__':
    main()

[davepeck]

Hi @ianjosephwilson -- thanks for playing with tdom!

The good news is that tdom.html() already does all the caching I think you're looking for.

The first time you call html() with a given t-string, it'll parse it and cache the results of the parse. Every time you call html() after that, it'll just swap in the substituted values — no parsing required, and quite fast in practice.

We don't currently have tools for static analysis of your t-string contents; right now, the only way to find out if you typed in the HTML correctly is to call html() at runtime and wait for an exception. Eventually I expect we'll have something like ruff/flake8/etc that identifies problems in your t-string content, but Python 3.14 and tdom are very new and that's a ways off. Writing unit tests is probably the best best for now.

The straightforward way to write your code would be:

from tdom import html, Node


def html_selected_nav_item(label: str) -> Node:
    return html(t"<li><b>{label}</b></li>")


def html_nav_item(label: str, href: str) -> Node:
    return html(t"<li><a href={href}>{label}</a></li>")


def html_nav(items: list[tuple[str, str]], selected_href: str | None = None) -> Node:
    nav_items = [
        html_selected_nav_item(label)
        if href == selected_href
        else html_nav_item(label, href)
        for label, href in items
    ]
    return html(t"<ul>{nav_items}</ul>")


def html_header(title: str, nav_ul: Node) -> Node:
    return html(t"""
    <div class="heading-container"><h1>{title}</h1>
         <div class="nav">{nav_ul}</div>
     </div>
    """)


def main():
    node = html_header(
        "My Cool Site",
        html_nav(
            [("Home", "/"), ("About", "/about"), ("News", "/news")],
            selected_href="/news",
        ),
    )
    print(str(node))


if __name__ == "__main__":
    main()

Hope this makes sense! If there's something else you're looking to do not covered here, please let me know.

[pauleveritt]

If you're looking for checking correctness, that's also something I'm really interested in: tooling.

Imagine this was an f-string. It wouldn't really mean much to look at the f-string on its own. You need to look at the scope it is in, to see what values it is using, the types of those values, etc. It's more in the realm of listing, type checking, and other static analysis tools.

The first good news: ruff already supports t-strings, so you already get some benefit if you are a ruff user. As more tools (e.g. type checkers) support t-strings, you'll get more.

For the tdom semantics, such as the component syntax, the spread operator, etc. -- I think these should also be thought of as tooling. Something you run occasionally, during development, like a linter or a type checker. Agree?

[ianjosephwilson]

Sorry my questions was a bit jumbled. I did see it was cached as you guys eluded to in the PEP or PEP discussion. I guess a better description of what I was trying to do was to prime the cache to get the HTML parsing out of the way upfront... for example before I would start serving requests on a server. I was feeding in dummy data because I understand the templates are context dependent and need something passed in.

After reading more code I realize now I'm not sure that would work for component tags. It is probably a terrible pre-optimization as well (I didn't even profile anything! I'm THAT GUY). It just feels unnatural to not "precompile the templates". Although I thought more about how I'm using mako templates and it works the same way, the templates are compiled and cached on demand but it seems that I might have 10x/100x or more template strings. I guess I'll just have to try that out and see how it works with real code.

In the future were you planning to make the parser (and caching, etc.) pluggable somehow? I like that using html() just works and I don't have to think about it but in the future I'd probably want something like TProcessor(html_parser=MyHTMLParser(), parser_cache_size=None, callable_cache_size=None, **more_configuration_kitchensink) that I just made at startup time and then I ran everything through my_template_processor.html(template: Template) -> Node when I needed the html strings out like in my web request handler/view.

Regarding correctness/linting, I do think I'll want something in the future. Again I mainly use mako now and it doesn't have any checks but if they were there it seems like "free money" to be able to lint the HTML. It almost seems like there needs to be more metadata/structure around the template itself for the tooling to be effective but I can imagine adding that too early could potentially push things in a wrong direction rather than letting the appropriate patterns emerge over time. I would probably use it like a linter but I can also imagine turning on a full blown html validator with the strictest settings during testing.

I'm going to try it out more this weekend. I still have to upgrade from 3.10.* to 3.14.*...

Thanks for making this and taking the time to respond.

[davepeck]

I'd probably want something like TProcessor(html_parser=MyHTMLParser(), parser_cache_size=None, callable_cache_size=None, **more_configuration_kitchensink) that I just made at startup time and then I ran everything through my_template_processor.html(template: Template) -> Node when I needed the html strings out like in my web request handler/view.

Ah, I see; that makes sense. Right now there's no real way to control the caching in tdom. I do think something like this is where we'll probably head, for another reason: middleware. We want people to be able to plug in code that modifies trees after they've been created . The exact details are TBD, but presumably the default tdom.html() has defaults for caching/middleware, and if you want a really custom html() you can use the more advanced Processor.html() or similar that... doesn't exist yet. :-) Good fodder for future thinking!

[pauleveritt]

Of course, you might not want to wait that long for static tools to catch up. You could write a tool that recursed your code base, using inspect for functions that returned a Node. You could then grab the return code (e.g. html(t'<div>Hello {name}</div>') and use the tdom machinery to see if it passes some of the tdom syntax.

I should also add:

• Thanks for the comment!
• I use testing for this and I'm writing a library for easing the testing. Interested?

[pauleveritt]

As you allude to, pre-warming leads to slower startup times. But that's a trade most folks would be willing to make.

I believe we'd also want an on-disk cache of the parsed templates, to match what people do with some other template languages. Which would then address the startup time.

Funny you should mention pluggability. I had a branch that provided a "context" object that you could pass to html(), which would then get passed down the tree. It would look for some well-known plug points. In my case, locating the function for a component (to allow replaceability and overrides in a site) as well as bringing over my Hopscotch injector.

[ianjosephwilson]

Funny you should mention pluggability. I had a branch that provided a "context" object that you could pass to html(), which would then get passed down the tree. It would look for some well-known plug points. In my case, locating the function for a component (to allow replaceability and overrides in a site) as well as bringing over my Hopscotch injector.

I didn't fully understand the context approach at first but after some reflection I see that the calls to html() are actually made recursively and that does seem like it would be an issue for pluggability. I'm going to continue that discussion in the issue or in another discussion though. I'm going to close this for now because my original topic was answered by davepeck.