Most programs contain a lot of logic, and a little bit of literal textual data. Programming languages are designed to be good for this sort of programming. But some programming tasks involve only a little bit of logic, and a great deal of textual data. For these tasks, we'd like to have a tool better suited to these text-heavy problems. A template engines is such a tool. In this chapter, we build a simple template engine.
The most common example of one of these text-heavy tasks is in web applications. An important phase in any web application is generating HTML to be served to the browser. Very few HTML pages are completely static: they involve at least a small amount of dynamic data, such as the user's name. Usually, they contain a great deal of dynamic data: product listings, friends' news updates, and so on.
At the same time, every HTML page contains large swaths of static text. And these pages are large, containing tens of thousands of bytes of text. The web application developer has a problem to solve: how best to generate a large string containing a mix of static and dynamic data? To add to the problem, the static text is actually HTML markup that is authored by another member of the team, the front-end designer, who wants to be able to work with it in familiar ways.
For purposes of illustration, let's imagine we want to produce this toy HTML:
<p>Welcome, Charlie!</p>
<p>Products:</p>
<ul>
<li>Apple: $1.00</li>
<li>Fig: $1.50</li>
<li>Pomegranite: $3.25</li>
</ul>
Here, the user's name will be dynamic, as will the names and prices of the products. Even the number of products isn't fixed: at another moment, there could be more or fewer products to display.
One simple way to make this HTML would be to have string constants in our code, and join them together to produce the page. Dynamic data would be inserted with string substitution of some sort. Some of our dynamic data is repetitive, like our lists of products. This means we'll have chunks of HTML that repeat, so those would have to be handled separately and combined with the rest of the page.
Producing our toy page in this way might look like this:
# The main HTML for the whole page.
PAGE_HTML = """
<p>Welcome, {name}!</p>
<p>Products:</p>
<ul>
{products}
</ul>
"""
# The HTML for each product displayed.
PRODUCT_HTML = "<li>{prodname}: {price}</li>\n"
def make_page(username, products):
product_html = ""
for prodname, price in products:
product_html += PRODUCT_HTML.format(prodname=prodname, price=format_price(price))
html = PAGE_HTML.format(name=username, products=product_html)
return html
This works, but we have a mess on our hands. The HTML is in multiple string constants embedded in our application code. The logic of the page is hard to see because the static text is broken into separate pieces. The details of how data is formatted is lost in the Python code. In order to modify the HTML page, our front-end designer would need to be able to edit Python code to make HTML changes. Imagine what the code would look like if the page were ten (or one hundred!) times more complicated; it would quickly become unworkable.
The better way to produce HTML pages is with templates. The HTML page is authored as a template, meaning that the file is mostly static HTML, with dynamic pieces embedded in it with special notation. Our toy page above could look like this as a template:
<p>Welcome, {{user_name}}!</p>
<p>Products:</p>
<ul>
{% for product in product_list %}
<li>{{ product.name }}:
{{ product.price|format_price }}</li>
{% endfor %}
</ul>
Here the focus is on the HTML text, with logic embedded in the HTML. Contrast this document-centric approach with our logic-centric code above. Our earlier program was mostly Python code, with HTML embedded in the Python logic. Here our program is mostly static HTML markup.
The mostly-static style used in templates is the opposite of how most programming languages work. For example, with Python, most of the source file is executable code, and if you need literal static text, you embed it in a string literal:
def hello():
print("Hello, world!")
hello()
When Python reads this source file, it interprets text like def hello(): as
instructions to be executed. The double quote character in
print("Hello, world!") indicates that the following text is meant literally,
until the closing double quote. This is how most programming languages work:
mostly dynamic, with some static pieces embedded in the instructions. The
static pieces are indicated by the double-quote notation.
A template language flips this around: the template file is mostly static literal text, with special notation to indicate the executable dynamic parts.
<p>Welcome, {{user_name}}!</p>
Here the text is meant to appear literally in the resulting HTML page, until the
{{ notation indicates a switch into dynamic mode, where the user_name variable
will be substituted into the output.
String formatting functions such as Python's "foo = {foo}!".format(foo=17)
are examples of mini-languages used to create text from a string literal and the data
to be inserted. Templates extend this idea to include logic constructs like
conditionals and loops, but are really just a difference of degree.
These files are called templates because they are used to produce many different pages with similar structure but differing details.
To use HTML templates in our programs, we need a template engine: a function that takes a static template describing the structure and static content of the page, and a dynamic context that provides the dynamic data to plug into the template. The template engine combines the template and the context to produce a complete string of HTML. The job of a template engine is to interpret the template, replacing the dynamic pieces with real data.
By the way, there's often nothing particular about HTML in a template engine, it could be used to produce any textual result. For example, they are also used to produce plain-text email messages. But usually they are used for HTML, and occasionally have HTML-specific features, such as escaping, which makes it possible to insert values into the HTML without worrying about which characters are special in HTML.
Templates are also not always the best choice for producing large chunks of text, even if they do follow a common pattern. Very data-heavy outputs might just be easier to do the logic-centric way rather than a document-centric template way.
Template engines vary in the syntax they support. Our template syntax is based on Django, a popular web framework. Since we are implementing our engine in Python, some Python concepts will appear in our syntax. We've already seen some of this syntax in our toy example at the top of the chapter, but this is a quick summary of all of the syntax we'll implement.
Data from the context is inserted using double curly braces:
<p>Welcome, {{user_name}}!</p>
The data available to the template is provided in the context when the template is rendered. More on that later.
Template engines usually provide access to elements within data using a simplified and relaxed syntax. In Python, these expressions all have different effects:
dict["key"]
obj.attr
obj.method()
In our template syntax, all of these operations are expressed with a dot:
dict.key
obj.attr
obj.method
The dot will access object attributes or dictionary values, and if the resulting value is callable, it's automatically called. This is different than the Python code, where you need to use different syntax for those operations. This results in simpler template syntax:
<p>The price is: {{product.price}}, with a {{product.discount}}% discount.</p>
Dots can be used multiple times on a single value to navigate down an attribute or element chain.
You can use helper functions, called filters, to modify values. Filters are invoked with a pipe character:
<p>Short name: {{story.subject|slugify|lower}}</p>
Building interesting pages usually requires at least a small amount of logic, so conditionals are available:
{% if user.is_logged_in %}
<p>Welcome, {{ user.name }}!</p>
{% else %}
<p><a href="/login">Log in </a></p>
{% endif %}
Looping lets us include collections of data in our pages:
<p>Products:</p>
<ul>
{% for product in product_list %}
<li>{{ product.name }}: {{ product.price|format_price }}</li>
{% endfor %}
</ul>
As with other programming languages, conditionals and loops can be nested to build complex logical structures.
Lastly, so that we can document our templates, comments appear between brace-hashes:
{# This is the best template ever! #}
In broad strokes, the template engine will have two main phases:
- Parse the template
- Render the template to assemble the text result, which involves:
- Manage the dynamic context, the source of the data
- Execute the logic elements
- Implement dot access and filter execution
A key decision in the implementation is what will be passed from the parsing phase to the rendering phase. What does parsing produce that can be rendered? There are two main options here. We'll call them interpretation and compilation, using the terms loosely from other language implementations.
In an interpretation model, parsing produces a data structure representing the structure of the template. The rendering phase walks that data structure, assembling the result text based on the instructions it finds. For a real-world example, the Django template engine uses this approach.
In a compilation model, parsing produces some form of directly executable code. The rendering phase executes that code, producing the result. Jinja2 and Mako are two examples of template engines that use the compilation approach.
Our implementation of the engine uses compilation: we compile the template into Python code. When run, the Python code assembles the result.
The template engine described here was originally written as part of coverage.py, to produce HTML reports. In coverage.py, there are only a few templates, and they are used over and over to produce many files from the same template. Overall, the program ran faster if the templates were compiled to Python code, because even though the compilation process was a bit more complicated, it only had to run once, while the execution of the compiled code ran many times, and was faster than interpreting a data structure many times.
It's a bit more complicated to compile the template to Python, but it's not as bad as you might think. And besides, as any developer can tell you, it's more fun to write a program to write a program than it is to write a program!
Our template compiler is a small example of a general technique called code generation. Code generation underlies many powerful and flexible tools, including programming language compilers. Code generation can get complex, but is a useful technique to have in your toolbox.
Another application of templates might prefer the interpreted approach, if templates will be used only a few times each. Then the effort to compile to Python won't pay off in the long run, and a simpler interpretation process might perform better overall.
Before we get to the code of the template engine, let's look at the code it produces. The parsing phase will convert a template into a Python function. Here is our small template again:
<p>Welcome, {{user_name}}!</p>
<p>Products:</p>
<ul>
{% for product in product_list %}
<li>{{ product.name }}:
{{ product.price|format_price }}</li>
{% endfor %}
</ul>
Our engine will compile this template to Python code. The resulting Python code looks unusual, because we've chosen some shortcuts that produce slightly faster code. Here is the Python (slightly reformatted for readability):
def render_function(context, do_dots):
c_user_name = context['user_name']
c_product_list = context['product_list']
c_format_price = context['format_price']
result = []
append_result = result.append
extend_result = result.extend
to_str = str
extend_result([
'<p>Welcome, ',
to_str(c_user_name),
'!</p>\n<p>Products:</p>\n<ul>\n'
])
for c_product in c_product_list:
extend_result([
'\n <li>',
to_str(do_dots(c_product, 'name')),
':\n ',
to_str(c_format_price(do_dots(c_product, 'price'))),
'</li>\n'
])
append_result('\n</ul>\n')
return ''.join(result)
Each template is converted into a render_function function that takes a
dictionary of data called the context. The body of the function starts by
unpacking the data from the context into local names, because they are faster
for repeated use. All the context data goes into locals with a c_ prefix so
that we can use other local names without fear of collisions.
The result of the template will be a string. The fastest way to build a
string from parts is to create a list of strings, and join them together at the
end. result will be the list of strings. Because we're going to add strings
to this list, we capture its append and extend methods in the local names
result_append and result_extend. The last local we create is a to_str
shorthand for the str built-in.
These kinds of shortcuts are unusual, let's look at them more closely. In
Python, a method call on an object like result.append("hello") is executed in
two steps. First, the append attribute is fetched from the result object:
result.append. Then the value fetched is invoked as a function, passing it
the argument "hello". Although we're used to seeing those steps performed
together, they really are separate. If you save the result of the first step,
you can perform the second step on the saved value. So these two Python
snippets do the same thing:
# The way we're used to seeing it:
result.append("hello")
# But this works the same:
append_result = result.append
append_result("hello")
In the template engine code, we've split it out this way so that we only do the first step once, no matter how many times we do the second step. This saves us a small amount of time, because we avoid the time to look up the append attribute.
This is an example of a micro-optimization: an unusual coding technique that gains us tiny improvements in speed. They can be less readable, or more confusing, so they are only justified for code that is a proven performance bottleneck. Developers disagree on how much micro-optimization is justified, and some beginners overdo it. The optimizations here were added only after timing experiments showed that they improved the speed, even if only a little bit. Micro-optimizations can be instructive, as they make use of some exotic aspects of Python, but don't over-use them in your own code.
The shortcut for str is also a micro-optimization. Names in Python can be
local to a function, global to a module, or built-in to Python. Looking up a
local name is faster than looking up a global or a built-in. We're used to the
fact that str is a builtin that is always available, but Python still has to
look up the name str each time it is used. Putting it in a local saves us
another small slice of time because locals are faster than builtins.
Once those shortcuts are defined, we're ready for the Python lines created from
our particular template. Strings will be added to the result list using the
append_result or extend_result shorthands, depending on whether we have one
string to add or more than one. Literal text in the template becomes a simple
string literal.
Having both append and extend is extra complexity, but remember we're aiming for the fastest execution of the template, and using extend for one item means making a new list of one item so that we can pass it to extend.
Expressions in {{ ... }} are computed, converted to strings, and added to the
result. Dots in the expression are handled by the do_dots function passed
into our function, because the meaning of the dotted expressions depends on the
data in the context: it could be attribute access or item access, and it could
be a callable.
The logical structures {% if ... %} and {% for ... %} are converted into
Python conditionals and loops. The expression in the {% if/for ... %} tag
will become the expression in the if or for statement, and the contents up
until the {% end... %} tag will become the body of the statement.
Now that we understand what the engine will do, let's walk through the implementation.
The heart of the template engine is the Templite class. (Get it? It's a template, but it's lite!)
The Templite class has a small interface. You construct one with the
text of the template, then later you can use the render method on it to
render a particular context, the dictionary of data, through the template:
# Make a Templite object.
templite = Templite('''
<h1>Hello {{name|upper}}!</h1>
{% for topic in topics %}
<p>You are interested in {{topic}}.</p>
{% endif %}
''',
{'upper': str.upper},
)
# Later, use it to render some data.
text = templite.render({
'name': "Ned",
'topics': ['Python', 'Geometry', 'Juggling'],
})
We pass the text of the template when the object is created so that we can
do the compile step just once, and later call render many times to reuse the
compiled results.
The constructor also accepts a dictionary of values, an initial context. These
are stored in the Templite object, and will be available when the template is
later rendered. These are good for defining functions or constants we want to
be available everywhere, like our upper function in the previous example.
Before we discuss the implementation of Templite, we have a helper to define first: CodeBuilder.
The bulk of the work in our engine is parsing the template and producing the
necessary Python code. To help with producing the Python, we have the
CodeBuilder class, which handles the bookkeeping for us as we construct the
the Python code. It adds lines of code, manages indentation, and
finally gives us values from the compiled Python.
One CodeBuilder object is responsible for a complete chunk of Python code. As used by our template engine, the chunk of Python is always a single complete function definition. But the CodeBuilder class makes no assumption that it will only be one function. This keeps the CodeBuilder code more general, and less coupled to the rest of the template engine code.
As we'll see, we also use nested CodeBuilders to make it possible to put code at the beginning of the function even though we don't know what it will be until we are nearly done.
A CodeBuilder object keeps a list of strings that will together be the final Python code. The only other state it needs is the current indentation level:
class CodeBuilder(object):
"""Build source code conveniently."""
def __init__(self, indent=0):
self.code = []
self.indent_level = indent
CodeBuilder doesn't do much, it has:
- a method to add a new line of code, which automatically indents the text to the current indentation level, and supplies a newline:
def add_line(self, line):
"""Add a line of source to the code.
Indentation and newline will be added for you, don't provide them.
"""
self.code.extend([" " * self.indent_level, line, "\n"])
- two methods to increase or decrease the indentation level:
INDENT_STEP = 4 # PEP8 says so!
def indent(self):
"""Increase the current indent for following lines."""
self.indent_level += self.INDENT_STEP
def dedent(self):
"""Decrease the current indent for following lines."""
self.indent_level -= self.INDENT_STEP
- a method to add a section, which is a piece of code managed by another CodeBuilder object. This lets us keep a reference to a place in the code, and add text to it later. The self.code list is mostly a list of strings, but will also hold references to these sections:
def add_section(self):
"""Add a section, a sub-CodeBuilder."""
section = CodeBuilder(self.indent_level)
self.code.append(section)
return section
- a
__str__method for producing a single string with all the code. This simply joins together all the strings inself.code. Note that becauseself.codecan contain sections, this might call otherCodeBuilderobjects recursively:
def __str__(self):
return "".join(str(c) for c in self.code)
- a method to produce the final values by executing the code. This stringifies the object, executes it to get its definitions, and returns the resulting values:
def get_globals(self):
"""Execute the code, and return a dict of globals it defines."""
# A check that the caller really finished all the blocks they started.
assert self.indent_level == 0
# Get the Python source as a single string.
python_source = str(self)
# Execute the source, defining globals, and return them.
global_namespace = {}
exec(python_source, global_namespace)
return global_namespace
This last method uses some exotic features of Python. The exec function
executes a string containing Python code. The second argument to exec is
a dictionary that will collect up the globals defined by the code. So for
example, if we do this:
python_source = """\
SEVENTEEN = 17
def three():
return 3
"""
global_namespace = {}
exec(python_source, global_namespace)
then global_namespace['SEVENTEEN'] is 17, and global_namespace['three'] is
an actual function named three.
Although we only use CodeBuilder to produce one function, there's nothing here that limits it to that use. This makes the class simpler to implement, and easier to understand.
CodeBuilder lets us create a chunk of Python source code, and has no specific
knowledge about our template engine at all. We could use it in such a way that
three different functions were defined in the Python, and then get_globals
would return a dict of three values, the three functions. As it happens, our
template engine only needs to define one function. But it's better software
design to keep that implementation detail in the template engine code, and out
of our CodeBuilder class.
Even as we're actually using it, to define a single function, having get_globals
return the dictionary keeps the code more modular because it doesn't need to
know the name of the function we've defined. Whatever function name we define
in our Python source, we can retrieve that name from the dict returned by
get_globals.
Now we can get into the implementation of the Templite class itself, and see how CodeBuilder gets used.
Most of our code is in the Templite class. As we've discussed, it has two phases: compilation and rendering.
All of the work to compile the template into a Python function happens in the Templite constructor. First the contexts are saved away:
def __init__(self, text, *contexts):
"""Construct a Templite with the given `text`.
`contexts` are dictionaries of values to use for future renderings.
These are good for filters and global values.
"""
self.context = {}
for context in contexts:
self.context.update(context)
Notice we used *contexts as the parameter. The asterisk denotes that any
number of positional arguments will be packed into a tuple and passed in as
contexts. This is called argument unpacking, and means that the caller can
provide a number of different context dictionaries. Now any of these calls are
valid:
t = Templite(template_text)
t = Templite(template_text, context1)
t = Templite(template_text, context1, context2)
The context arguments (if any) are supplied to the constructor as a tuple of
contexts. We can then iterate over the contexts tuple, dealing with each of
them in turn. We simply create one combined dictionary called self.context
which has the contents of all of the supplied contexts. If duplicate names are
provided in the contexts, the last one wins.
To make our compiled function as fast as possible, we extract context variables into Python locals. We'll get those names by keeping a set of variable names we encounter, but we also need to track the names of variables defined in the template, the loop variables:
self.all_vars = set()
self.loop_vars = set()
Later we'll see how these get used to help contruct the prologue of our function.
Now we use the CodeBuilder class we wrote earlier to start to build our compiled function:
code = CodeBuilder()
code.add_line("def render_function(context, do_dots):")
code.indent()
vars_code = code.add_section()
code.add_line("result = []")
code.add_line("append_result = result.append")
code.add_line("extend_result = result.extend")
code.add_line("to_str = str")
Here we construct our CodeBuilder object, and start writing lines into it. Our
Python function will be called render_function, and will take two arguments:
context is the data dictionary it should use, and do_dots is a function
implementing dot attribute access.
The context here is the combination of the data context passed to the Templite constructor, and the data context passed to the render function. It's the complete set of data available to the template that we made in the Templite constructor.
Notice that CodeBuilder is very simple: it doesn't "know" about function definitions, just lines of code. This keeps CodeBuilder simple, both in its implementation, and in its use. We can read our generated code here without having to mentally interpolate too many specialized CodeBuilder methods.
We create a section called vars_code. Later we'll write the variable
extraction lines into that section. The vars_code object lets us save a
place in the function that can be filled in later when we have the information
we need.
Then four fixed lines are written, defining a result list, shortcuts for the
methods to append to or extend that list, and a shortcut for the str()
builtin. As we discussed earlier, this odd step squeezes just a little bit
more performance out of our rendering function.
The reason we have both the append and the extend shortcut is so we can
use the most effective method, depending on whether we have one line to add to
our result, or more than one.
Next we define an inner function to help us with buffering output strings:
buffered = []
def flush_output():
"""Force `buffered` to the code builder."""
if len(buffered) == 1:
code.add_line("append_result(%s)" % buffered[0])
elif len(buffered) > 1:
code.add_line("extend_result([%s])" % ", ".join(buffered))
del buffered[:]
As we create chunks of output that need to go into our compiled function, we need to turn them into function calls that append to our result. We'd like to combine repeated append calls into one extend call. This is another micro-optimization. To make this possible, we buffer the chunks.
The buffered list holds strings that are yet to be written to our function
source code. As our template compilation proceeds, we'll append strings to
buffered, and flush them to the function source when we reach control flow
points, like if statements, or the beginning or ends of loops.
The flush_output function is a closure, which is a fancy word for a function
that refers to variables outside of itself. Here flush_output refers to
buffered and code. This simplifies our calls to the function: we don't
have to tell flush_output what buffer to flush, or where to flush it to, it
knows all that implicitly.
If only one string has been buffered, then the append_result shortcut is used
to append it to the result. If more than one is buffered, then all of them are
used with the extend_result shortcut to add them to the result.
Then the buffered list is cleared so more strings can be buffered.
The rest of the compiling code will add lines to the function by appending them
to buffered, and eventually call flush_output to write them to the
CodeBuilder.
With this function in place, we can have a line of code in our compiler like this:
buffered.append("'hello'")
which will mean that our compiled Python function will have this line:
append_result('hello')
which will add the string hello to the rendered output of the template. The
multiple levels of abstraction here, it's important to keep them straight as we
read the code.
Back to our Templite class: as we parse control structures, we want to check
that they are properly nested. The ops_stack list is a stack of strings:
ops_stack = []
When we encounter an {% if .. %} tag (for example), we'll push 'if' onto
the stack. When we find an {% endif %} tag, we can pop the stack and report
an error if it wasn't 'if' at the top of the stack.
Now the real parsing begins. We split the template text into a number of tokens using a regular expression, or regex. Regexes can be daunting: they are a very compact notation for complex pattern matching. They are also very efficient, since the complexity of matching the pattern is implemented in C in the regular expression engine, rather than in your own Python code. Here's our regex:
tokens = re.split(r"(?s)({{.*?}}|{%.*?%}|{#.*?#})", text)
This looks complicated, let's break it down. The re.split function will
split a string using a regex. Our pattern is parenthesized,
so the matches will be used to split the string, and will also be returned as
pieces in the split list. Our pattern will match our tag syntaxes, but we've
parenthesized it so that the string will be split at the tags, and the tags
will also be returned.
The (?s) flag in the regex means that a dot should match even a newline. Then
we have our parenthesized group of three alternatives: {{.*?}} matches an
expression, {%.*?%} matches a tag, and {#.*?#} matches a comment. In all
of these, we use .*? to match any number of characters, but the shortest
sequence that matches.
The result of re.split is a list of strings. For example, this template text:
<p>Topics for {{name}}: {% for t in topics %}{{t}}, {% endfor %}</p>
would be split into these pieces:
[
'<p>Topics for ', # literal
'{{name}}', # expression
': ', # literal
'{% for t in topics %}', # tag
'', # literal (empty)
'{{t}}', # expression
', ', # literal
'{% endfor %}', # tag
'</p>' # literal
]
Once the text is split into tokens like this, we can loop over the tokens, and deal with each in turn. By splitting them according to their type, we can handle each type separately.
The compilation code is a loop over these tokens:
for token in tokens:
Each token is examined to see which of the four cases it is. Just looking at the first two characters is enough. The first case is a comment, which is easy to handle: just ignore it and move on to the next token:
if token.startswith('{#'):
# Comment: ignore it and move on.
continue
For the case of {{...}} expressions, we cut off the two braces at the front
and back, strip off the white space, and pass the entire expression to
_expr_code:
elif token.startswith('{{'):
# An expression to evaluate.
expr = self._expr_code(token[2:-2].strip())
buffered.append("to_str(%s)" % expr)
The _expr_code method will compile the template expression into a Python
expression. We'll see that function later. We use the to_str function to
force the expression's value to be a string, and add that to our result.
The third case is the big one: {% ... %} tags. These are control structures
that will become Python control structures. First we have to flush our
buffered output lines, then we extract a list of words from the tag:
elif token.startswith('{%'):
# Action tag: split into words and parse further.
flush_output()
words = token[2:-2].strip().split()
Now we have three sub-cases, based on the first word in the tag: if, for,
or end. The if case shows our simple error handling and code generation:
if words[0] == 'if':
# An if statement: evaluate the expression to determine if.
if len(words) != 2:
self._syntax_error("Don't understand if", token)
ops_stack.append('if')
code.add_line("if %s:" % self._expr_code(words[1]))
code.indent()
The if tag should have a single expression, so the words list should have
only two elements in it. If it doesn't, we use the _syntax_error helper
method to raise a syntax error exception. We push 'if' onto ops_stack so
that we can check the endif tag. The expression part of the if tag is compiled
to a Python expression with _expr_code, and is used as the conditional
expression in a Python if statement.
The second tag type is for, which will be compiled to a Python for statement:
elif words[0] == 'for':
# A loop: iterate over expression result.
if len(words) != 4 or words[2] != 'in':
self._syntax_error("Don't understand for", token)
ops_stack.append('for')
self._variable(words[1], self.loop_vars)
code.add_line(
"for c_%s in %s:" % (
words[1],
self._expr_code(words[3])
)
)
code.indent()
We do a check of the syntax and push 'for' onto the stack. The
_variable method checks the syntax of the variable, and adds it to the set
we provide. This is how we collect up the names of all the variables
during compilation. Later we'll need to write the prologue of our function,
where we'll unpack all the variable names we get from the context. To do that
correctly, we need to know the names of all the variables we encountered,
self.all_vars, and the names of all the variables defined by loops, self.loop_vars.
We add one line to our function source, a for statement. All of our template
variables are turned into Python variables by prepending c_ to them so that
we know they won't collide with other names we're using in our Python function.
We use _expr_code to compile the iteration expression from the template into
an iteration expression in Python.
The last kind of tag we handle is and end tag, either {% endif %} or
{% endfor %}. The effect on our compiled function source is the same: simply
unindent to end the if statement or for statement that was started earlier:
elif words[0].startswith('end'):
# Endsomething. Pop the ops stack.
if len(words) != 1:
self._syntax_error("Don't understand end", token)
end_what = words[0][3:]
if not ops_stack:
self._syntax_error("Too many ends", token)
start_what = ops_stack.pop()
if start_what != end_what:
self._syntax_error("Mismatched end tag", end_what)
code.dedent()
Notice here that the actual work needed for the end tag is one line: unindent the function source. The rest of this clause is all error checking to make sure that the template is properly formed. This isn't unusual in program translation code.
Speaking of error handling, if the tag isn't an if, a for, or an end, then
we don't know what it is, so raise a syntax error:
else:
self._syntax_error("Don't understand tag", words[0])
We're done with the three different special syntaxes ({{...}}, {#...#}, and
{%...%}), what's left is literal content. We'll add the literal string to
the buffered output, using the repr built-in function to produce a Python
string literal for the token:
else:
# Literal content. If it isn't empty, output it.
if token:
buffered.append(repr(token))
If we didn't use repr, then we'd end up with lines like this in our compiled
function:
append_result(abc) # Error! abc isn't defined
We need the value to be quoted like this:
append_result('abc')
The repr function supplies the quotes around the string for us, and also
provides backslashes where needed:
append_result('"Don\'t you like my hat?" he asked.')
Notice that we check first if the token is an empty string with if token:,
since there's no point adding an empty string to the output. Empty tokens
happen if two template tags are adjacent. Because our regex is splitting on
tag syntax, adjacent tags will have an empty string between them. The check
here is an easy way to avoid putting useless append_result("") statements
into our compiled function.
That completes the loop over all the tokens in the template. When the loop is
done, all of the template has been processed. We have one last check to make:
if ops_stack isn't empty, then we must be missing an end tag. Then we flush
the buffered output to the function source:
if ops_stack:
self._syntax_error("Unmatched action tag", ops_stack[-1])
flush_output()
We had created a section at the beginning of the function. Its role was to unpack template variables from the context into Python locals. Now that we've processed the entire template, we know the names of all the variables, so we can write the lines in this prologue.
We have to do a little work to know what names we need to define. If we look again at our sample template:
<p>Welcome, {{user_name}}!</p>
<p>Products:</p>
<ul>
{% for product in product_list %}
<li>{{ product.name }}:
{{ product.price|format_price }}</li>
{% endfor %}
</ul>
There are two variables used here, user_name and product. The all_vars
set will have both of those names, because both are used in {{...}}
expressions. But only user_name needs to be extracted from the context in
the prologue, because product is defined by the loop.
All the variables used in the template are in the set all_vars, and all the
variables defined in the template are in loop_vars. All of the names in
loop_vars have already been defined in the code because they are used in
loops. So we need to unpack any name in all_vars that isn't in loop_vars:
for var_name in self.all_vars - self.loop_vars:
vars_code.add_line("c_%s = context[%r]" % (var_name, var_name))
Each name becomes a line in the function's prologue unpacking the context variable into a suitably-named local variable.
We're almost done compiling the template into a Python function. Our function
has been appending strings to result, so the last line of the function is
simply to join them all together and return them:
code.add_line("return ''.join(result)")
code.dedent()
Now that we've completed writing the source for our compiled Python function,
we need to get the function itself from our CodeBuilder object. The
get_globals method executes the Python code we've been assembling. Remember
that our code is a function defintion (starting with def render_function(..):),
so executing the code will define render_function, but not execute the body
of render_function.
The result of get_globals is the dictionary of values defined in the code.
We grab the render_function value from it, and save it as an attribute in our
Templite object:
self._render_function = code.get_globals()['render_function']
Now self._render_function is a callable Python function. We'll use it later
during the rendering phase.
We haven't yet seen a significant piece of the compiling process: the
_expr_code method that compiles a template expression into a Python
expression. Our template expressions can be as simple as a single name:
{{user_name}}
or can be a complex sequence of attribute access and filters:
{{user.name.localized|upper|escape}}
Our _expr_code method will handle all of these possibilities. As with
expressions in any language, ours are built recursively: big expressions are
composed of smaller expressions. A full expression is pipe-separated, where
the first piece is dot-separated, and so on. So our function naturally takes a
recursive form:
def _expr_code(self, expr):
"""Generate a Python expression for `expr`."""
The first case to consider is that our expression has pipes in it. If it does,
then we split it into a list of pipe-pieces. The first pipe-piece is passed
recursively to _expr_code to turn it into a Python expression.
if "|" in expr:
pipes = expr.split("|")
code = self._expr_code(pipes[0])
for func in pipes[1:]:
self._variable(func, self.all_vars)
code = "c_%s(%s)" % (func, code)
Each of the remaining pipe pieces is the name of a function. The value is
passed through the function to produce the final value. Each function name is
a variable that gets added to all_vars so that we can extract it properly in
the prologue.
If there were no pipes, there might be dots. If so, split on the dots. The
first part is passed recursively to _expr_code to turn it into a Python
expression, then each dot name is handled in turn:
elif "." in expr:
dots = expr.split(".")
code = self._expr_code(dots[0])
args = ", ".join(repr(d) for d in dots[1:])
code = "do_dots(%s, %s)" % (code, args)
To understand how dots get compiled, remember that x.y in the template could
mean either x['y'] or x.y in Python, depending on which works, and if the
result is callable, it's called. This uncertainty means that we have to try
those possibilities at run time, not compile time. So we compile x.y.z into
a function call, do_dots(x, 'y', 'z'). The dot function will try the
various access methods and return the value that succeeded.
The do_dots function is passed into our compiled Python function at run time.
We'll see its implementation in just a bit.
The last clause in the _expr_code function handles the case that there was no
pipe or dot in the input expression. In that case, it's just a name. We
record it in all_vars, and access the variable using its prefixed Python
name:
else:
self._variable(expr, self.all_vars)
code = "c_%s" % expr
return code
During compilation, we used a few helper functions. The _syntax_error method
simply puts together a nice error message and raises the exception:
def _syntax_error(self, msg, thing):
"""Raise a syntax error using `msg`, and showing `thing`."""
raise TempliteSyntaxError("%s: %r" % (msg, thing))
The _variable method helped us with validating variable names and adding them
to the sets of names we collected during compilation. We use a
regex to check that the name is a valid Python identifier, then add the name to
the set:
def _variable(self, name, vars_set):
"""Track that `name` is used as a variable.
Adds the name to `vars_set`, a set of variable names.
Raises an syntax error if `name` is not a valid name.
"""
if not re.match(r"[_a-zA-Z][_a-zA-Z0-9]*$", name):
self._syntax_error("Not a valid name", name)
vars_set.add(name)
With that, the compilation code is done!
All that's left is to write the rendering code. Since we've compiled our template to a Python function, the rendering code doesn't have much to do. It has to get the data context ready, and then call the compiled Python code:
def render(self, context=None):
"""Render this template by applying it to `context`.
`context` is a dictionary of values to use in this rendering.
"""
# Make the complete context we'll use.
render_context = dict(self.context)
if context:
render_context.update(context)
return self._render_function(render_context, self._do_dots)
Remember that when we constructed the Templite object, we started with a data
context. Here we copy it, and merge in whatever data has been passed in for
this rendering. The copying is so that successive rendering calls won't see
each others' data, and the merging is so that we have a single dictionary to
use for data lookups. This is how we build one unified data context from the
contexts provided when the template was constructed, with the data provided now
at render time.
Notice that the data passed to render could overwrite data passed to the
Templite constructor. That tends not to happen, because the context passed to
the constructor has global-ish kinds of things like filter definitions and
constants, and the context passed to render has specific data for that one
rendering.
Then we simply call our compiled render_function. The first argument is the
complete data context, the second argument is the function that will implement
the dot semantics. We use the same implementation every time, our own
_do_dots method, which is the last piece of code to look at:
def _do_dots(self, value, *dots):
"""Evaluate dotted expressions at runtime."""
for dot in dots:
try:
value = getattr(value, dot)
except AttributeError:
value = value[dot]
if callable(value):
value = value()
return value
During compilation, a template expression like x.y.z gets turned into
do_dots(x, 'y', 'z'). This function loops over the dot-names, and for each
one tries it as an attribute, and if that fails, tries it as a key. This is
what gives our single template syntax the flexibility to act as either x.y or
x['y']. At each step, we also check if the new value is callable, and if it
is, we call it. Once we're done with all the dot-names, the value in hand is
the value we want.
Here we used Python argument unpacking again (*dots) so that _do_dots could
take any number of dot names. This gives us a flexible function that will work
for any dotted expression we encounter in the template.
Note that when calling self._render_function, we pass in a function to use
for evaluating dot expressions, but we always pass in the same one. We could
have made that code part of the compiled template, but it's the same eight
lines for every template, and those eight lines are part of the definition of
how templates work, not part of the details of a particular template. It feels
cleaner to implement it like this than to have that code be part of the
compiled template.
Provided with the template engine is a suite of tests that cover all of the behavior and edge cases. I'm actually a little bit over my 500-line limit: the template engine is 252 lines, and the tests are 275 lines. This is typical of well-tested code: you have more code in your tests than in your product.
Full-featured template engines provide much more than we've implemented here. To keep this code small, we're leaving out interesting ideas like:
- Template inheritance and inclusion
- Custom tags
- Automatic escaping
- Arguments to filters
- Complex logic like elif and for/else
- Loops with more than one loop variable
- Whitespace control
Even so, our simple template engine is useful. In fact, it is the template engine used in coverage.py to produce its HTML reports.
In 252 lines, we've got a simple yet capable template engine. Real template engines have many more features, but this code lays out the basic ideas of the process: compile the template to a Python function, then execute the function to produce the text result.