Complete refactor of TableView and table.html template

[simonw]

Split from #878. The current TableView class is by far the most complex part of Datasette, and the most difficult to work on: https://github.com/simonw/datasette/blob/0.59.2/datasette/views/table.py

In #878 I started exploring a new pattern for building views. In doing so it became clear that TableView is the first beast that I need to slay - if I can refactor that into something neat the pattern for building other views will emerge as a natural consequence.

I've been trying to build this as a register_routes() plugin, as originally suggested in #870 - though unfortunately it looks like those plugins can't replace existing Datasette default views at the moment, see #1517. [UPDATE: I was wrong about this, plugins can over-ride default views just fine]

I also know that I want to have a fully documented template context for table.html as a major step on the way to Datasette 1.0, see #1510.

All of this adds up to the TableView factor being a major project that will unblock a whole flurry of other things - so I'm going to work on that in this separate issue.

[simonw]

Here's where I got to with my hacked-together initial plugin prototype - it managed to render the table page with some rows on it (and a bunch of missing functionality such as filters): https://gist.github.com/simonw/281eac9c73b062c3469607ad86470eb2

[simonw]

Ideally I'd like to execute the existing test suite against the new implementation - that would require me to solve this so I can replace the view with the plugin version though:

• Let register_routes() over-ride default routes within Datasette #1517

[simonw]

I was wrong about that, you CAN over-ride default routes already.

[simonw]

A (likely incomplete) list of features on the table page:

• Display table/database/instance metadata
• Show count of all results
• Display table of results
  • Special table display treatment for URLs, numbers
  • Allow plugins to modify table cells
  • Respect ?_col= and ?_nocol=
• Show interface for filtering by columns and operations
• Show search box, support executing FTS searches
• Sort table by specified column
• Paginate table
• Show facet results
• Show suggested facets
• Link to available exports
• Display schema for table
  • Maybe it should show the SQL for the query too?
• Handle various non-obvious querystring options, like ?_where= and ?_through=

[simonw]

My goal is to break up a lot of this functionality into separate methods. These methods can be executed in parallel by asyncinject, but more importantly they can be used to build a much better JSON representation, where the default representation is lighter and ?_extra=x options can be used to execute more expensive portions and add them to the response.

So the HTML version itself needs to be re-written to use those JSON extras.

[simonw]

Right now the HTML version gets to cheat - it passes through objects that are not JSON serializable, including custom functions that can then be called by Jinja.

I'm interested in maybe removing this cheating - if the HTML version could only request JSON-serializable extras those could be exposed in the API as well.

It would also help cleanup the kind-of-nasty pattern I use in the current BaseView where everything returns both a bunch of JSON-serializable data AND an awaitable function that then gets to add extra things to the HTML context.

[simonw]

... and while I'm doing all of this I can rewrite the templates to not use those cheating magical functions AND document the template context at the same time, refs:

• Datasette 1.0 documented template context (maybe via API docs) #1510.

[simonw]

Very confused by this piece of code here:

datasette/datasette/views/table.py

Lines 37 to 63 in 1c13e1a

	class Row:
	def __init__(self, cells):
	self.cells = cells
	def __iter__(self):
	return iter(self.cells)
	def __getitem__(self, key):
	for cell in self.cells:
	if cell["column"] == key:
	return cell["raw"]
	raise KeyError
	def display(self, key):
	for cell in self.cells:
	if cell["column"] == key:
	return cell["value"]
	return None
	def __str__(self):
	d = {
	key: self[key]
	for key in [
	c["column"] for c in self.cells if not c.get("is_special_link_column")
	]
	}
	return json.dumps(d, default=repr, indent=2)

I added it in 754836e - in the new world that should probably be replaced by pure JSON.

Aha - this comment explains it: #521 (comment)

I think the trick is to redefine what a "cell_row" is. Each row is currently a list of cells:

datasette/datasette/views/table.py

Lines 159 to 163 in 6341f8c

	if truncate_cells and len(display_value) > truncate_cells:
	display_value = display_value[:truncate_cells] + u"\u2026"
	cells.append({"column": column, "value": display_value})
	cell_rows.append(cells)

I can redefine the row (the cells variable in the above example) as a thing-that-iterates-cells (hence behaving like a list) but that also supports __getitem__ access for looking up cell values if you know the name of the column.

The goal was to support neater custom templates like this:

{% for row in display_rows %}
  <h2 class="scientist">{{ row["First_Name"] }} {{ row["Last_Name"] }}</h2>
  ...

This may be an argument for continuing to allow non-JSON-objects through to the HTML templates. Need to think about that a bit more.

[simonw]

I'm going to try leaning into the asyncinject mechanism a bit here. One method can execute and return the raw rows. Another can turn that into the default minimal JSON representation. Then a third can take that (or take both) and use it to inflate out the JSON that the HTML template needs, with those extras and with the rendered cells from plugins.

[simonw]

This may be an argument for continuing to allow non-JSON-objects through to the HTML templates. Need to think about that a bit more.

I can definitely support this using pure-JSON - I could make two versions of the row available, one that's an array of cell objects and the other that's an object mapping column names to column raw values.

[simonw]

Two new requirements inspired by work on the datasette-table (and datasette-notebook) projects:

• Add Link: rel="alternate" header pointing to JSON for a table/query #1533
• Maybe return JSON from HTML pages if Accept: application/json is sent #1534

[simonw]

I'm also going to use the new datasette-table Web Component to help guide the design of the new API, which relates directly to this issue too:

• Use datasette-table Web Component to guide the design of the JSON API for 1.0 #1532