Ability to sort (and paginate) by column

[simonw]

As requested in #185 (comment)

I've previously avoided this for performance reasons: sort-by-column on a column without an index is likely to perform badly for hundreds of thousands of rows.

That's not a good enough reason to avoid the feature entirely though. A few options:

• Allow sort-by-column by default, give users the option to disable it for specific tables/columns
• Disallow sort-by-column by default, give users option (probably in metadata.json) to enable it for specific tables/columns
• Automatically detect if a column either has an index on it OR a table has less than X rows in it

We already have the mechanism in place to cut off SQL queries that take more than X seconds, so if someone DOES try to sort by a column that's too expensive it won't actually hurt anything - but it would be nice to not show people a "sort" option which is guaranteed to throw a timeout error.

The vast majority of datasette usage that I've seen so far is on smaller datasets where the performance penalties of sort-by-column are extremely unlikely to show up.

---

Still left to do:

• UI that shows which sort order is currently being applied (in HTML and in JSON)
• UI for applying a sort order (with rel=nofollow to avoid Google crawling it)
• Sort column names should be escaped correctly in generated SQL
• Validation that the selected sort order is a valid column
• Throw error if user attempts to apply _sort AND _sort_desc at the same time
• Ability to disable sorting (or sort only for specific columns) in metadata.json
• Fix "201 rows where sorted by sortable_with_nulls " bug

[simonw]

Given how unlikely it is that this will pose a real problem I think I like option 1: enable sort-by-column by default for all tables, then allow power users to instead switch to explicit enabling of the functionality in their metadata.json if they know their data is too big.

[simonw]

I think this can work with a ?_sort=xxx parameter - and ?_sort=-xxx to sort in the opposite direction.

I'd like to support "sort by X descending, then by Y ascending if there are dupes for X" as well. Two ways that could work:

?_sort=-xxx,yyy

Or...

?_sort=-xxx&_sort=yyy

The second option is probably better in that it makes it easier for columns to have a comma in their name.

Is it possible for a SQLite column to start with a - character?

[simonw]

Might have to do something special to get sort-by-nulls-last: https://stackoverflow.com/questions/12503120/how-to-do-nulls-last-in-sqlite

order by ifnull(column_name, -999999)

Would need to figure out a smart way to get the default value - maybe by running a min() or max() against the column first?

[simonw]

This is a better pattern as you don't have to pick a minimum value:

ORDER BY CASE WHEN SOMECOL IS NULL THEN 1 ELSE 0 END, SOMECOL

[simonw]

I think there are actually four kinds of sort order we need to support;

• ascending
• descending
• ascending, nulls last
• descending, nulls last

It looks like [-blah] is a valid SQLite table name, so mark I descending with a hyphen prefix isn't good.

Instead, maybe this:

?_sort_asc=col1&_sort_desc_nulls_last=col2

[simonw]

I'd like to continue to support _next=token pagination even for custom sort orders.

To do that I should include rowid (or general primary key) as the tie breaker on all sorts so I can incorporate that it into the _next= token.

[simonw]

In terms of user interface: the obvious place to put this is as a drop down menu on the column headers.

This also means the UI can support combined sort orders. Assuming you are already sorted by county descending and you select the candidate column header, the options could be:

• sort all by candidate
• sort all by candidate, descending
• sort by county descending, then by candidate
• sort by county descending, then by candidate descending

[simonw]

I'm tempted to put these verbose sorting options inline in the page HTML but have them in the table footer so they don't clog up the top half of the page with uninteresting links - then use JavaScript to hoik them out into a dropdown menu attached to each column header.

[simonw]

There is one other interesting option for auto-enabling/disabling sort: the inspect command could include data about column index presence and whether or not a column has any null values in it.

This would allow us to dynamically include a "nulls last" option but only for columns that contain at least one null.

It's quite a lot of additional engineering for a very minor feature though, so I think I'll punt on that for the moment.

We may find that the _group_count feature can benefit from column value statistics later on though.

[simonw]

Alternative idea: by default enable all sorting in the UI.

If a table has more than 100,000 rows disable sorting UI except for columns that have an index.

Allow this to be overridden in metadata.json

[simonw]

I'm not entirely sure how to get _next= pagination working against sorted collections when a tie-breaker is needed.

Consider this data:

https://fivethirtyeight.datasettes.com/fivethirtyeight-2628db9?sql=select+rowid%2C+*+from+%5Bnfl-wide-receivers%2Fadvanced-historical%5D%0D%0Aorder+by+case+when+career_ranypa+is+null+then+1+else+0+end%2C+career_ranypa%2C+rowid+limit+11

If the page size was set to 9 rather than 11, the page divide would be between those two rows with the same value in the career_ranypa column. What would the ?_next= token look like such that the correct row would be returned?

[simonw]

The problem is that our _next= pagination currently works based on a > - but for this case a >= for the value is needed combined with a > on the tie-breaker (which would be the rowid column).

So I think this is the right SQL:

select rowid, * from [nfl-wide-receivers/advanced-historical]
where career_ranypa >= -6.331167749 and rowid > 2736
order by case when career_ranypa is null then 1 else 0 end, career_ranypa, rowid limit 11

https://fivethirtyeight.datasettes.com/fivethirtyeight-2628db9?sql=select+rowid%2C+*+from+%5Bnfl-wide-receivers%2Fadvanced-historical%5D%0D%0Awhere+career_ranypa+%3E%3D+-6.331167749+and+rowid+%3E+2736%0D%0Aorder+by+case+when+career_ranypa+is+null+then+1+else+0+end%2C+career_ranypa%2C+rowid+limit+11

But how do I encode a _next token that means ">= X and > Y"?

[simonw]

Maybe the answer here is that anything that's encoded in the next token is treated as >= with the exception of columns known to be primary keys, which are treated as >