filter: Rewrite using SQLite3

[victorlin]

Description of proposed changes

A work in progress - see commit messages

Related issue(s)

Related to #866

Testing

• Create a nextstrain/base image from this branch: nextstrain/docker-base@5d4fad9
• Verify that --query still works for existing workflows.
• Verify that --query-sqlite works with SQLite query syntax.
  • I'm going to run a ncov rebuild using this. Will post results in a thread.

Checklist

• Delete the temporary DB file (implemented)
• Add a --debug option to show database file size breakdown
• Address disk space usage concerns (ref)
• Add a message in CHANGES.md summarizing the changes in this PR that are end user focused. Keep headers and formatting consistent with the rest of the file.

[codecov]

Codecov Report

Attention: 80 lines in your changes are missing coverage. Please review.

Comparison is base (e2ca468) 67.67% compared to head (ee1a62b) 70.26%.
Report is 1 commits behind head on master.

❗ Current head ee1a62b differs from pull request most recent head 6b4b9f1. Consider uploading reports for the commit 6b4b9f1 to get more accurate results

Files	Patch %	Lines
augur/io/tabular_file.py	66.66%	10 Missing and 9 partials ⚠️
augur/filter/io.py	89.02%	16 Missing and 2 partials ⚠️
augur/filter/include_exclude_rules.py	90.74%	8 Missing and 7 partials ⚠️
augur/filter/debug.py	56.25%	6 Missing and 1 partial ⚠️
augur/filter/dates.py	94.28%	6 Missing ⚠️
augur/io/sqlite3.py	93.33%	6 Missing ⚠️
augur/filter/_run.py	86.95%	1 Missing and 2 partials ⚠️
augur/dates/__init__.py	77.77%	1 Missing and 1 partial ⚠️
augur/filter/report.py	96.77%	1 Missing and 1 partial ⚠️
augur/io/metadata.py	81.81%	0 Missing and 2 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #1242      +/-   ##
==========================================
+ Coverage   67.67%   70.26%   +2.59%     
==========================================
  Files          69       71       +2     
  Lines        7518     7440      -78     
  Branches     1844     1775      -69     
==========================================
+ Hits         5088     5228     +140     
+ Misses       2158     1895     -263     
- Partials      272      317      +45     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[victorlin]

One significant change in this rewrite is that the metadata is no longer processed in chunks. This has implementation benefits, but comes with the downside of substantially increased disk usage.

Previously, there was the issue of augur filter using too much memory, which prompted the current chunked implementation. Now, I'm worried about augur filter using too much disk space, especially for large datasets such as ncov/open/metadata.tsv.zst, which results in a 12GB database file based on a trial run just now. This should be addressed before merging.

Two ideas to address this:

1. Make improvements to minimize the amount of size of the database file. The easy win here is to read a subset of the metadata columns, since most usages of augur filter only use a few columns for computation, while the remainder is only used for final output. I started on this in bc2a030. The difficult part will be determining which columns are used by --query-pandas/--query-sqlite.
2. Warn users if they don't have enough disk space. This requires an estimation of how large the database file will be, which isn't trivial. I started on this in b411c58.

I think (1) is worthwhile, and might remove the need for (2) if disk usage is reduced significantly.

[tsibley]

Generally concur with you here. Some ideas for implementing the disk usage reduction mitigations of (1)…

For determining which columns are used by --query-sqlite, two possibilities come to mind:

1. Parse the SQL query into tokens/an AST and inspect that. Two examples of where I did this previously using the sqlparse Python package: checking usage of transactions and checking usage of SET search_path statements. While both split a SQL script into a list of statements, they take two tacks from there: the first walks the tokens in each statement, the second matches a regex against each statement. We'd probably want the first tack here.
2. Use SQLite's introspection capabilities to ask what the query will use/do (e.g. with EXPLAIN or other methods). I've done this manually before in other database systems (with structured EXPLAIN output), but not programmatically in SQLite. It may not be possible and/or feasible, but seems worth looking into esp. if (1) isn't workable for some reason.

For determining what's used by --query-pandas, I'd think to either:

1. Leverage Pandas' internals here (if reasonably possible), as I believe you've prototyped elsewhere…
2. Implement a minimal parser for the Pandas query grammar, e.g. using pyparsing. This doesn't have to support the whole grammar, just what's sufficient to extract column names. In a similar vein, I took this approach for Markdown modification in Nextstrain CLI.
3. Rig up a pandas.DataFrame with monkeypatching, subclassing, or a proxy object that monitors and inspects column access and then apply the query to it.

[victorlin]

@tsibley thanks for the suggestions! As you mentioned elsewhere, Python's ast library seems to work well for pandas queries. I've successfully implemented (1) loading a subset of columns in #1294. If that gets merged, I'll rework this PR around those changes.

[victorlin]

Updates:

• For --query-sqlite, I've started implementation using sqlparse in 3cf5670.
• For --query-pandas using ast, that was recently added in 0c5a137, released, broke some workflows, and now we are trying to leverage Pandas internals (again, but hopefully better).

[victorlin]

Rebased this PR onto #1294 since it builds on those changes.

I am also becoming less confident about supporting a SQLite-backed --query so I've gathered related changes into 3cf5670. The existing --query is still supported by loading metadata from the database into DataFrame chunks.