Consider adopting a parsing library for query lines to replace hard-to-maintain regular expressions

[claremacrae]

Description

Problem

We have a number of bug reports around problems interpreting complex query lines. For example:

• malformed boolean query for valid regex search that includes () #1500
• Escaping path includes #2078
• Allow parenthesis to be used in search strings in boolean filters #1308

Idea

I have been thinking for some time that as we add more features, our regular expressions are getting more complex.

And it would be worth investigating whether re-using an existing parsing solution would be more sustainable, long-term.

Options

I asked in OMG Discord - #plugin-dev on 2023-05-29, and the replies were:

• Both From River in Discord - plugin-dev 2023-05-29 Mon
  • PEG.js – Parser Generator for JavaScript
  • GitHub - peggyjs/peggy: Peggy: Parser generator for JavaScript
• ab1908
  • To add to this, ledger-obsidian uses moo for parsing and there's some decent examples in there you could work off of.
  • It uses some sort of combination of moo and nearley from what I saw and there were a good number of tests to work with.
  • There’s also lezer, but I don’t know whether it’s suited to Tasks
    • https://lezer.codemirror.net/
    • It also takes some getting used to.
• johanfriis
  • https://nearley.js.org/ would be an option to study as well.

Required Steps

• Establish our requirements for parsers

• Pick a library

  • Review the options listed above (and any more found since)
  • See how much they increase release distribution size by
  • Show what the code would look like in Tasks

• Start migrating code, incrementally.

How to Test

Confirm that the existing Jest tests and smoke tests all work.

[esm7]

We do have a parsing library, boon-js. The regular expressions are about preparing the expressions to be processable by that library. And while it is surely preferable to not have them, while searching for viable solutions at the time I checked a few directions, and my conclusion was that all the parsers that I found are considerably harder to use and would require considerably more code (and formal language definitions, which are not much better than cryptic regex) to maintain.
That being said, this research is still very worth doing. It's definitely possible that better solutions are out there, either ones that I reviewed, did not find, or maybe even progressed in the year that has passed since the original implementation :)

[claremacrae]

We do have a parsing library, boon-js.

😄

(and formal language definitions, which are not much better than cryptic regex)

Yes, I'm very conscious of that too.

while searching for viable solutions at the time I checked a few directions

That's great to hear, @esm7...

If you happened to have any notes about the ones you investigated, would you mind dropping them in here when you have time...

[esm7]

If you happened to have any notes about the ones you investigated, would you mind dropping them in here when you have time...

Will try to dig up the relevant notes. I found the note where I found and decided upon boon-js, but I did not properly link it to the previous alternatives (one previous alternative I actually wrote a full implementation for), so it's much harder to find than it should have been 😞
(luckily my note-taking practices have vastly improved in the meantime)

[sbliven]

Even if it's not used for parsing, defining the grammar for tasks queries would be good documentation.

I suspect that creating a clean grammar will require changing the behavior of some edge cases. What level of breaking changes are acceptable for the project?

[claremacrae]

What level of breaking changes are acceptable for the project?

For accidental behaviour that was not documented, then changes in behaviour where the user is informed, those would be OK. (For example, a new error message that says “this used to do X, now that doesn’t work anymore - change your query to Y instead”)

For documented behaviour, I would want to discuss it on a case-by-case basis, but the default answer would be “none”.

[claremacrae]

When specific changes are discovered, it will definitely be worth discussing them.
For example, if a filter started matching more tasks, that would be more acceptable than one starting to match fewer tasks. Missing results being a lot more serious for users than extra ones, as a general principle of search software…/