Fixed thread-safety in Earley + added the 'edit_terminals' option

• Fixed a bug in Earley where running it from different threads produced bad results

• Improved error reporting when using LALR

• Added 'edit_terminals' option, to allow programmatical manipulation of terminals, for example to support keywords in different languages.

Note: This release skips 0.7.6, due to simple oversight on my part. Hopefully that shouldn't be a problem.

Transform tokens + Minor bugfixes

Lark transformers can now visit tokens as well. Use like this:

class MyTransformer(Transformer):
    def TOKEN1(self, tok):
        return tok.upper()

    def rule_as_usual(self, children):
        return children

MyTransformer(visit_tokens=True).transform(tree)

Fixed a few regressions that I accidentally added to 0.7.4

Fixed non-determinism in Earley (!), and many other bugfixes

• Fixed long-standing non-determinism and prioritization bugs in Earley.

• Serialize tool now supports multiple start symbols

• iter_subtrees, find_data and find_pred methods are now included in standalone parser

• Bugfixes for the transformer interface, for the custom lexer, for grammar imports, and many more

Serialize (new tool), and a minor line-counting bugfix

• Added a new tool called Serialize, that stores Lark's internal state as JSON. That will allow for integration with other languages. I have already started such a project for Julia: https://github.com/erezsh/Lark_Julia (It's working, but still in early stages)

• Minor bugfix regarding line-counting and the \s regex

Multiple start rules, negative priority, and various bugfixes

New features:

• Lark now allows you to specify the start symbol when calling Lark.parse() (requires pre-declaration of all possible start states, see the start option)

• Negative priority now allows in rules and terminals (default value is still 1, may change in 0.8)

Also includes many minor bugfixes, optimizations, and improvements to documentation

Improvements and Bugfixes

• Lark can now serialize its parsers, resulting in simplified stand-alone code.

• Bugfix for v_args (Issue #350)

• Improvements and bugfixes for importing rules from grammar files

• Performance improvement for the reconstructor feature

Major Release: New Earley implementation with SPPF support

This new version includes a brand new Earley implementation, with support for a Shared Packed Parse Forest (or SPPF), providing much better performance for ambiguous grammars, both in terms of run time and memory consumption.

Users might notice a small degradation in Earley's run-time performance for deterministic grammars. Hopefully future releases will take care of that as well.

Big thanks to @night199uk for his great contribution.

Other features worth mentioning (though they exist in the previous release):

• Added support for importing rules between grammars. The import mechanism is namespace-aware.

• Added the maybe_placeholders option, which causes optionals of the form [expr] to return None when not matched, instead of just not appearing. (optionals of the form expr? maintain the previous behavior of not appearing unless matched)

• Plenty of bugfixes, better errors, and better docs

Bugfix release

• Fixes regarding VisitError, and Discard

• Fixes to the lexer callbacks feature

• Fix for indenter, allowing re-use of same instance

Experimental "maybe_placeholders" feature, and several minor improvements

(Points to the wrong commit, due to technical issues. Actually refers to commit 13ddc43)

This release includes:

• Better error reporting

• Several bugfixes

• A new experimental feature: "maybe_placeholders", which replaces missing "maybe"s with a None, instead of removing them.

For example:

>>> p = Lark("""!start: "a"? "b"? "c"? """, maybe_placeholders=True)
>>> p.parse('b')
Tree(start, [None, Token(B, 'b'), None])
>>> p.parse('ac')
Tree(start, [Token(A, 'a'), None, Token(C, 'c')])

Contextual Standalone, import rules, and many bugfixes

New features:

• Standalone parser now uses the contextual lexer

• Experimental support for importing rules in the grammar

Bugfixes:

• Fixed a hidden bug in the LALR algorithm (no, it probably doesn't affect you, see issue #250)
• Fixed v_args handling of staticmethod and classmethod (Issue #246, #249)
• Many fixes to the standalone parser
• Fixed propagate_positions (and added start_pos/end_pos attributes to Tree.Meta)
• Fixed automatic terminal naming
• EOF token now gets proper line/column number

Also improved error messages and documentation