Go deeper into the workaround to support 'token()' #210
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
they will produce a node. (WIP - proper testing is needed)
tree-sitter-hack after simplify-grammar step.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
We initially hit an error with
token(prec.dynamic(1, 'as'))in the tree-sitter-hack grammar, and then we (original author and myself) didn't find another way of writing the grammar. The problem is thattoken(...)sometimes produces a node and sometimes doesn't, and the previous workaround consisting in creating a named rule fortoken(...)would make the precedence annotation ineffective.I had to figure out in which cases a token node is missing and handle these cases accordingly. This results in 3 cases, described and tested in the script
tests/inline-token/check-test-output.The file
Name_pattern.mlthat was renamedMissing_node.mlfor clarity on its purpose. This is where we have the logic for working around missing nodes due to patterns (regexps) or due to thetoken()construct.Conclusion: a
token(prec.dynamic(/[a-z]+/))will produce no node in the original CST and a node of type unit in the typed CST, with no way to recover the token unless it has its own rule in the original grammar. Other combinations are handled fine.