Skip to content

Improve tokenization #1

Closed
Closed
@blenderskool

Description

Tokenization currently doesn't use look-forward to create tokens. It groups tokens based on their individual classification.
This causes - symbol to be treated as a Separator even though it could be Literal (as there is no special meaning with just - symbol in the syntax apart from its usage in -> symbol).

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions