Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a lexer for untokenised BBC BASIC files #1280

Merged
merged 21 commits into from
Aug 2, 2019

Commits on Jul 26, 2019

  1. Add a lexer for untokenised BBC BASIC files

    Includes full support for ARM BBC BASIC. Some features of BBC BASIC for Windows
    and BBC BASIC for SDL2.0 are not included.
    bavison committed Jul 26, 2019
    Configuration menu
    Copy the full SHA
    23a6c68 View commit details
    Browse the repository at this point in the history

Commits on Jul 30, 2019

  1. Configuration menu
    Copy the full SHA
    f559c65 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    c3eefd0 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    b8b3aeb View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    96c5780 View commit details
    Browse the repository at this point in the history

Commits on Jul 31, 2019

  1. Configuration menu
    Copy the full SHA
    1629f21 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    8662327 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    bae4a5f View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    eb0390b View commit details
    Browse the repository at this point in the history
  5. [bbcbasic] imperative ERROR keyword needs to be captured at higher pr…

    …iority than built-in function ERR
    bavison committed Jul 31, 2019
    Configuration menu
    Copy the full SHA
    db27bd2 View commit details
    Browse the repository at this point in the history
  6. [bbcbasic] attempt to reduce indcidences of * operator matching CLI c…

    …ommand introducer
    
    This is done by introducing an extra state where CLI commands are not valid,
    and classifying the control flow keywords according to whether they
    transition in or out of this state. Not 100% safe yet, since detecting any
    expression in :root state should mean that * can only be an operator.
    bavison committed Jul 31, 2019
    Configuration menu
    Copy the full SHA
    918e1ec View commit details
    Browse the repository at this point in the history

Commits on Aug 1, 2019

  1. [bbcbasic] simplify expression states

    By using a negative lookahead for when DIM and POINT are statements, rather
    that a positive lookahead when DIM() and POINT() are used as built-in
    functions, we can merge state :builtin_function into :expression. As a bonus,
    since built-in functions are now tested at lower priority than control flow
    statements, `ERROR` no longer needs its own rule.
    bavison committed Aug 1, 2019
    Configuration menu
    Copy the full SHA
    46171d4 View commit details
    Browse the repository at this point in the history
  2. [bbcbasic] fix * operators being misidentified as inline commands

    This is achieved to inserting `goto :no_further_imperatives` every time we
    identify a component of an expression, or reach a state where we expect an
    expression next. Because these will not be balanced by the number of times we
    return to :root, we can no longer use a state stack. This means that instead
    of being able to do `mixin :expression` from state `:assembly2`, we need to
    write out all the same rules again, but without the `goto`s.
    bavison committed Aug 1, 2019
    Configuration menu
    Copy the full SHA
    a8861a1 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    b030693 View commit details
    Browse the repository at this point in the history
  4. [bbcbasic] stricter checking of control flow statements

    Some of those keywords previously listed are not valid as an imperative
    statement.
    bavison committed Aug 1, 2019
    Configuration menu
    Copy the full SHA
    40f57d3 View commit details
    Browse the repository at this point in the history
  5. [bbcbasic] further simplification

    I don't think it realle helps to distinguish control flow statements from
    other statements, especially since they are tokenised the same. Some keywords
    (like `TINT`) have ended up in control3 even though they have nothing to do
    with conrol flow, and others (like `OF`) can appear within either sort of
    statement (`COLOUR OF x` and `CASE x OF`).
    bavison committed Aug 1, 2019
    Configuration menu
    Copy the full SHA
    33d500f View commit details
    Browse the repository at this point in the history
  6. [bbcbasic] improvements to handling of PROC

    * `PROC` is permitted in an `ON` statement
    * An imperative keyword is permitted immediately after `DEFPROCthing` without a
      separating colon
    
    Added a few more examples to the visual spec
    bavison committed Aug 1, 2019
    Configuration menu
    Copy the full SHA
    50f3d0f View commit details
    Browse the repository at this point in the history
  7. [bbcbasic] treat FN as a built-in function

    It isn't really  a built-in function (since it needs a function name to qualify
    it, which is lexed as a variable), but it can appear within an expression.
    bavison committed Aug 1, 2019
    Configuration menu
    Copy the full SHA
    3d84b20 View commit details
    Browse the repository at this point in the history
  8. [bbcbasic] use a different approach for detecting CLI commands

    Using a pair of states just wan't working - there are just too many exceptions
    to the rules to handle cleanly. So instead, here we take the approach of only
    permitting `*` to be lexed as a command introducer if it is at the start of a
    line, or follows `:` or one of a restrictive set of other keywords.
    
    Unfortunately, it appears that the lookbehind regexp syntax `(?<=)` doesn't
    work here, so we have to lex the string including the lead up to the `*` as a
    multi-token rule. This is fine in all cases except when the first line of the
    file is a `*` command, but hopefully this will be rare enough that nobody
    cares.
    bavison committed Aug 1, 2019
    Configuration menu
    Copy the full SHA
    cce0cf4 View commit details
    Browse the repository at this point in the history
  9. Configuration menu
    Copy the full SHA
    62819fd View commit details
    Browse the repository at this point in the history
  10. Configuration menu
    Copy the full SHA
    d70c214 View commit details
    Browse the repository at this point in the history