Regular expressions does not work

[yisonPylkita]

Hi @jasonwilliams

Thank you for starting this project!

I tried to run this JS code

var re2 = /abc/

and got

thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Expected([], Token { data: Punctuator(Div), pos: Position { column_number: 11, line_number: 1 } }, "script")', src/libcore/result.rs:997:5
note: Run with `RUST_BACKTRACE=1` environment variable to display a backtrace.

[jasonwilliams]

Thanks for trying this out @yisonPylkita !
Yeah I haven't done any work on regular expressions yet, so it doesn't know how to handle those tokens. I can break it down why this doesn't work.

Lexer
If you look here, the lexer doesn't create any literal regular expression tokens, only divide, single line comments and multi-line comments https://github.com/jasonwilliams/boa/blob/master/src/lib/syntax/lexer.rs#L361-L388

We can prove this by looking at the token output for your source code, we end up with an identifier between 2 divide tokens which is clearly wrong

[
    Token {
        data: Keyword(
            Var
        ),
        pos: Position {
            column_number: 1,
            line_number: 1
        }
    },
    Token {
        data: Identifier(
            "re2"
        ),
        pos: Position {
            column_number: 5,
            line_number: 1
        }
    },
    Token {
        data: Punctuator(
            Assign
        ),
        pos: Position {
            column_number: 9,
            line_number: 1
        }
    },
    Token {
        data: Punctuator(
            Div
        ),
        pos: Position {
            column_number: 11,
            line_number: 1
        }
    },
    Token {
        data: Identifier(
            "abc"
        ),
        pos: Position {
            column_number: 12,
            line_number: 1
        }
    },
    Token {
        data: Punctuator(
            Div
        ),
        pos: Position {
            column_number: 15,
            line_number: 1
        }
    },
    Token {
        data: Punctuator(
            Semicolon
        ),
        pos: Position {
            column_number: 16,
            line_number: 1
        }
    }
]

We would need to add some additional logic here to check for an expresion between the 2 / characters and mark that value as a regex token.
I actually think this is pretty dooable.

Parser
If you're interested, the error is being emitted from here:
https://github.com/jasonwilliams/boa/blob/master/src/lib/syntax/parser.rs#L524
There are not expressions currently that begin with a / so that case isn't covered and it just fails.
We would need to create a new Const::RegExp() expression here and then return it similar to this https://github.com/jasonwilliams/boa/blob/master/src/lib/syntax/parser.rs#L303

The error message could also be a bit more helpful too.

[yisonPylkita]

Thanks for a quick replay.

Woow, that is a quite detailed problem description. It will definitely help me with a PR. I will start working on this in coming week.

[jasonwilliams]

@yisonPylkita how you getting on?

[jasonwilliams]

@yisonPylkita work has happened on this here:
jasonwilliams#94

[jasonwilliams]

Milestone has now been created for regular expressions so this can be closed:
https://github.com/jasonwilliams/boa/milestone/3