illegal encoding warning

[bob-carpenter]

Summary:

We should alert users when they use non-ASCII characters outside of comments. This can be handled in the preprocessor and flagged as such.

Related issues:

• stanc gives useless error message when model source has non-ASCII characters rstan#501

• feature request: unicode in source stanc3#1406

Obviously if we allow UTF-8 encoded Unicode, it's a different set of illegal byte sequences we have to flag (not all byte sequences form legal UTF-8).

Current Version:

v2.17.1

[VMatthijs]

When trying to compile

model {
  real £; 
}

stanc3 now says

Syntax error at file "test.stan", line 2, character 6, lexing error:
   -------------------------------------------------
     1:  model {
     2:    real £;
               ^
     3:  }
   -------------------------------------------------

Invalid input "\194"

[bob-carpenter]

Is there a way to render it as you did in the program? I I think "\194" is going to be very confusing for our users!

…

On Dec 13, 2018, at 7:53 AM, Matthijs Vákár ***@***.***> wrote: When trying to compile model { real £; } stanc3 now says Syntax error at file "test.stan", line 2, character 6, lexing error: ------------------------------------------------- 1: model { 2: real £; ^ 3: } ------------------------------------------------- Invalid input "\194" — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

[VMatthijs]

That's fair.

Is the lazy solution of just having

Syntax error at file "test.stan", line 2, column 6, lexing error:
   -------------------------------------------------
     1:  model {
     2:    real £;
               ^
     3:  }
   -------------------------------------------------

Invalid character found.

OK?

[bob-carpenter]

I liked the first version better. It's not clear why it's illegal input with this message. Can we say a "non-ASCII character" or something like that? Or do you get the same error if you say real 1; or real _abc; neither of which is legal in Stan.

…

On Dec 13, 2018, at 10:03 AM, Matthijs Vákár ***@***.***> wrote: That's fair. Is the lazy solution of just having Syntax error at file "test.stan", line 2, column 6, lexing error: ------------------------------------------------- 1: model { 2: real £; ^ 3: } ------------------------------------------------- Invalid input OK? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

[VMatthijs]

It's not only non-ASCII that should trigger lexer errors, right? For instance, shouldn't $ also do that? The difficulty is that ocamllex captures the problematic token and returns it to the error message, but ocamllex cannot deal with unicode properly. It might be possible to just capture the original string using the positions specified by ocamllex.

[bob-carpenter]

On Dec 13, 2018, at 10:31 AM, Matthijs Vákár ***@***.***> wrote: It's not only non-ASCII that should trigger lexer errors, right? For instance, shouldn't $ also do that?

That's why I asked---I didn't know how the error was being triggered. Are there situations where we know it's an identifier, so we could say something like expected variable name but found illegal identifier '$'

…

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

[VMatthijs]

We could for ASCII characters if that is preferable.

For non-ASCII, we could try replacing ocamllex with sedlex, which seems to accept unicode. Then, we could also print the £ in the error. I didn't use sedlex to start with, because I thought our conclusion was that we don't care about unicode, because C++ can't handle it. That's why I stuck with ocamllex, as it's tried and tested technology with lots of examples out there.

I can open an issue for it in stanc3.

For the time being, should I move things back to the original error?

[VMatthijs]

I looked into sedlex a bit more and I think I'm tempted to stay with ocamllex for now. The switch would be some work as their syntax and wiring is quite different and I have a preference to stick with the more standard technology until more people are using sedlex or we've spoken to people who have first hand experience with it.