bug: parser.js can make an endless loop

[cuixiping]

Describe your issue/suggestions

The browser was freezed when I played the online IDE with some code looks right.
I tracked the following code

https://github.com/LingDong-/wenyan-lang/blob/master/src/parser.js#L308-L322

308:    while (!(gettok(i, 0) == "ctrl" && gettok(i, 1) == "funbody")) {
309:      if (gettok(i, 0) == "num") {
            ....
322:      }
323:    }

There will be an endless loop once gettok(i, 0) == "num" is false.
In my case, the result of gettok(i, 0) here is "data".
I don't quite understand the logic here.
Does this place need a i++ ？

Checklist

Before you create this issue, Please make sure ALL the following items are checked

• If this is a feature request, it's NOT listed Here
• If this is a bug report, it's NOT listed Here

[antfu]

Hi @cuixiping. Can you provide the code that causes the problem? It can help us investigate. Thanks

[cuixiping]

I debugged with the online IDE.
I noticed that the white-sapce is U+00A0 character, which was handled incorrectly by wy2tokens.

Code:

	吾有一術。名之曰「術甲」。
	欲行是術。必先得一物曰「物乙」。
	是術曰。
	是謂「術甲」之術也。

It will be ok if pasted from notepad to IDE.
It will cause endless loop if pasted from vscode (with your 文言 Wenyan Lang installed) to IDE. (ed.innerText includes U+00A0 character)

    var code = compile("js", ed.innerText, {

Here is the tokens. You can see ["data", " ", some times, that's is the reason.

tokens: Array(19)
0: (3) ["data", "    ", 4]
1: (3) ["decl", "uninit", 5]
2: (3) ["num", 1, 7]
3: (3) ["type", "fun", 7]
4: (3) ["name", undefined, 11]
5: (3) ["iden", "術甲", 15]
6: (3) ["data", "    ", 22]
7: (3) ["ctrl", "funstart", 25]
8: (3) ["ctrl", "funarg", 29]
9: (3) ["num", 1, 31]
10: (3) ["type", "obj", 31]
11: (3) ["assgn", undefined, 32]
12: (3) ["iden", "物乙", 36]
13: (3) ["data", "    ", 43]
14: (3) ["ctrl", "funbody", 45]
15: (3) ["data", "    ", 52]
16: (3) ["ctrl", "bigend", 53]
17: (3) ["iden", "術甲", 57]
18: (3) ["ctrl", "funend", 60]

I think adding "\u00A0" to the white-space list can resolve this issue.
It's better to fix the logic to avoid endless loop parsing, since there may be other unexpected characters.

if (
  txt[i] == "。" ||
  txt[i] == "、" ||
  txt[i] == "\n" ||
  txt[i] == "\r" ||
  txt[i] == "\t" ||
  txt[i] == " " ||
  txt[i] == "\u00A0"    //add more white space characters
) {

[LingDong-]

Hi @cuixiping ,

• \u00A0 is the "no break space" separate from common \u0020 space. If we're going to support this, it seems natural to also support em-space \u2003 en-space \u2002 figure-space \u2007 hair-space \u200A ideographic space \u3000 zero-width space \u200B zero-width no-break space \uFEEF and so on. Another solution is that the IDE replaces them immediately to normal space as soon as you type these special ones.

• gettok() will call the provided error-handler when it reaches unexpected EOF. In node's case this involves process.exit() which avoids any eternal loop. In the browser it is slightly harder to interrupt in the middle of a function from another function, but I think we can make the error-handler return a code that tells the parser function to stop. Will investigate.

Thanks for bringing this up!