Skip to content

add full unicode support #2449

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 7 commits into from
Closed

add full unicode support #2449

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
666ebe4
add full unicode support
andimarek Feb 8, 2020
2ca1165
apply prettier
andimarek Feb 8, 2020
386788f
fix test
andimarek Feb 9, 2020
b437de9
more tests
andimarek Feb 9, 2020
f41b3c7
add test
andimarek Feb 19, 2020
17201f9
Merge remote-tracking branch 'upstream/main' into full-unicode-support
leebyron Apr 23, 2021
01f055a
Add full unicode spec change support
leebyron Apr 23, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
207 changes: 200 additions & 7 deletions src/language/__tests__/lexer-test.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -262,6 +262,95 @@ describe('Lexer', () => {
end: 34,
value: 'unicode \u1234\u5678\u90AB\uCDEF',
});

expect(lexOne('"unicode \\u{1234}\\u{5678}\\u{90AB}\\u{CDEF}"')).to.contain(
{
kind: TokenKind.STRING,
start: 0,
end: 42,
value: 'unicode \u1234\u5678\u90AB\uCDEF',
},
);

expect(
lexOne('"string with unicode escape outside BMP \\u{1F600}"'),
).to.contain({
kind: TokenKind.STRING,
start: 0,
end: 50,
value: 'string with unicode escape outside BMP 😀',
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't trust Unicode symbols in tests, because you can't visually inspect that codes are matching.
Can we use JS escape sequences instead?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually, I wrote a simple ESLint rule to find non-ASCII characters in #3053
And discovered that my editor (NeoVim) use strange encoding for ⏱️ emoji:
image

});

expect(lexOne('"unicode \\u{10FFFF}"')).to.contain({
kind: TokenKind.STRING,
start: 0,
end: 20,
value: 'unicode \u{10FFFF}',
});

expect(
lexOne('"string with unicode code point outside BMP 😀"'),
).to.contain({
kind: TokenKind.STRING,
start: 0,
end: 47,
value: 'string with unicode code point outside BMP 😀',
});

expect(
lexOne(
'"string with unicode code point outside BMP escaped \\uD83D\\uDE00"',
),
).to.contain({
kind: TokenKind.STRING,
start: 0,
end: 65,
value: 'string with unicode code point outside BMP escaped 😀',
});

expect(
lexOne(
'"string with unicode code point outside BMP escaped \\uD800\\uDC00"',
),
).to.contain({
kind: TokenKind.STRING,
start: 0,
end: 65,
value: 'string with unicode code point outside BMP escaped \uD800\uDC00',
});

expect(
lexOne(
'"string with unicode code point outside BMP escaped \\uDBFF\\uDC00"',
),
).to.contain({
kind: TokenKind.STRING,
start: 0,
end: 65,
value: 'string with unicode code point outside BMP escaped \uDBFF\uDC00',
});

expect(
lexOne(
'"string with unicode code point outside BMP escaped \\uDBFF\\uDFFF"',
),
).to.contain({
kind: TokenKind.STRING,
start: 0,
end: 65,
value: 'string with unicode code point outside BMP escaped \uDBFF\uDFFF',
});

expect(
lexOne(
'"string with unicode code point outside BMP escaped \\uD800\\uDFFF"',
),
).to.contain({
kind: TokenKind.STRING,
start: 0,
end: 65,
value: 'string with unicode code point outside BMP escaped \uD800\uDFFF',
});
});

it('lex reports useful string errors', () => {
Expand Down Expand Up @@ -314,39 +403,136 @@ describe('Lexer', () => {
});

expectSyntaxError('"bad \\z esc"').to.deep.equal({
message: 'Syntax Error: Invalid character escape sequence: \\z.',
message: 'Syntax Error: Invalid character escape sequence: "\\z".',
locations: [{ line: 1, column: 7 }],
});

expectSyntaxError('"bad \\x esc"').to.deep.equal({
message: 'Syntax Error: Invalid character escape sequence: \\x.',
message: 'Syntax Error: Invalid character escape sequence: "\\x".',
locations: [{ line: 1, column: 7 }],
});

expectSyntaxError('"bad \\u1 esc"').to.deep.equal({
message: 'Syntax Error: Invalid character escape sequence: \\u1 es.',
message: 'Syntax Error: Invalid Unicode escape sequence: "\\u1 es".',
locations: [{ line: 1, column: 7 }],
});

expectSyntaxError('"bad \\u1"').to.deep.equal({
message: 'Syntax Error: Invalid Unicode escape sequence: "\\u1".',
locations: [{ line: 1, column: 7 }],
});

expectSyntaxError('"bad \\u0XX1 esc"').to.deep.equal({
message: 'Syntax Error: Invalid character escape sequence: \\u0XX1.',
message: 'Syntax Error: Invalid Unicode escape sequence: "\\u0XX1".',
locations: [{ line: 1, column: 7 }],
});

expectSyntaxError('"bad \\uXXXX esc"').to.deep.equal({
message: 'Syntax Error: Invalid character escape sequence: \\uXXXX.',
message: 'Syntax Error: Invalid Unicode escape sequence: "\\uXXXX".',
locations: [{ line: 1, column: 7 }],
});

expectSyntaxError('"bad \\uFXXX esc"').to.deep.equal({
message: 'Syntax Error: Invalid character escape sequence: \\uFXXX.',
message: 'Syntax Error: Invalid Unicode escape sequence: "\\uFXXX".',
locations: [{ line: 1, column: 7 }],
});

expectSyntaxError('"bad \\uXXXF esc"').to.deep.equal({
message: 'Syntax Error: Invalid character escape sequence: \\uXXXF.',
message: 'Syntax Error: Invalid Unicode escape sequence: "\\uXXXF".',
locations: [{ line: 1, column: 7 }],
});

expectSyntaxError('"bad \\u{} esc"').to.deep.equal({
message: 'Syntax Error: Invalid Unicode escape sequence: "\\u{}".',
locations: [{ line: 1, column: 7 }],
});

expectSyntaxError('"bad \\u{XXXF} esc"').to.deep.equal({
message: 'Syntax Error: Invalid Unicode escape sequence: "\\u{XXXF}".',
locations: [{ line: 1, column: 7 }],
});

expectSyntaxError('"bad \\u{XXXF esc"').to.deep.equal({
message: 'Syntax Error: Invalid Unicode escape sequence: "\\u{XXXF es".',
locations: [{ line: 1, column: 7 }],
});

expectSyntaxError('"bad \\u{X"').to.deep.equal({
message: 'Syntax Error: Invalid Unicode escape sequence: "\\u{X".',
locations: [{ line: 1, column: 7 }],
});

expectSyntaxError('"bad \\u{XXXF e}scape"').to.deep.equal({
message: 'Syntax Error: Invalid Unicode escape sequence: "\\u{XXXF e}".',
locations: [{ line: 1, column: 7 }],
});

expectSyntaxError('"bad \\u{110000} esc"').to.deep.equal({
message: 'Syntax Error: Undefined Unicode code-point: "\\u{110000}".',
locations: [{ line: 1, column: 7 }],
});

expectSyntaxError('"bad \uDEAD esc"').to.deep.equal({
message: 'Syntax Error: Invalid low surrogate within String: "\\uDEAD".',
locations: [{ line: 1, column: 6 }],
});

expectSyntaxError('"bad \\uDEAD esc"').to.deep.equal({
message: 'Syntax Error: Invalid low surrogate within String: "\\uDEAD".',
locations: [{ line: 1, column: 6 }],
});

expectSyntaxError('"bad \\u{DEAD} esc"').to.deep.equal({
message: 'Syntax Error: Invalid low surrogate within String: "\\uDEAD".',
locations: [{ line: 1, column: 6 }],
});

expectSyntaxError('"bad \uD83D esc"').to.deep.equal({
message:
'Syntax Error: Invalid high surrogate "\\uD83D" followed by a non-low surrogate " " in String.',
locations: [{ line: 1, column: 6 }],
});

expectSyntaxError('"bad \\uD83D esc"').to.deep.equal({
message:
'Syntax Error: Invalid high surrogate "\\uD83D" followed by a non-low surrogate " " in String.',
locations: [{ line: 1, column: 6 }],
});

expectSyntaxError('"bad \\u{D83D} esc"').to.deep.equal({
message:
'Syntax Error: Invalid high surrogate "\\uD83D" followed by a non-low surrogate " " in String.',
locations: [{ line: 1, column: 6 }],
});

expectSyntaxError('"bad \uD83D\uDBFF esc"').to.deep.equal({
message:
'Syntax Error: Invalid high surrogate "\\uD83D" followed by a non-low surrogate "\\uDBFF" in String.',
locations: [{ line: 1, column: 6 }],
});

expectSyntaxError('"bad \\uD83D\\uDBFF esc"').to.deep.equal({
message:
'Syntax Error: Invalid high surrogate "\\uD83D" followed by a non-low surrogate "\\uDBFF" in String.',
locations: [{ line: 1, column: 6 }],
});

expectSyntaxError('"bad \uD83D\\uDBFF esc"').to.deep.equal({
message:
'Syntax Error: Invalid high surrogate "\\uD83D" followed by a non-low surrogate "\\uDBFF" in String.',
locations: [{ line: 1, column: 6 }],
});

expectSyntaxError('"bad \\uD83D\uDBFF esc"').to.deep.equal({
message:
'Syntax Error: Invalid high surrogate "\\uD83D" followed by a non-low surrogate "\\uDBFF" in String.',
locations: [{ line: 1, column: 6 }],
});

expectSyntaxError('"bad \\uD83D\\escape"').to.deep.equal({
message: 'Syntax Error: Invalid character escape sequence: "\\e".',
locations: [{ line: 1, column: 13 }],
});
});

it('lexes block strings', () => {
Expand Down Expand Up @@ -406,6 +592,13 @@ describe('Lexer', () => {
value: 'unescaped \\n\\r\\b\\t\\f\\u1234',
});

expect(lexOne('"""unescaped unicode outside BMP 😀"""')).to.contain({
kind: TokenKind.BLOCK_STRING,
start: 0,
end: 38,
value: 'unescaped unicode outside BMP 😀',
});

expect(lexOne('"""slashes \\\\ \\/"""')).to.contain({
kind: TokenKind.BLOCK_STRING,
start: 0,
Expand Down
Loading