Support for trailing whitespaces & newlines #34
Conversation
complected
changed the title
jams.js
Outdated
| @@ -14,6 +14,8 @@ ANY ::= (SAFE | WS | SYN | #x5C) | |||
| SAFE ::= #x21 | [#x23-#x5A] | [#x5E-#x7A] | #x7C | #x7E | |||
| `) | |||
|
|
|||
| // Expect a well-formed JSON string, already decoded, not UTF-8/UTF-16 encoded bytes etc. | |||
There was a problem hiding this comment.
This comment is not clear, implies the function is accepting, well, a JSON string, probably different interpretation than you meant
There was a problem hiding this comment.
You're right. I meant jams expects a JAMS string, but the leave strings have to be JSON strings.
| case 'bare': | ||
| case 'quote': { | ||
| const quoted = String.raw`"${ast.text}"` | ||
| const json = JSON.parse(quoted) |
There was a problem hiding this comment.
Did we verify json strings escape unicode and not utf16?
There was a problem hiding this comment.
Did we verify json strings escape unicode...?
Yep.
Longer answer is, I verified that a file of \n needs to transform to String.raw"\n"` (note the quotes) for JSON.parse to parse, otherwise it's a parsing exception. Did I answer your question?
There was a problem hiding this comment.
and not utf16?
I think I am not understanding the question. Could you please break it down a bit?
JSON strings ----can transform to---> JS Objects ---> UTF-8 encoded files.
Could you please let me know if this and the comment in the diff test.js helps?
There was a problem hiding this comment.
I’m talking about escape sequences, how you specify them like \uff vs U+FF etc, json spec encodes utf16 I’m 99% sure and I want it to be unicode instead
Will merge this but investigate some more later
| if (ast.children.length !== 1) throw new Error(`Invalid string`) | ||
| const utf8_encoded = String.raw`"${ast.children[0].text}"` | ||
| const json = JSON.parse(utf8_encoded) | ||
| return (ast.text.includes('""') && ast.children.length === 0) ? "" : _jams(ast.children[0]) |
Trailing spaces/lines
""\nor"a "turn out they don't work the whole time. It doesn't work because we are not accepting trailing lines/spaces for standalone strings (bare or not).After changing via the grammar, need to change empty string check because
""\ndoesn't equal"".Encoding
Following up last convo. I read around, I think the link below help and the comments added will help others.
What I gathered: as long as the string passed into
jamis correctly decoded i.e. the string in the JS runtime is a well formed JSON string, our parsing code works. The encoding of the file is only relevant when converting between the raw bytes and JS in memory representation.