Repository navigation
Unicode characters #4
Description
Activity
- added a commit that references this issue
on Jul 12, 2021 - added a commit that references this issue
on Jul 12, 2021 Hey @mame!
I hit this same thing with ripper when I was writing prettier. I ended up solving it by taking the source, splitting it up into multiples lines, and converting each into an object that responded to
#[]so that I could get the right indices.Here are some links to the source:
- https://github.com/prettier/plugin-ruby/blob/f6a4a6d4299a91692a8b1aa287103d1e7e887b18/src/ruby/parser.rb#L23-L54
- https://github.com/prettier/plugin-ruby/blob/f6a4a6d4299a91692a8b1aa287103d1e7e887b18/src/ruby/parser.rb#L104-L120
- https://github.com/prettier/plugin-ruby/blob/f6a4a6d4299a91692a8b1aa287103d1e7e887b18/src/ruby/parser.rb#L132-L139
I hope it's helpful!
Thanks for the information. I think it is about the issue (1) that I said. Yeah, it is solvable by converting the indices.
The tougher issue is (2). Unfortunately, some Unicode characters (mainly Chinese, Japanese, and Korean characters) are rendered as if they have two columns.
あis one Japanese letter that takes two columns in the terminal. To highlight the letter, we need to put two^s under the line. To implement this, error_highlight needs a table to tell what character takes two (or more) columns.Just FYI: To make matters worse, the column count may change depending on a font and a terminal. This issue is called East Asian Width:
Ambiguous width characters are all those characters that can occur as fullwidth characters in any of a number of East Asian legacy character encodings. They have a “resolved” width of either narrow or wide depending on the context of their use.
To be honest, I don't want to face this problem for now 😇
Currently, error_highlight does not handle Unicode characters well. There are two subissues.
(1) is relatively simple, but (2) is a bit tough. It requires a table telling how many columns each character has. It is known that Reline has such a table. But because error_highlight is a built-in gem that is loaded at Ruby process invocation, it is not good for error_highlight to depend on Reline (unless we make Reline a special built-in gem). We need to discuss how we make the table available to error_highlight.