Is UTF-8 supported inside placeholders ?

[sylvain-pajaud]

Hi,

I'm currently trying to use Handlebars.NET (2.0.4)
One of the need I have is to use Japanese "placeholders" (thus with UTF-8 and not ASCII)

Here is an example with Handlebar.js:

https://handlebarsjs.com/playground.html#format=1&currentExample=%7B%22template%22%3A%22Hello%20%7B%7B%E5%A7%93%7D%7D%20%7B%7B%E5%90%8D%7D%7D%2C%5Cn%22%2C%22partials%22%3A%5B%5D%2C%22input%22%3A%22%7B%5Cn%20%20%E5%90%8D%3A%20%5C%22%E5%A4%AA%E9%83%8E%5C%22%2C%5Cn%20%20%E5%A7%93%3A%20%5C%22%E5%B1%B1%E7%94%B0%5C%22%5Cn%7D%5Cn%22%2C%22output%22%3A%22Hello%20%E5%B1%B1%E7%94%B0%20%E5%A4%AA%E9%83%8E%2C%5Cn%22%2C%22preparationScript%22%3A%22%22%2C%22handlebarsVersion%22%3A%224.7.6%22%7D

So it seems it works well on the JS version

Unfortunately, when I try the same thing with the .NET version the parsing fail with an error like this:
": 'Reached unparseable token in expression: 姓}}"

Interestingly, if I name my placeholder starting with ASCII characters (ex: {{a姓}}, or {{_姓}}) it works well no matter how many UTF-8 characters there is inside the placeholder

So it really seems that the first character is somehow important

Is this a limitation of the .NET implementation ? Or maybe I need to change some settings ?

In the worst case I'm thinking, maybe I can do a pre-treatment to change placeholders like {{姓}} into something like {{_姓}}
But it's still a cumbersome, so if some solution can be found I would be happy

[oformaniuk]

Hello @Sigma87
This one definitely looks like a bug in Handlebars.Net parser as per Handlebars spec most of UTF-8 chars are fine.

[mhornbacher]

@Sigma87 do you have a basic implementation example that fails?

[sylvain-pajaud]

I'm not sure if I understand well your question but I'll try to answer

The example I provided just before fails:

Handlebars.Compile("Hello {{姓}} {{名}}");

The Compile method throws a HandlebarsDotNet.HandlebarsParserException:

Reached unparseable token in expression: 姓}} {{名}}

Occured at: 0:17

But this does not fails:

Handlebars.Compile("Hello {{a姓}} {{a名}}");

The execution continues normally
It's why I said that the first character seems to be important (If it is non-ascii it bugs)

By the way I saw the same issue with helpers:

Handlebars.Compile("{{#empty 姓}}Is empty{{/empty}}");

Also throws a HandlebarsDotNet.HandlebarsParserException:

Reached unparseable token in expression: }}Is empty{{/empty}}

Occured at: 0:30

[mhornbacher]

Perfect so Handlebars.Compile("Hello {{姓}} {{名}}"); fails right?

I'll try to find some time to clone this over the weekend and begin taking a look at this if no one else picks it up.

[oformaniuk]

@mhornbacher , I did some research some time ago and looks like the problem is in WordParser and the way it validates first char by using predefined list in ValidWordStartCharactersString. Instead of that it should validate if it's valid literal char plus $, ., @, [, ] (maybe some other chars as well).

[mhornbacher]

Thanks, looks promising

[sylvain-pajaud]

Hi,

Sorry, is there any update about this issue ?

[mhornbacher]

It sadly slipped off my plate. Perhaps I'll make some time tomorrow to take another crack at it. Thank you for bringing it back to my attention :)

[davidkhachatryan123]

I have the same issue. When I read file with utf-8 content and pass dictionary with utf-8 content it returns result with correct chars from file but unknown ones in placeholders.