merge() result incorrect (NAs) when by column contains non-ASCII characters

[madlogos]

When the by column contains non-ASCII characters (e.g, Chinese), the merge() function returns NA unexpectedly.

d1=data.table(a=c("你", "我", "a", "他"), b=1:4)
d2=data.table(a=c("我", "他", "a"), c=3:5)
merge(d1, d2, by="a", all.x=TRUE, sort=FALSE)

should return

   a b  c
1 你 1 NA
2 我 2  3
3  a 3  5
4 他 4  4

right?

but actually turns out to be:

    a b  c
1: 你 1 NA
2: 我 2  3
3:  a 3  5
4: 他 4 NA

Could you look into this problem?

> sessionInfo()
R version 3.3.3 (2017-03-06)
Platform: i386-w64-mingw32/i386 (32-bit)
Running under: Windows 7 (build 7601) Service Pack 1

locale:
[1] LC_COLLATE=English_United States.1252       LC_CTYPE=Chinese (Simplified)_People's Republic of China.936
[3] LC_MONETARY=English_United States.1252   LC_NUMERIC=C                                                
[5] LC_TIME=English_United States.1252                          

attached base packages:
[1] compiler  stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
 [1] data.table_1.10.4  

[madlogos]

The function works well under Linux.
If I use merge(as.data.frame(d1), as.data.frame(d2), by="a", all.x=TRUE, sort=FALSE), then I can also get the expected result.
I guess it is caused by encoding problems in Windows platform.

[arunsrinivasan]

Confirmed on Windows 10 under R-GUI, DT v1.10.4.

[jangorecki]

I would say to best way to go is to use ascii column names, and if non-ascii names are needed, use labels instead. That will be covered by #623.

[shrektan]

@jangorecki No, this issue has nothing to do with non-ascii colnames. It's about the non-ascii values and has been fixed already.

I just confirm that on Windows 10 with the dev version, the result is expected.

        a     b     c
   <char> <int> <int>
1:     你     1    NA
2:     我     2     3
3:      a     3     5
4:     他     4     4

We should close this issue.

[jangorecki]

Uh, OK sorry then. Thanks for checking.
We should try to close it by providing unit test. AFAIK it is possible to have non-ascii in tests, but has to be encoded properly, so requires to map chinesse letters to unicode chars like \u000.