Data becomes truncated when database uses win1251 charset and client uses UTF8

[The-o]

I work with a database that runs with NLS_LANG set to AMERICAN_AMERICA.CL8MSWIN1251 and have no write access to it. On my side NLS_LANG is set to AMERICAN_AMERICA.UTF8. When I execute a select statement, the result contains truncated data for cyrillic VARCHAR2 fields. It looks like the buffer for the incoming data uses its length in bytes in cp1251, not in utf8.

You can perform the following request to test this behaviour:

SELECT 'ААБББВВВ' AS "col1", RAWTOHEX('ААБББВВВ') AS "col2" FROM DUAL

I get this as a result:

[ { col1: 'ААББ', col2: 'C0C0C1C1C1C2C2C2' } ]

tcpdump shows that the server's response is correct cp1251 data with no truncation.

[cjbj]

A full, short testcase which includes data would be helpful.

Reviewing NLS handling is on our roadmap. Other similar issues have been reported here, like #62

[paulish]

I have fixed similar problem in another nodejs oracle driver: joeferner/node-oracle#238

[cjbj]

@paulish thanks. The problem is indeed likely due to buffer sizing but we need to analyze where so we fix all potential issues and don't unnecessarily inflate memory use. The general rule of thumb is to multiply by the client/server character expansion ratio, which could be less than the '4' you have chosen.

[cjbj]

@The-o try node-oracledb 1.0. We did increase the internal buffer sizes, as predicted. The space optimization I mentioned is on our ToDo list.

Also see https://github.com/oracle/node-oracledb/blob/master/doc/api.md#nls