properly handle unicode characters

[rabbah]

The decoding is localized to reading records from the database (on the client side) which is normalizing to the HTTP default charset ISO-8859-1 instead of UTF-8.

As a sanity check, I confirmed that for a sample code:

function main() {
  var frenchMessage = "Le périphérique est connecté";
  var chineseMesage = "装置被连接";
  return ({"fr": frenchMessage, "zh-Hans": chineseMesage});
}

The record is stored correctly in the database. If I were to encode the string using base64 encoding and logging the result, the activation result and logs are correctly stored in the database:

function main() {
  var msg = 'eyJmciI6IkxlIHDDqXJpcGjDqXJpcXVlIGVzdCBjb25uZWN0w6kiLCJ6aC1IYW5zIjoi6KOF572u6KKr6L+e5o6lIn0=';
  var dec = new Buffer(msg, 'base64').toString('utf-8');
  console.log(dec);
  var res = JSON.parse(dec);
  return res;
}

results in this activation record:

"response": {
    "statusCode": 0,
    "result": {
      "fr": "Le périphérique est connecté",
      "zh-Hans": "装置被连接"
    }
  },
  "logs": [
    "2016-04-23T13:36:16.030199912Z stdout: {\"fr\":\"Le périphérique est connecté\",\"zh-Hans\":\"装置被连接\"}"
  ],

@psuter FYI this is true both using the Cloudant SDK as well as the Spray client.

[stevenatkin]

Can we just normalize to UTF-8 instead of ISO8859-1. Until this is fixed it will be difficult for anyone to use OpenWhisk in anything other than English.

[midheartty]

Using Unicode escape sequence in action code also could produce UTF-8 output:

var chineseMesage = "\u88c5\u7f6e\u88ab\u8fde\u63a5";

[rabbah]

@psuter can you summarize your findings for future investigators?

[midheartty]

I found a S/O discussion regarding the encoding declaration for HttpEntity in scala/spray

http://stackoverflow.com/questions/36456796/spray-test-case-doesnt-property-unmarshal-utf-8-encoded-json-using-httpentity

Hope this solution would help.

[psuter]

So this is slightly fuzzy in my head.. I believe I had tracked it down to being an issue only happening when the JVMs are running inside a Docker container. (The DB seems to store everything properly.) I had diff'ed the JVM system properties when running outside of a container and when running with the image we use for all Scala containers, and there was only one difference. I believe it was a property related to the byte order mark... not 100% sure. I also believe this was a fixed property, not one that the user can configure.

[psuter]

Also, my branch https://github.com/psuter/openwhisk/tree/utf-8-db had some tests and (unsuccessful) attempts.

[midheartty]

I confirmed the DB (CouchDB) stores action codes with non-English data correctly by reviewing the CouchDB web interface for my vagrant install.

http://172.17.0.1:5984/_utils/database.html?vagrant_vagrant-ubuntu-trusty-64_whisks
(Overview > vagrant_vagrant-ubuntu-trusty-64_whisks > guest/myaction)

This may mean wrong charset conversion happens when loading action codes or producing a response in the Invoker.

[steveatkin]

I suspect that OpenWhisk is using the wrong codepage when actions are being executed. My guess is that they are using ISO8859-1 (Latin 1) when they should be using UTF-8 (Unicode). This should be a simple configuration change and would enable them to avoid having to do a conversion as the actions and data are most likely in UTF-8 already.

[csantanapr]

Getting more reports from users being hit with this problem, like using a parameter with value Aurélien the é not being handle correctly.