toUTF8Array will convert ° into °

[jssuttles]

When using the write function of DicomDict, writeString uses toUTF8Array. This function takes in the degree symbol and spits it out as 2 bytes. Is there something that I'm missing? I fixed this in my implementation by changing it to something like Buffer.from(str, 'binary') for Node.js.

[pieper]

Hi - Can you provide test code to reproduce the issue?

[jssuttles]

I've been building out reports with DICOM SR. So, in my specific case, the problem occurs in the ContentSequence of a DICOM SR. However, for example, let's say that you put a ° in the series description, as per the attached example DICOM file. (Github doesn't take DCM files, so it's a TXT.)

IM-0001-0001.txt

var fs = require('fs');
var dcmjs = require('.');
var buf = fs.readFileSync('./IM-0001-0001.txt');
var dicomDict = dcmjs.data.DicomMessage.readFile(buf.buffer);
var newBuf = dicomDict.write();
fs.writeFileSync('./IM-0001-0001-copy.txt', Buffer.from(newBuf));

If you open both files in https://rawgit.com/cornerstonejs/dicomParser/master/examples/dumpWithDataDictionary/index.html, you'll notice that SeriesDescription is said to contain binary data. If you open the console and paste in dataSet.string('x0008103e'), you'll see that the original has a ° and the copy has °.
That it is said to contain binary data seems to indicate to me that it's somehow not legally DICOM to include a non-ASCII character in a string. Does that sound right? Is there another way I'm supposed to handle degree symbols? (HOROS doesn't seem to parse it correctly in the ContentSequence tag but does seem to parse it in the series description.)

[pieper]

Hi @jssuttles - Sorry I haven't had time to look into this. It sounds like an encoding issue that hasn't gotten much attention in dcmjs. If you can suggest a solution that would be great.

[jssuttles]

The way that I'm currently handling it is to convert to latin1 instead of utf8.

[pieper]

Would it be suitable for a pull request?

[jssuttles]

As it is, it isn't suitable. I'm using Node Buffers which have the ability to convert to latin1 built in.
Buffer.from(value, 'binary') (https://nodejs.org/api/buffer.html#buffer_buffers_and_character_encodings)
The following shows something that does what looks like the equivalent for browsers:

(The top on the node REPL side shows the default behavior which uses utf8.)
Taken from:
https://gist.github.com/lihnux/2aa4a6f5a9170974f6aa
But again, I'm not sure if this is how DICOM is supposed to be encoded. Do you happen to have knowledge about how it's supposed to work?
If this is ok, then I can write it up in a pull request.

[pieper]

Hmm, no, I don't know enough to say for sure what the right answer is.

We could start an experimental pull request (title it starting with WIP) and include a test for this file. Then we could compare the behavior against dcmtk, pydicom, pixelmed, etc and see how things compare.

These files from David Clunie could also be used for testing: http://www.dclunie.com/images/charset/index.html

[florianlink]

This is actually a big issue, maybe one of the biggest that are not yet addressed in dcmjs...

What you have to do is:

a) When reading a DICOM file:

• look at SpecificCharacterSet (0008,0005) attribute and decode all VRs that support is with the given characterset, store the result as unicode string in JavaScript

• The alternative would be to only read unmodified bytes and keep as UInt8ArrayBuffer and provide access methods that do the actual decoding, but that would be bad for users that want to get the strings or want to modify tags

• Pitfalls: It is even supported to have multiple SpecificCharacterSet values in the same document and switch them inside of the VRs (Crazy stuff, many libraries don't support that, but e.g. DCMTK does).

• In MeVisLab (C++) we use DCMTK to convert all loaded DICOM tags from their SpecificCharacterSet to UTF-8 using DcmSpecificCharacterSet class from DCMTK

b) When writing a DICOM file:

• There are two options:

1. Try to keep the SpecificCharacterSet from the original file (this works if the DICOM tags have not been modified or if the modified tags can be encoded in the charset. If there are chars that can't be encoded (e.g. original was ISO-IR 6 with 94 possible chars and user changed a tag to contain umlauts), then the writer should switch SpecificCharacterSet to ISO_IR 192 (UTF-8) and encode all strings as UTF-8
2. Just don't try to keep the original SpecificCharacterSet and always write as ISO_IR 192 (UTF-8), which is modern and should be supported by any DICOM reader.

It will need investigation, what charset encoding/decoding libraries there are for JavaScript.

[florianlink]

https://encoding.spec.whatwg.org/

The TextEncoder and TextDecoder seem to support various encodings, probably most of the specific charsets from Dicom. Only drawback is missing support for older browsers (and TextEncoder is a problem on Safari). There are some polyfills, but they look a bit unmaintained (like 3 years last commit).

[pieper]

Thanks for the explanation and research @florianlink 👍

Your interpretation sounds good to me. For my purposes I think using UTF-8 whenever possible makes perfect sense. caniuse indicates that Text{En,De}coder should be good to use on most modern platforms.

[florianlink]

I will see if I can get resources to work on this...

…

On Tue 30. Jun 2020 at 21:27, Steve Pieper ***@***.***> wrote: Thanks for the explanation and research @florianlink <https://github.com/florianlink> 👍 Your interpretation sounds good to me. For my purposes I think using UTF-8 whenever possible makes perfect sense. caniuse <https://caniuse.com/#feat=textencoder> indicates that Text{En,De}coder should be good to use on most modern platforms. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#84 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABKHPHBLV3YDEJTX35YAAMDRZI4APANCNFSM4J4P3I2Q> .

[florianlink]

I think I will get some funding to do this, but am on holidays the next three weeks.
So I plan to tackle the SpecificCharacterSet handing in dcmjs starting the second week of August.
I will provide a pull request when I have something to share and will write unit tests using David Clunies example datasets.
Can we place those datasets into the dcmjs test data repository, or is that a licensing problem?

I also suggest to switch dcmjs to a modern unit-test framework, I would suggest using jest (https://jestjs.io/) which I used successfully in recent projects.