unicode plane 2 codepoints not saved correctly by 'annotate'

[smemsh]

in 2.6.2, it seems that codepoints that require more than 4 hex digits do not get stored correctly when given to annotate, for example the rustlang crab glyph '🦀':

task 1 annotate -- $'\U0001f980'

when viewed in task 1 edit this appears in my text editor as two 2-byte characters 0xd83e and 0xdd80, which actually do look up to the crab glyph in UTF-16 "surrogates" table, but this will not be what gets displayed when viewing the task or exporting (it shows as '��')

Note that I can paste the crab glyph into an annotation using task 1 edit and save it, and taskwarrior stores and displays it correctly.

I stumbled on the below when playing with the 16-bit values that got stored, to figure out how these surrogate values are encoded:

 $ printf '\xd8\x3e\xdd\x80' | iconv -f UTF-16BE -t UTF-8
🦀

That is, in fact the correct glyph, so the two characters that get stored are real UTF-16 surrogate characters. But why doesn't it store them as UTF-8, and it does when pasted? Note, my locale is set to en_US.UTF-8.

I am not sure what is correct behavior, or if this is a bug, as my understanding of unicode is weak. However, I expect when I annotate like so with a plane 2 code point, that is what will get stored, not some UTF-16 surrogate values. I know taskwarrior can store them, because I can directly add them in task edit and they store and display correctly.

Also note, plane 1 annotates just fine using initially described method. It's only when we get the codepoint above 16 bits that this is happening.

[djmitche]

Thanks for the report! If you're able, can you try this experiment with the develop branch? It would be good to know if this is a bug in the command-line handling, or in the task storage (which has been replaced in the develop branch).

I don't know much about how Unicode is treated by terminals. I'd kind of expect it to interpret surrogate pairs!

[smemsh]

can you try this experiment with the develop branch?

Indeed, the bug is not present in develop. Comparing old and new outputs of the sequence:

$ task 1 annotate -- $'\U0001f980'
$ task 1 export |
   jq -j '.[].annotations.[].description' |
   od -An -tx1

develop: f0 9f a6 80
2.6.2: ef bf bd ef bf bd

WIthout the octal dump filter, develop correctly displays the crab glyph.

I don't know much about how Unicode is treated by terminals. I'd kind of expect it to interpret surrogate pairs!

Leaving aside the question of whether my terminal should support UTF-16 surrogate pairs, my locale is set to UTF-8.

I will leave it to you whether to close the bug. I'm not sure if there is an intention for another 2.x release ever.

[djmitche]

Ah, I didn't read carefully enough to see that you were seeing two UTF-16 encoded characters! A surrogate pair is valid in utf-8, but each character in the pair has to be encoded correctly in UTF-8.

We're not planning another 2.x release, so I'll close the bug. I'm glad to know it's fixed!