tr: how should non-UTF-8 input be handled?

[andrewliebenow]

Introduction

The current tr implementation assumes that the data fed into standard input is valid UTF-8.

Most implementations of tr work on a byte by byte basis, which allows non-UTF-8 data to be handled.

BusyBox, GNU Core Utilities, and uutils' coreutils:

❯ printf '\xFF\n' | /usr/bin/tr '\377' 'A'
A

Multibyte UTF-8 data is obviously not handled correctly:

❯ printf '%s\n' 'ᛆᚠᛏᚢᛆᛘᚢᚦᛌᛏᚭᚿᛏᛆᚱᚢᚿᛆᛧᚦᛆᛧ' | /usr/bin/tr -d 'ᛆᚠ'
����������������

bsdutils (port of FreeBSD tools to Linux) handles multibyte UTF-8 input correctly:

❯ printf '%s\n' 'ᛆᚠᛏᚢᛆᛘᚢᚦᛌᛏᚭᚿᛏᛆᚱᚢᚿᛆᛧᚦᛆᛧ' | /usr/ucb/tr -d 'ᛆᚠ'
ᛏᚢᛘᚢᚦᛌᛏᚭᚿᛏᚱᚢᚿᛧᚦᛧ

But that means it can't handle non-Unicode input:

❯ printf '\xFF\n' | /usr/ucb/tr '\377' 'A'
tr: Invalid or incomplete multibyte or wide character

The posixutils implementation is currently similar to the bsdutils implementation, but the error message needs to be changed:

❯ printf '\xFF\n' | ./target/release/tr '\377' 'A'
thread 'main' panicked at text/./tr.rs:1200:25:
assertion failed: leftover_bytes == 0_usize
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

Options

In no particular order:

1. Keep the current implementation (assume UTF-8) input, but just fix the error message when tr is fed non-UTF-8 data
   • Pros: supports some use cases that the widely used implementations do not
   • Cons: users may expect to be able to use tr to perform arbitrary byte transformations (not thinking of their input as text)
2. Mirror most implementations and just process input on a byte by byte basis.
   • Pros: better performance, simpler code, behaves like the most widely used implementations
   • Cons: can't perform transformations of multibyte text
3. Support both byte by byte processing and UTF-8 input, either by starting in UTF-8 mode and then switching to byte by byte mode when non-UTF-8 data is detected, or by providing additional/non-standard arguments
   • Pros: supports basically every use case
   • Cons: code complexity, deviation from existing implementations

I don't think there's one clear right option here, so please chime in if you have any suggestions or thoughts.

[jgarzik]

There is a core directive, and a practical directive:

Core directive: Be aggressive in pushing UTF-8 support, as an upgrade to the old utilities. This is our fresh take.

Practical directive: Do legacy scripts in the BSD and Linux world require byte-for-byte? More practically, does our tr break lots of in-the-field existing scripts? If yes, we should support B4B mode because posixutils should be useful in the real world and not break lots of existing scripts.

That argues for either UTF8 only (number 1) or UTF8 + B4B (number 3).

[andrewliebenow]

Thanks, that helps. Did some quick Googling:

Here's someone being advised to use tr to remove non-ASCII characters from file with a mixture of ASCII and arbitrary/unknown bytes:
https://stackoverflow.com/questions/15133607/how-to-remove-non-ascii-chars-using-sed/15134539#15134539

Similar to above:
https://stackoverflow.com/questions/3337936/remove-non-ascii-characters-from-csv/47918586#47918586

Considering this is just one use case out of many that one could think of that would require binary/non-UTF-8 handling, and how long tr has been around, I would guess that only supporting UTF-8 would cause significant breakage.

Please let me know if you're okay with number 3, and if so, if you think auto-fallback should be used, or command-line arguments. My suggestion would probably be to do both (something like --force-binary, --force-utf-8), so the user has recourse in case something goes wrong with the UTF-8 to binary auto-fallback. This might sound like a lot, but I don't think it will make the code much more complicated.

[jgarzik]

The wc utility has a bytes mode and a chars mode...

[jgarzik]

Closed by #354