tr: implement support for non-UTF-8 input

[andrewliebenow]

No description provided.

[andrewliebenow]

Other implementations can either handle only UTF-8 (FreeBSD/bsdutils), or always operate on a byte by byte basis, meaning that they cannot handle Unicode characters that have a multi-byte UTF-8 representation.

This implementation is about 2.5x as fast as FreeBSD/bsdutils', a bit faster than the implementation in uutils' coreutils, but slower than BusyBox and GNU Core Utilities (which don't have to deal with looking multiple bytes ahead).

Looks to be about twice as fast as it was before I touched the tr code.

❯ coreutils printf 'AA ᚱᚱ \xFF\xFF \xFE\xFE 11 ᚢᚢ BB ᛇᛇ 22\n' | ./target/release/tr -d -s 'A ᚱ \377 \376' '1 ᚢ B ᛇ 2'
1ᚢBᛇ2

❯ coreutils printf 'AA ᚱᚱ \xFF\xFF \xFE\xFE 11 ᚢᚢ BB ᛇᛇ 22\n' | /usr/bin/tr -d -s 'A ᚱ \377 \376' '1 ᚢ B ᛇ 2'
1�B����2

❯ coreutils printf 'AA ᚱᚱ \xFF\xFF \xFE\xFE 11 ᚢᚢ BB ᛇᛇ 22\n' | coreutils tr -d -s 'A ᚱ \377 \376' '1 ᚢ B ᛇ 2'
1�B����2

❯ coreutils printf 'AA ᚱᚱ \xFF\xFF \xFE\xFE 11 ᚢᚢ BB ᛇᛇ 22\n' | busybox tr -d -s 'A ᚱ \377 \376' '1 ᚢ B ᛇ 2'
1�B����2

❯ coreutils printf 'AA ᚱᚱ \xFF\xFF \xFE\xFE 11 ᚢᚢ BB ᛇᛇ 22\n' | /usr/ucb/tr -d -s 'A ᚱ \377 \376' '1 ᚢ B ᛇ 2'
tr: Invalid or incomplete multibyte or wide character

The obvious downside is about 550 more lines of code (excluding new tests), and the added complexity of the multi-byte lookahead needed to support both binary and UTF-8 input.