[SPARK-9238][SQL] Remove two extra useless entries for bytesOfCodePointInUTF8

[zhichao-li]

Only a trial thing, not sure if I understand correctly or not but I guess only 2 entries in bytesOfCodePointInUTF8 for the case of 6 bytes codepoint(1111110x) is enough.
Details can be found from https://en.wikipedia.org/wiki/UTF-8 in "Description" section.

[zhichao-li]

cc @chenghao-intel @rxin

[chenghao-intel]

LGTM

[rxin]

cc @davies

[SparkQA]

Test build #38026 has finished for PR 7582 at commit 8bddd01.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[davies]

@zhichao-li Two entries are enough for correctness. 254 and 255 are invalid, using 6 for them are even confusing. Right now, we didn't handle invalid UTF8 well (right now, it may raise exception), we could improve that, should pay attention to reduce the overhead for correct one.

[zhichao-li]

@davies, currently if the first byte is 254 or 255, numBytesForFirstByte would return 6 which somehow hide the invalid case not raising exception. What's your opinion? 1)Just stay the same 2)Explicitly raise exception here? or 3)Delete these two entries which would raise outOfRange exception when facing invalid char?

[davies]

I think it's better to raise an exception than parse it in wrong way silently. If we want to have better behavior, then it should be done case by case for every function, it's not trivial to me. So I'd like to peek 3), not to have this two additional entries.

[zhichao-li]

yeah, that's what this pr target to, I guess it's ready to be merged?

[davies]

LGTM, merging this into master and 1.4!