utf8n_to_uvchr_msgs: Use different DFA tables

I'm uncertain about this commit.

There are three separate DFA tables already in core.  One accepts Perl
extended UTF-8; one accepts only strict Unicode UTF-8; and the third
accepts modified Unicode UTF-8 spelled out by them in Corrigendum Perl#9.

Both the Unicode varieties reject surrogate code points and anything
above U+10FFFF.  C9 accepts, but the other rejects non-character code
points.

Without this commit, the way it works is it uses the most restrictive
table for the DFA.  Anything it accepts is always valid.  Anything it
rejects is potentially problematic, and it calls a non-inlined function
to examine the input more slowly to determine if it is acceptable and/or
if a warning needs to be raised.

This commit examines the input flags to determine which DFA to use
in this situation.  The benefit is that the slower routine could be
avoided for many more code points.

But the vast vast majority of calls to this function aren't for any
problematic code points, so the extra cost of this will very rarely be
recouped. The translation from UTF-8 is critically important.  We want
it to be as fast as possible.  I would not even consider this commit if
the extra cost weren't very small.

A complicating factor is that 2048 (approximately 20% of the total)
Korean Hangul syllable code points are not handled by the strict table,
so must be by the slower function; though they're handled at the very
beginning of it.  These code points are never problematic, so it is
unfortunate that they have to be handled via the slower function.  But
still, rarely will this function be called with them.  Only the strict
table has this problem

The way this commit works is to have a table containing pointers to the
three DFA tables.  The function looks at the input flags; if none are
present, it uses the loosest dfa; if any restrictions are present, it
adds 1 to the index to use, and it the C9 resetrictions are present, it
adds an extra 1.  The flags are cast to bools to get each addition.  If
the bool casts didn't generate conditionals, the only cost to this would
be two additions and an indirection; and I would say that that cost is
so tiny that this would be worth it.  But I looked at godbolt, and
casting to bool requires a comparison on both modern clang and gcc.
That makes me unsure of the tradeoff.

Another option would be to just juse two DFAs, loose and most strict.
Then there would be a single conditional, and the Hanguls still would be
handled by the DFA when there were no flags restricting things

Author: khwilliamson

globvar.sym

@@ -9,6 +9,7 @@ PL_bincompat_options
 PL_bitcount
 PL_block_type
 PL_c9_utf8_dfa_tab
+PL_which_utf8_dfa_tab
 PL_charclass
 PL_check
 PL_core_reg_engine

inline.h

@@ -3023,6 +3023,15 @@ Perl_utf8n_to_uvchr_msgs(const U8 * const s0,
         const U8 * s = s0;
         const U8 * send = s + curlen;
 
+        /* Find which dfa table to use.  If no restrictions, use [0].  If any,
+         * use at least [1]; and use [2] for the most restrictive */
+        const U8 * table =
+            PL_which_utf8_dfa_tab[
+                (bool) (flags & ( UTF8_WARN_ILLEGAL_INTERCHANGE
+                                 |UTF8_DISALLOW_ILLEGAL_INTERCHANGE))
+              + (bool) (flags & (UTF8_WARN_NONCHAR | UTF8_DISALLOW_NONCHAR))
+            ];
+
         /* This dfa is fast.  If it accepts the input, it was for a
          * well-formed, non-problematic code point, which can be returned
          * immediately.  Otherwise we call a helper function to figure out the
@@ -3038,13 +3047,13 @@ Perl_utf8n_to_uvchr_msgs(const U8 * const s0,
          * The terminology of the dfa refers to a 'class'.  The variable 'type'
          * would have been named 'class' except that is a reserved word in C++
          * */
-        PERL_UINT_FAST8_T type = PL_strict_utf8_dfa_tab[*s];
-        PERL_UINT_FAST8_T state = PL_strict_utf8_dfa_tab[256 + type];
+        PERL_UINT_FAST8_T type = table[*s];
+        PERL_UINT_FAST8_T state = table[256 + type];
         UV uv = (0xff >> type) & NATIVE_UTF8_TO_I8(*s);
 
         while (state > 1 && ++s < send) {
-            type  = PL_strict_utf8_dfa_tab[*s];
-            state = PL_strict_utf8_dfa_tab[256 + state + type];
+            type  = table[*s];
+            state = table[256 + state + type];
 
             uv = UTF8_ACCUMULATE(uv, *s);
         }

perl.h

@@ -6927,12 +6927,27 @@ EXTCONST U8 PL_c9_utf8_dfa_tab[] = {
 /*N7*/   1,  1,  1,  1,  1,  1,  1,  1,  1, N2,  1,  1,
 };
 
+/* There are 3 different tables driving the dfa, depending on what sort of
+ * restrictions are wanted.
+ *  [0] is where all of Perl's extended UTF-8 is accepted.
+ *  [2] is where what is acceptable is the subset of [0] that conforms to
+ *      Unicode's requirements for free exchange between processes.
+ *  [1] is where what is acceptable is the superset of [2] that conforms to
+ *      Unicode Corrigendum #9, for exchange between processes that each have
+ *      agreed not to send certain portions of [2]. */
+EXTCONST U8 * PL_which_utf8_dfa_tab[] = {
+                                          PL_extended_utf8_dfa_tab,
+                                          PL_c9_utf8_dfa_tab,
+                                          PL_strict_utf8_dfa_tab
+                                        };
+
 #    endif /* defined(PERL_CORE) */
 #  else     /* End of is DOINIT */
 
 EXTCONST U8 PL_extended_utf8_dfa_tab[];
 EXTCONST U8 PL_strict_utf8_dfa_tab[];
 EXTCONST U8 PL_c9_utf8_dfa_tab[];
+EXTCONST U8 * PL_which_utf8_dfa_tab[];
 
 #  endif
 #endif    /* end of isn't EBCDIC */