PATCH: [perl #89774] multi-char fold + its fold in char class

The design for handling characters that fold to multiple characters when
the former are encountered in a bracketed character class is defective.
The ticket reads, "If a bracketed character class includes a character
that has a multi-char fold, and it also includes the first character of
that fold, the multi-char fold will never be matched; just the first
character of the fold.".   Thus, in the class /[\0-\xff]/i, \xDF will
never be matched, because its fold is 'ss', the first character of
which, 's', is also in the class.

The reason the design is defective is that it doesn't allow for
backtracking and trying the other options.

This commit solves this by effectively rewriting the above to be
/ (?: \xdf | [\0-\xde\xe0-\xff] ) /xi.  And so the backtracking gets
handled automatcially by the regex engine.

Author: Karl Williamson

embedvar.h

@@ -53,6 +53,7 @@
 #define PL_DBtrace		(vTHX->IDBtrace)
 #define PL_Dir			(vTHX->IDir)
 #define PL_Env			(vTHX->IEnv)
+#define PL_HasMultiCharFold	(vTHX->IHasMultiCharFold)
 #define PL_L1Cased		(vTHX->IL1Cased)
 #define PL_L1PosixAlnum		(vTHX->IL1PosixAlnum)
 #define PL_L1PosixAlpha		(vTHX->IL1PosixAlpha)

intrpvar.h

@@ -609,6 +609,7 @@ PERLVAR(I, XPosixXDigit, SV *)
 PERLVAR(I, VertSpace,   SV *)
 
 PERLVAR(I, NonL1NonFinalFold,   SV *)
+PERLVAR(I, HasMultiCharFold,   SV *)
 
 /* utf8 character class swashes */
 PERLVAR(I, utf8_alnum,	SV *)

pod/perldelta.pod

@@ -53,6 +53,30 @@ XXX For a release on a stable branch, this section aspires to be:
 
 [ List each incompatible change as a =head2 entry ]
 
+=head2 New Restrictions in Multi-Character Case-Insensitive Matching in Regular Expression Bracketed Character Classes
+
+Unicode has now withdrawn their previous recommendation for regular
+expressions to automatically handle cases where a single character can
+match multiple characters case-insensitively; for example, the letter
+LATIN SMALL LETTER SHARP S and the sequence C<ss>.  This is because
+it turns out to be impracticable to do this correctly in all
+circumstances.  Because Perl has tried to do this as best it can, it
+will continue to do so.  (We are considering an option to turn it off.)
+However, a new restriction is being added on such matches when they
+occur in [bracketed] character classes.  People were specifying
+things such as C</[\0-\xff]/i>, and being surprised that it matches the
+two character sequence C<ss> (since LATIN SMALL LETTER SHARP S occurs in
+this range).  This behavior is also inconsistent with the using a
+property instead of a range:  C<\p{Block=Latin1}> also includes LATIN
+SMALL LETTER SHARP S, but C</[\p{Block=Latin1}]/i> does not match C<ss>.
+The new rule is that for there to be a multi-character case-insensitive
+match within a bracketed character class, the character must be
+explicitly listed, and not as an end point of a range.  This more
+closely obeys the Principle of Least Astonishment.  See
+L<perlrecharclass/Bracketed Character Classes>.  Note that a bug [perl
+#89774], now fixed as part of this change, prevented the previous
+behavior from working fully.
+
 =head1 Deprecations
 
 XXX Any deprecated features, syntax, modules etc. should be listed here.  In
@@ -315,7 +339,9 @@ well.
 
 =item *
 
-XXX
+Case-insensitive matching inside a [bracketed] character class with a
+multi-character fold, no longer excludes one of the possibilities in the
+circumstances that it used to. [perl #89774].
 
 =back
 

pod/perlre.pod

@@ -72,27 +72,13 @@ are split between groupings, or when one or more are quantified.  Thus
  # be even if it did!!
  "\N{LATIN SMALL LIGATURE FI}" =~ /(f)(i)/i;      # Doesn't match!
 
-Perl doesn't match multiple characters in an inverted bracketed
-character class, which otherwise could be highly confusing.  See
+Perl doesn't match multiple characters in a bracketed
+character class unless the character that maps to them is explicitly
+mentioned, and it doesn't match them at all if the character class is
+inverted, which otherwise could be highly confusing.  See
+L<perlrecharclass/Bracketed Character Classes>, and
 L<perlrecharclass/Negation>.
 
-Another bug involves character classes that match both a sequence of
-multiple characters, and an initial sub-string of that sequence.  For
-example,
-
- /[s\xDF]/i
-
-should match both a single and a double "s", since C<\xDF> (on ASCII
-platforms) matches "ss".  However, this bug
-(L<[perl #89774]|https://rt.perl.org/rt3/Ticket/Display.html?id=89774>)
-causes it to only match a single "s", even if the final larger match
-fails, and matching the double "ss" would have succeeded.
-
-Also, Perl matching doesn't fully conform to the current Unicode C</i>
-recommendations, which ask that the matching be made upon the NFD
-(Normalization Form Decomposed) of the text.  However, Unicode is
-in the process of reconsidering and revising their recommendations.
-
 =item x
 X</x>
 

pod/perlrecharclass.pod

@@ -441,7 +441,8 @@ Examples:
 
 * There is an exception to a bracketed character class matching a
 single character only.  When the class is to match caselessly under C</i>
-matching rules, and a character inside the class matches a
+matching rules, and a character that is explicitly mentioned inside the
+class matches a
 multiple-character sequence caselessly under Unicode rules, the class
 (when not L<inverted|/Negation>) will also match that sequence.  For
 example, Unicode says that the letter C<LATIN SMALL LETTER SHARP S>
@@ -450,6 +451,18 @@ should match the sequence C<ss> under C</i> rules.  Thus,
  'ss' =~ /\A\N{LATIN SMALL LETTER SHARP S}\z/i             # Matches
  'ss' =~ /\A[aeioust\N{LATIN SMALL LETTER SHARP S}]\z/i    # Matches
 
+For this to happen, the character must be explicitly specified, and not
+be part of a multi-character range (not even as one of its endpoints).
+(L</Character Ranges> will be explained shortly.)  Therefore,
+
+ 'ss' =~ /\A[\0-\x{ff}]\z/i        # Doesn't match
+ 'ss' =~ /\A[\0-\N{LATIN SMALL LETTER SHARP S}]\z/i    # No match
+ 'ss' =~ /\A[\xDF-\xDF]\z/i    # Matches on ASCII platforms, since \XDF
+                               # is LATIN SMALL LETTER SHARP S, and the
+                               # range is just a single element
+
+Note that it isn't a good idea to specify these types of ranges anyway.
+
 =head3 Special Characters Inside a Bracketed Character Class
 
 Most characters that are meta characters in regular expressions (that