mb_detect_encoding() results for UTF-7 differ between PHP 8.0 and 8.1 (if UTF-7 is present in the encodings list and the string contains '+' character)

[unix-world]

Description

This is a very serious bug that may impact all strings in PHP.

The following code:

<?php

ini_set('display_errors', '1');	// display runtime errors
error_reporting(E_ALL & ~E_NOTICE & ~E_STRICT & ~E_DEPRECATED); // error reporting
date_default_timezone_set('UTC');

function detect_encoding($ystr, $csetlist='UTF-8, ISO-8859-1, ISO-8859-15, ISO-8859-2, ISO-8859-9, ISO-8859-3, ISO-8859-4, ISO-8859-5, ISO-8859-6, ISO-8859-7, ISO-8859-8, ISO-8859-10, ISO-8859-13, ISO-8859-14, ISO-8859-16, UTF-7, ASCII, SJIS, EUC-JP, JIS, ISO-2022-JP, EUC-CN, GB18030, ISO-2022-KR, KOI8-R, KOI8-U') {
	return mb_detect_encoding((string)$ystr, (string)$csetlist, true); // mixed: (bool) FALSE or (string) 'CHARSET'
}

echo detect_encoding('A + B'); // expected output: UTF-8, but on PHP 8.1.x / 8.2.x returns UTF-7 if the '+' character is present in a string ; the PHP 8.0.x and 7.4.x behaves correctly and outputs UTF-8 also in this case
echo "\n";
echo detect_encoding('A - B'); // expected output: UTF-8 ; correct on all PHP versions 8.2.x / 8.1.x / 8.0.x / 7.4.x

Resulted in this output ; On PHP 8.2 and 8.1, if the plus (+) character is present in a string will detect UTF-7 instead of UTF-8 as expected:

UTF-7
UTF-8

But I expected this output instead ; on PHP 7.4 and PHP 8.0 works correctly. Output is this):

UTF-8
UTF-8

I tested here with different PHP versions.
https://onlinephp.io/
I also tested in my computer with PHP 8.1.12 and 8.0.25

PHP Version

8.1.x / 8.2.x

Operating System

All

[Girgias]

I honestly doubt the severity of this bug as AFAIK any UTF-7 string is valid UTF-8, moreover mb_detect_encoding() is a misnomer as it tries to guess what encoding a string might be.

But I'll let @alexdowad comment, as he's the character encoding expect and has been refactoring mbstring.

[unix-world]

It is very serious. Because if you need to normalize the output you rely on re-encoding all strings as UTF-8.
Thus, if re-encode 'A + B' from UTF-7 (which is actually UTF-8 already, but is wrong detected by PHP 8.1/8.2) to UTF-8 results in 'A B' (the plus sign disappears), as the strings get corrupted.
Actually this is how I start investigating and get to this bug.
Scenario:
Read from a data source -> Filter UTF-8 or re-encode as UTF-8 -> Output. If a string contains '+' sign will be mangled after normalization (re-encode as UTF-8 which actually would be not necessary if UTF-8 would be correctly detected as in PHP 8.0 / 7.4).

[unix-world]

It impacts so many things in an unpredictable way so I can't explain all situations here.

[unix-world]

I updated the example for you to see the real disaster !

<?php

ini_set('display_errors', '1');	// display runtime errors
error_reporting(E_ALL & ~E_NOTICE & ~E_STRICT & ~E_DEPRECATED); // error reporting
date_default_timezone_set('UTC');

function detect_encoding($ystr, $csetlist='UTF-8, ISO-8859-1, ISO-8859-15, ISO-8859-2, ISO-8859-9, ISO-8859-3, ISO-8859-4, ISO-8859-5, ISO-8859-6, ISO-8859-7, ISO-8859-8, ISO-8859-10, ISO-8859-13, ISO-8859-14, ISO-8859-16, UTF-7, ASCII, SJIS, EUC-JP, JIS, ISO-2022-JP, EUC-CN, GB18030, ISO-2022-KR, KOI8-R, KOI8-U') { // Fix: starting from PHP 7.1 it warns about illegal argument if using: ISO-8859-11
	return mb_detect_encoding((string)$ystr, (string)$csetlist, true); // mixed: (bool) FALSE or (string) 'CHARSET'
}

$str = 'A + B';
echo detect_encoding($str);
echo "\n";
echo mb_convert_encoding($str, 'UTF-8', detect_encoding($str));
echo "\n";
echo "\n";
$str = 'A - B';
echo detect_encoding('A - B');
echo "\n";
echo mb_convert_encoding($str, 'UTF-8', detect_encoding($str));

expected output (OK in PHP 8.0 / 7.4)

UTF-8
A + B

UTF-8
A - B

wrong output in PHP 8.1 / 8.2 (the + character is mangled, disappeared)

UTF-7
A  B

UTF-8
A - B

Thus in my opinion would affect a lot of database data. This is the most serious bug in PHP since I have seen. If you re-save in database all the data becomes corrupted without a chance to go backward only if you have a valid backup !!

[unix-world]

I suspect a memory corruption there or something very ugly ...
If I convert a string from UTF-7 to UTF-8 should be preserved as is. Right ?
But actually because a string which contains + character is already UTF-8 but PHP detects it as UTF-7 it corrupts that string on re-encoding and + characters vanish ... !

[unix-world]

And if 'A + B' is detected as UTF-7 instead of UTF-8 on PHP 8.1/8.2,... why 'A - B' is detected as UTF-8 ? Can't you see the anomaly in this case ?