[Bug]: NC24 + PHP8.1 break UTF-8 compatibility

[ambraspace]

⚠️ This issue respects the following points: ⚠️

• This is a bug, not a question or a configuration/webserver/proxy issue.
• This issue is not already reported on Github (I've searched it).
• Nextcloud Server is up to date. See Maintenance and Release Schedule for supported versions.
• I agree to follow Nextcloud's Code of Conduct.

Bug description

Due to changes introduced in PHP8.1 creating files or directories with non-ascii characters result with problems such as:

• not displaying file/directory name correctly
• not being able to edit file
• not being able to delete file

I'm guessing it's this change from PHP change log:

htmlspecialchars(), htmlentities(), htmlspecialchars_decode(), html_entity_decode(), and get_html_translation_table() now use ENT_QUOTES | ENT_SUBSTITUTE rather than ENT_COMPAT by default. This means that ' is escaped to ' while previously nothing was done. Additionally, malformed UTF-8 will be replaced by a Unicode substitution character, instead of resulting in an empty string.

Actually, not being able to edit or delete files have been problem with Arch patched version of NC23 and PHP8.1. With NC24+PHP8.1 I am able to edit or delete text file, but the file name is not properly displayed.

I also noticed that a few contact names with non-ascii characters in name have not been displayed correctly.

Steps to reproduce

1. Install NC24 from master branch (I tested with last commit 9026455)
2. Create admin user and install Plain text editor.
3. Go to Files and create a directory named "šđčćž". The directory created will be displayed as "Å¡ÄÄÄž".
4. Create new text file named "šđčćž". An error ("An internal server error occurred") will be displayed, but the file will be created, although it will be displayed as "Å¡ÄÄÄž.txt".

Expected behavior

The file which has non-ascii characters in file name should be displayed as entered by a user (i.e. to fully support UTF-8 character set).

Installation method

Manual installation

Operating system

Other

PHP engine version

PHP 8.1

Web server

Apache (supported)

Database engine version

MariaDB

Is this bug present after an update or on a fresh install?

Fresh Nextcloud Server install

Are you using the Nextcloud Server Encryption module?

No response

What user-backends are you using?

• Default user-backend (database)
• LDAP/ Active Directory
• SSO - SAML
• Other

Configuration report

{
    "system": {
        "instanceid": "***REMOVED SENSITIVE VALUE***",
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "secret": "***REMOVED SENSITIVE VALUE***",
        "trusted_domains": [
            "localhost:8080"
        ],
        "datadirectory": "***REMOVED SENSITIVE VALUE***",
        "dbtype": "mysql",
        "version": "24.0.0.4",
        "overwrite.cli.url": "http:\/\/localhost:8080",
        "dbname": "***REMOVED SENSITIVE VALUE***",
        "dbhost": "***REMOVED SENSITIVE VALUE***",
        "dbport": "",
        "dbtableprefix": "oc_",
        "mysql.utf8mb4": true,
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "installed": true,
        "app_install_overwrite": [
            "files_texteditor"
        ]
    }
}

List of activated Apps

Enabled:
  - accessibility: 1.10.0
  - cloud_federation_api: 1.7.0
  - comments: 1.14.0
  - contactsinteraction: 1.5.0
  - dashboard: 7.4.0
  - dav: 1.22.0
  - federatedfilesharing: 1.14.0
  - federation: 1.14.0
  - files: 1.19.0
  - files_sharing: 1.16.0
  - files_texteditor: 2.14.0
  - files_trashbin: 1.14.0
  - files_versions: 1.17.0
  - lookup_server_connector: 1.12.0
  - oauth2: 1.12.0
  - provisioning_api: 1.14.0
  - settings: 1.6.0
  - sharebymail: 1.14.0
  - systemtags: 1.14.0
  - theming: 1.15.0
  - twofactor_backupcodes: 1.13.0
  - updatenotification: 1.14.0
  - user_status: 1.4.0
  - weather_status: 1.4.0
  - workflowengine: 2.6.0
Disabled:
  - admin_audit
  - bruteforcesettings
  - calendar
  - contacts
  - encryption
  - files_external
  - files_markdown
  - testing
  - user_ldap

Nextcloud Signing status

Integrity checker has been disabled. Integrity cannot be verified.

Nextcloud Logs

{"reqId":"wTxxhMBMjGnH9YtVcnrT","level":2,"time":"2022-02-15T20:59:58+00:00","remoteAddr":"10.0.2.2","user":"--","app":"no app in context","method":"GET","url":"/index.php","message":"Host localhost was not connected to because it violates local access rules","userAgent":"Mozilla/5.0 (X11; Linux x86_64; rv:97.0) Gecko/20100101 Firefox/97.0","version":""}
{"reqId":"wTxxhMBMjGnH9YtVcnrT","level":2,"time":"2022-02-15T20:59:58+00:00","remoteAddr":"10.0.2.2","user":"--","app":"no app in context","method":"GET","url":"/index.php","message":"Host localhost was not connected to because it violates local access rules","userAgent":"Mozilla/5.0 (X11; Linux x86_64; rv:97.0) Gecko/20100101 Firefox/97.0","version":""}
{"reqId":"YJdOQ0murWuS3PsCAWUL","level":2,"time":"2022-02-15T21:01:05+00:00","remoteAddr":"10.0.2.2","user":"--","app":"no app in context","method":"POST","url":"/index.php","message":"Host localhost was not connected to because it violates local access rules","userAgent":"Mozilla/5.0 (X11; Linux x86_64; rv:97.0) Gecko/20100101 Firefox/97.0","version":""}
{"reqId":"YJdOQ0murWuS3PsCAWUL","level":2,"time":"2022-02-15T21:01:05+00:00","remoteAddr":"10.0.2.2","user":"--","app":"no app in context","method":"POST","url":"/index.php","message":"Host localhost was not connected to because it violates local access rules","userAgent":"Mozilla/5.0 (X11; Linux x86_64; rv:97.0) Gecko/20100101 Firefox/97.0","version":""}
{"reqId":"k5RFMVQsebjj9ibhgVwg","level":2,"time":"2022-02-16T12:26:41+00:00","remoteAddr":"10.0.2.2","user":"--","app":"no app in context","method":"GET","url":"/","message":"Host localhost was not connected to because it violates local access rules","userAgent":"Mozilla/5.0 (X11; Linux x86_64; rv:97.0) Gecko/20100101 Firefox/97.0","version":""}
{"reqId":"k5RFMVQsebjj9ibhgVwg","level":2,"time":"2022-02-16T12:26:41+00:00","remoteAddr":"10.0.2.2","user":"--","app":"no app in context","method":"GET","url":"/","message":"Host localhost was not connected to because it violates local access rules","userAgent":"Mozilla/5.0 (X11; Linux x86_64; rv:97.0) Gecko/20100101 Firefox/97.0","version":""}
{"reqId":"10v48tQUEdbrkeM8owOE","level":2,"time":"2022-02-16T12:27:06+00:00","remoteAddr":"10.0.2.2","user":"--","app":"no app in context","method":"POST","url":"/index.php","message":"Host localhost was not connected to because it violates local access rules","userAgent":"Mozilla/5.0 (X11; Linux x86_64; rv:97.0) Gecko/20100101 Firefox/97.0","version":""}
{"reqId":"10v48tQUEdbrkeM8owOE","level":2,"time":"2022-02-16T12:27:06+00:00","remoteAddr":"10.0.2.2","user":"--","app":"no app in context","method":"POST","url":"/index.php","message":"Host localhost was not connected to because it violates local access rules","userAgent":"Mozilla/5.0 (X11; Linux x86_64; rv:97.0) Gecko/20100101 Firefox/97.0","version":""}
{"reqId":"j4SGoponDpUMKbocZOqh","level":2,"time":"2022-02-16T12:27:32+00:00","remoteAddr":"10.0.2.2","user":"--","app":"no app in context","method":"POST","url":"/index.php","message":"Host localhost was not connected to because it violates local access rules","userAgent":"Mozilla/5.0 (X11; Linux x86_64; rv:97.0) Gecko/20100101 Firefox/97.0","version":"24.0.0.4"}
{"reqId":"j4SGoponDpUMKbocZOqh","level":2,"time":"2022-02-16T12:27:32+00:00","remoteAddr":"10.0.2.2","user":"--","app":"no app in context","method":"POST","url":"/index.php","message":"Host localhost was not connected to because it violates local access rules","userAgent":"Mozilla/5.0 (X11; Linux x86_64; rv:97.0) Gecko/20100101 Firefox/97.0","version":"24.0.0.4"}

Additional info

No response

[PVince81]

there was a similar issue earlier, not sure if related: #29296

[come-nc]

• UTF-8 encoding detection fails on PHP 8.1 sabre-io/http#181

[come-nc]

@ambraspace So after looking into it this comes from a change in PHP itself which tried to get better at autodetecting encodings.
In this case the accentuated letter you used making no sense PHP consider it unlikely to be UTF-8.
Did you encounter this in a reallife usecase with a file name or was it only from testing with gibberish?

We are looking into not autodetecting encoding if it is valid UTF-8 but it is not clear yet if this is the right path.

[ambraspace]

@come-nc I encountered this in real life use case. Letters šđčćž are quite common in Slavic languages (Serbian, Croatian, Slovenian, Slovak, Czech and such). Probably more then 90% names in Serbian language end with "ić".

I thought everything is UTF-8 by default these days...

[come-nc]

@come-nc I encountered this in real life use case. Letters šđčćž are quite common in Slavic languages (Serbian, Croatian, Slovenian, Slovak, Czech and such). Probably more then 90% names in Serbian language end with "ić".

I thought everything is UTF-8 by default these days...

Then can you state so and give real life examples in php/php-src#8279 ?
Maybe in sabre-io/http#181 also.