[Bug]: NC28.0.2, post update encryption failing for one user

[mdunphy]

⚠️ This issue respects the following points: ⚠️

• This is a bug, not a question or a configuration/webserver/proxy issue.
• This issue is not already reported on Github OR Nextcloud Community Forum (I've searched it).
• Nextcloud Server is up to date. See Maintenance and Release Schedule for supported versions.
• I agree to follow Nextcloud's Code of Conduct.

Bug description

I updated from 27.1.6 to 28.0.2. After the update, one user (the admin user) is unable to see any files in the Web App -- loading the Files tab shows no files. This is similar to #43268 in that only one user is affected, but I have server side encryption on, so it's not exactly the same. The browser tools shows that the PROPFIND part returns a 500 Internal Server Error, which is similar to #43269, except again I have the server side encryption on.

On the server side, the nextcloud log (fragment included below) on the PROPFIND call says "Cannot decrypt this file, probably this is a shared file. Please ask the file owner to reshare the file with you.", however, adding a debug printout to lib/private/Encryption/Keys/Storage.php in getKey() reveals that it is trying to load the fileKey for a file that has not been shared. So this is probably a misleading error.

The unaffected users have {fileKey,username.shareKey} under files_encryption/.../OC_DEFAULT_MODULE for each file, and the admin user has {recoveryKey_########.sharekey,username.shareKey} for each file. So it looks like the problem could be that the admin user's fileKeys got erased during the upgrade? On the other hand, the admin user can see files via the "Activity" or "Photos" tabs, so it seems like the files are present and decrypt-able, so that makes it seem more like a bug. Eg, the decryption machinery works when reaching the files via Photos (which must not need PROPFIND) but fails to list all files (which does need PROPFIND).

Master key is off, recovery key is on for the affected admin user.

Steps to reproduce

1. Update next cloud from 27.1.6 to 28.0.2 using the web updater, it says successful
2. Login as the admin user to admire files
3. No files shown

Expected behavior

Files should show up in the files tab, no 500 errors

Installation method

Community Web installer on a VPS or web space

Nextcloud Server version

28

Operating system

Debian/Ubuntu

PHP engine version

PHP 8.1

Web server

Apache (supported)

Database engine version

MySQL

Is this bug present after an update or on a fresh install?

Upgraded to a MAJOR version (ex. 22 to 23)

Are you using the Nextcloud Server Encryption module?

Encryption is Enabled

What user-backends are you using?

• Default user-backend (database)
• LDAP/ Active Directory
• SSO - SAML
• Other

Configuration report

{
    "system": {
        "instanceid": "***REMOVED SENSITIVE VALUE***",
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "secret": "***REMOVED SENSITIVE VALUE***",
        "trusted_domains": [
            "***REMOVED SENSITIVE VALUE***"
        ],
        "datadirectory": "***REMOVED SENSITIVE VALUE***",
        "overwrite.cli.url": "***REMOVED SENSITIVE VALUE***",
        "dbtype": "mysql",
        "version": "28.0.2.5",
        "dbname": "***REMOVED SENSITIVE VALUE***",
        "dbhost": "***REMOVED SENSITIVE VALUE***",
        "dbport": "",
        "dbtableprefix": "oc_",
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "logtimezone": "UTC",
        "installed": true,
        "maintenance": false,
        "theme": "",
        "loglevel": 2,
        "encryption.legacy_format_support": false,
        "encryption.key_storage_migrated": false,
        "trashbin_retention_obligation": "auto, 10",
        "updater.release.channel": "stable",
        "default_phone_region": "CA",
        "updater.secret": "***REMOVED SENSITIVE VALUE***"
    }
}

List of activated Apps

Enabled:
  - activity: 2.20.0
  - bruteforcesettings: 2.8.0
  - circles: 28.0.0-dev
  - cloud_federation_api: 1.11.0
  - comments: 1.18.0
  - contactsinteraction: 1.9.0
  - dashboard: 7.8.0
  - dav: 1.29.1
  - encryption: 2.16.0
  - federatedfilesharing: 1.18.0
  - federation: 1.18.0
  - files: 2.0.0
  - files_pdfviewer: 2.9.0
  - files_reminders: 1.1.0
  - files_sharing: 1.20.0
  - files_trashbin: 1.18.0
  - firstrunwizard: 2.17.0
  - logreader: 2.13.0
  - lookup_server_connector: 1.16.0
  - nextcloud_announcements: 1.17.0
  - notes: 4.9.2
  - notifications: 2.16.0
  - oauth2: 1.16.3
  - password_policy: 1.18.0
  - photos: 2.4.0
  - privacy: 1.12.0
  - provisioning_api: 1.18.0
  - recommendations: 2.0.0
  - related_resources: 1.3.0
  - serverinfo: 1.18.0
  - settings: 1.10.1
  - sharebymail: 1.18.0
  - support: 1.11.0
  - survey_client: 1.16.0
  - systemtags: 1.18.0
  - text: 3.9.1
  - theming: 2.3.0
  - twofactor_backupcodes: 1.17.0
  - updatenotification: 1.18.0
  - user_status: 1.8.1
  - viewer: 2.2.0
  - weather_status: 1.8.0
  - workflowengine: 2.10.0
Disabled:
  - admin_audit: 1.18.0
  - files_external: 1.20.0
  - files_versions: 1.21.0 (installed 1.8.0)
  - suspicious_login: 6.0.0
  - twofactor_totp: 10.0.0-beta.2
  - user_ldap: 1.19.0

Nextcloud Signing status

No errors have been found.

Nextcloud Logs

{"reqId":"wFdAN9FN7eu2tyIH52KC","level":3,"time":"2024-02-03T07:27:49+00:00","remoteAddr":"aa.bb.cc.dd","user":"username","app":"webdav","method":"PROPFIND","url":"/nextcloud/remote.php/dav/files/username/","message":"Cannot decrypt this file, probably this is a shared file. Please ask the file owner to reshare the file with you.","userAgent":"Mozilla/5.0 (Windows NT 10.0; rv:121.0) Gecko/20100101 Firefox/121.0","version":"28.0.2.5","exception":{"Exception":"OC\\Encryption\\Exceptions\\DecryptionFailedException","Message":"Cannot decrypt this file, probably this is a shared file. Please ask the file owner to reshare the file with you.","Code":0,"Trace":[{"file":"/mnt/storage/nextcloud/lib/private/Files/Stream/Encryption.php","line":517,"function":"decrypt","class":"OCA\\Encryption\\Crypto\\Encryption","type":"->"},{"file":"/mnt/storage/nextcloud/lib/private/Files/Stream/Encryption.php","line":316,"function":"readCache","class":"OC\\Files\\Stream\\Encryption","type":"->"},{"function":"stream_read","class":"OC\\Files\\Stream\\Encryption","type":"->"},{"file":"/mnt/storage/nextcloud/lib/private/Files/Storage/Wrapper/Encryption.php","line":240,"function":"stream_get_contents"},{"file":"/mnt/storage/nextcloud/lib/private/Files/Storage/Wrapper/Wrapper.php","line":244,"function":"file_get_contents","class":"OC\\Files\\Storage\\Wrapper\\Encryption","type":"->"},{"file":"/mnt/storage/nextcloud/lib/private/Files/View.php","line":1161,"function":"file_get_contents","class":"OC\\Files\\Storage\\Wrapper\\Wrapper","type":"->"},{"file":"/mnt/storage/nextcloud/lib/private/Files/View.php","line":581,"function":"basicOperation","class":"OC\\Files\\View","type":"->"},{"file":"/mnt/storage/nextcloud/lib/private/Files/Node/File.php","line":54,"function":"file_get_contents","class":"OC\\Files\\View","type":"->"},{"file":"/mnt/storage/nextcloud/apps/text/lib/DAV/WorkspacePlugin.php","line":119,"function":"getContent","class":"OC\\Files\\Node\\File","type":"->"},{"file":"/mnt/storage/nextcloud/3rdparty/sabre/dav/lib/DAV/PropFind.php","line":95,"function":"OCA\\Text\\DAV\\{closure}","class":"OCA\\Text\\DAV\\WorkspacePlugin","type":"->"},{"file":"/mnt/storage/nextcloud/apps/text/lib/DAV/WorkspacePlugin.php","line":122,"function":"handle","class":"Sabre\\DAV\\PropFind","type":"->"},{"file":"/mnt/storage/nextcloud/3rdparty/sabre/event/lib/WildcardEmitterTrait.php","line":89,"function":"propFind","class":"OCA\\Text\\DAV\\WorkspacePlugin","type":"->"},{"file":"/mnt/storage/nextcloud/3rdparty/sabre/dav/lib/DAV/Server.php","line":1052,"function":"emit","class":"Sabre\\DAV\\Server","type":"->"},{"file":"/mnt/storage/nextcloud/3rdparty/sabre/dav/lib/DAV/Server.php","line":984,"function":"getPropertiesByNode","class":"Sabre\\DAV\\Server","type":"->"},{"file":"/mnt/storage/nextcloud/3rdparty/sabre/dav/lib/DAV/Server.php","line":1662,"function":"getPropertiesIteratorForPath","class":"Sabre\\DAV\\Server","type":"->"},{"file":"/mnt/storage/nextcloud/3rdparty/sabre/dav/lib/DAV/Server.php","line":1647,"function":"writeMultiStatus","class":"Sabre\\DAV\\Server","type":"->"},{"file":"/mnt/storage/nextcloud/3rdparty/sabre/dav/lib/DAV/CorePlugin.php","line":346,"function":"generateMultiStatus","class":"Sabre\\DAV\\Server","type":"->"},{"file":"/mnt/storage/nextcloud/3rdparty/sabre/event/lib/WildcardEmitterTrait.php","line":89,"function":"httpPropFind","class":"Sabre\\DAV\\CorePlugin","type":"->"},{"file":"/mnt/storage/nextcloud/3rdparty/sabre/dav/lib/DAV/Server.php","line":472,"function":"emit","class":"Sabre\\DAV\\Server","type":"->"},{"file":"/mnt/storage/nextcloud/3rdparty/sabre/dav/lib/DAV/Server.php","line":253,"function":"invokeMethod","class":"Sabre\\DAV\\Server","type":"->"},{"file":"/mnt/storage/nextcloud/3rdparty/sabre/dav/lib/DAV/Server.php","line":321,"function":"start","class":"Sabre\\DAV\\Server","type":"->"},{"file":"/mnt/storage/nextcloud/apps/dav/lib/Server.php","line":370,"function":"exec","class":"Sabre\\DAV\\Server","type":"->"},{"file":"/mnt/storage/nextcloud/apps/dav/appinfo/v2/remote.php","line":35,"function":"exec","class":"OCA\\DAV\\Server","type":"->"},{"file":"/mnt/storage/nextcloud/remote.php","line":172,"args":["/mnt/storage/nextcloud/apps/dav/appinfo/v2/remote.php"],"function":"require_once"}],"File":"/mnt/storage/nextcloud/apps/encryption/lib/Crypto/Encryption.php","Line":353,"Hint":"Cannot decrypt this file, probably this is a shared file. Please ask the file owner to reshare the file with you.","message":"Cannot decrypt this file, probably this is a shared file. Please ask the file owner to reshare the file with you.","exception":{},"CustomMessage":"Cannot decrypt this file, probably this is a shared file. Please ask the file owner to reshare the file with you."}}

Additional info

No response

[joshtrichards]

Reminds me a bit of when I recently looked into #42663

Does disabling rich workspaces globally workaround the behavior? If you're able to do so, test it like this:

occ config:app:set text workspace_available --value=0

(restart your app/FPM server for good measure after doing so).

[mdunphy]

@joshtrichards Ha - many thanks! That helped a bunch --- the files now show up in the Files part of the web app and PROPFIND no longer fails.

Manually browsing through the files now,

• Some are working (images, which is consistent with being able to see them via Photos before)
• Some are failing to decrypt with the same misleading "probably its a shared file", but it is definitely occurring on files that are not shared (this is mostly non-images such as ODT and PDFs, but not all of them)

[mdunphy]

In terms of unshared files, files that successfully decrypt and files that fail to decrypt, both have a recoveryKey_########.shareKey & username.shareKey in their corresponding files_encryption/.../OC_DEFAULT_MODULE location, and do not have a fileKey. Unclear why these are sufficient for some files but not for others. No pattern identified yet beyond that images generally work and PDF/ODT generally do not

[mdunphy]

For posterity: since this problem was contained to a single user, I was able to fix it by restoring an older backup of nextcloud/data/username, running occ files:scan username, and then running occ maintenance:data-fingerprint (https://docs.nextcloud.com/server/latest/admin_manual/maintenance/restore.html#synchronising-with-clients-after-data-recovery) to restore everything newer than the backup from a client, which required choosing "use client versions" when the nextcloud desktop client asked how to resolve the conflicts. The Activity tab is a bit full of superfluous activity but otherwise nothing was lost.