external smb/cifs storages not (always?) refreshing

[kkplein]

Nextcloud 18.04, debian 10 with stock smbclient / php-smbclient, connected to AD/ldap.

Configured external storages pointing to our internal fileserver, using cifs/smb, Log-in credentials, save in session.

If user has no rights on a shared folder, the directory is displayed as an Empty directory. ("No files in here")

The user requests access, we add the user to the AD group, user logs out NC, logs in NC again, tries again to access the same folder, but STILL: "No files in here". However, the user has access now, as we can confirm through windows fileserver access and also when manually using smbclient from the nextcloud server, the user CAN see and access the directory contents now.

In NC, the user has to create a NEW file in that (seemingly empty) directory, and then suddenly the contents is refreshed, and all existing files/folders show up.

So, NC was showing old, cached "No files in here" info.
Browser refresh button etc, makes no difference.

We have:

• user backend: AD/LDAP
• configured the smb/cifs storage to "Check for changes once every direct access".
• config.php contains: filesystem_check_changes = 1

We are running occ files:scan --all, but we guess it doesn't work, as the smbclient storage is only mounted on demand, and not always available to be scanned.

Operating system:
Debian 10.3

Web server:
Apache 2.4.38

Database:
mariadb 10.3.22

PHP version:
7.3.14

Nextcloud version: (see Nextcloud admin page)
18.04

Updated from an older Nextcloud/ownCloud or fresh install:
fresh

Where did you install Nextcloud from:
regular download from nextcloud

Login as admin user into your Nextcloud and access 
http://example.com/index.php/settings/integrity/failed 
No errors were found

List of activated apps:

App list

Enabled: - accessibility: 1.4.0 - admin_audit: 1.8.0 - bruteforcesettings: 1.6.0 - cloud_federation_api: 1.1.0 - comments: 1.8.0 - dav: 1.14.0 - federatedfilesharing: 1.8.0 - federation: 1.8.0 - files: 1.13.1 - files_external: 1.9.0 - files_pdfviewer: 1.7.0 - files_rightclick: 0.15.2 - files_sharing: 1.10.1 - files_trashbin: 1.8.0 - files_versions: 1.11.0 - files_videoplayer: 1.7.0 - logreader: 2.3.0 - lookup_server_connector: 1.6.0 - nextcloud_announcements: 1.7.0 - notifications: 2.6.0 - oauth2: 1.6.0 - privacy: 1.2.0 - provisioning_api: 1.8.0 - recommendations: 0.6.0 - serverinfo: 1.8.0 - settings: 1.0.0 - sharebymail: 1.8.0 - support: 1.1.0 - survey_client: 1.6.0 - systemtags: 1.8.0 - text: 2.0.0 - theming: 1.9.0 - twofactor_backupcodes: 1.7.0 - updatenotification: 1.8.0 - user_ldap: 1.8.0 - viewer: 1.2.0 - workflowengine: 2.0.0

Nextcloud configuration:

Config report

{ "system": { "lost_password_link": "https:\/\/id....removed", "auth.bruteforce.protection.enabled": true, "overwritehost": "cloud....removed", "overwriteprotocol": "https", "htaccess.RewriteBase": "\/", "filesystem_check_changes": 1, "instanceid": "***REMOVED SENSITIVE VALUE***", "passwordsalt": "***REMOVED SENSITIVE VALUE***", "secret": "***REMOVED SENSITIVE VALUE***", "trusted_domains": [ "cloud....removed" ], "memcache.local": "\\OC\\Memcache\\Redis", "memcache.locking": "\\OC\\Memcache\\Redis", "redis": { "host": "***REMOVED SENSITIVE VALUE***", "port": 0, "timeout": 0, "password": "***REMOVED SENSITIVE VALUE***", "dbindex": 0 }, "datadirectory": "***REMOVED SENSITIVE VALUE***", "dbtype": "mysql", "version": "18.0.4.2", "overwrite.cli.url": "https:\/\/cloud....removed", "dbname": "***REMOVED SENSITIVE VALUE***", "dbhost": "***REMOVED SENSITIVE VALUE***", "dbport": "", "dbtableprefix": "oc_", "mysql.utf8mb4": true, "dbuser": "***REMOVED SENSITIVE VALUE***", "dbpassword": "***REMOVED SENSITIVE VALUE***", "installed": true, "ldapIgnoreNamingRules": false, "ldapProviderFactory": "OCA\\User_LDAP\\LDAPProviderFactory", "mail_smtpmode": "smtp", "mail_smtphost": "***REMOVED SENSITIVE VALUE***", "mail_sendmailmode": "smtp", "mail_smtpport": "465", "maintenance": false, "app_install_overwrite": [ "rainloop" ] } }

Are you using external storage, if yes which one: local/smb/sftp/...
yes: smbclient

Are you using encryption: yes/no
No

Are you using an external user-backend, if yes which one: LDAP/ActiveDirectory/Webdav/...
Yes, LDAP.

LDAP configuration (delete this part if not used)

If needed< i can post LDAP config details, but I'd rather not. Everything group/ldap/auth seems to work fine, for the rest.

I think details are complete as I provided them. Please let me know if you need additional details.

[Ravinou]

I have a similar problem (i'm on 18.0.4). If local storage is inaccessible for a few minutes, it won't refresh. The external storage will have to be deleted and recreated. This will result in the loss of all public links created in the external storage.

[Ravinou]

Running "cron.php" not work for me.
Running "occ files:scan --all" works for me but all share links are removed !!

What is doing the option : "Check for changes : once every direct access" ?

Thanks

[thorgahr]

Same issue here with external storage support app, Samba share integrated into nextcloud.

For example:

A directory with five files, three of them added using the Windows Explorer (SMB/CIFS Share), and two of the files added using the nextcloud frontend (no difference if browser or app is used).
In Windows Explorer, all five files are available, in nextcloud are only the two added over nextcloud displayed.
When "occ files:scan" is run on the described directory, only the two nextcloud files are found (1 folder, 2 files). When the content of the directory is listed by use of "ls" on the server, all five files are displayed.
I have checked the access rights for filesystem, apache service, occ tool and so on, it seems all correct for me.

[RHCPNG]

I believe I have the same problem. I'm running nextcloud in a docker and I have a nfs share attached to the container, which is added to nextcloud as external storage. However, when I add files without using nextcloud, the files are not visible in nextcloud.

I have tried to set the permissions, owner as the docker user and 775 permissions, to no success. I have also run the occ file scan.

[kkplein]

It's a pity there is so little activity on this ticket. :-( If only there would be a way to completey disable any directory contents caching...

[Zeoic]

I am having the same/similar issue as well (mine is a local external storage pointed to an smb share added by the container. SMB external storages are brutally slow). I need to connect to my container and run the occ scan any time I add a file via smb. The folder is 32 TB with 280,000 files, so a full scan would take ages to complete, which means anytime I add a file and want it in next cloud I need to specify what folder the file change was in as well. It would be nice if the "Check for changes: Once every direct access" actually did anything, or at the very least, a re-scan current folder button was added.

[mr-prud]

Hi,
Same behavior with FTP external_files type.

[Ravinou]

Regardless of the protocol, the "Check for changes once every direct access" feature does not work and if unfortunately there is a connection problem with the external storage, upon reconnection all shares created inside this external storage are deleted!

[Ravinou]

I just upgraded to Nextcloud 19.01 and option "Check for changes once every direct access" seems working, somebody else can confirm ?
*Edit : this is still not optimal and in case of temporary disconnection of the external storage some directories still appear empty after reconnection. Shares are always deleted in case of temporary disconnection.
So it's still unusable for me. I hope that one day this ticket will be read 😅 🙏

[kkplein]

Just checked, and YES, it the contents of the share in nextcloud refreshed, after updating the cifs share outside nextcloud. So that bit seems to be solved. I have not yet noticed your issue of losing the defined shares, when the external storage becomes disconnected temporarily.

[Ravinou]

@kkplein
1- create a public share on a file or on a folder in your external storage.
2- Open the link in a tab.
3- Make your external storage inaccessible (without deleting it in the nextcloud's external storage options, just make sure your nextcloud can't access it anymore).
4- Refresh your tab open on step 2. Normally there will be an error, and it's normal.
5- Make your external storage accessible again.
6- Now, public link open on step 2 is delete for me and all shares created in this external storage are deleted.

[Ravinou]

I'm not a developper, but i find this :

server/apps/files_sharing/lib/External/Storage.php

Line 217 in 0934aee

// valid Nextcloud instance means that the public share no longer exists

// valid Nextcloud instance means that the public share no longer exists
// since this is permanent (re-sharing the file will create a new token)
// we remove the invalid storage

Would that explain what I'm experiencing ?

[kkplein]

Seems it would... a bit drastic to remove it completely...