export then import calender partially fails (error: 5 database is locked) #5647
Description
Steps to reproduce
- I was using an upgraded nextcloud 27.1.5 instance. I have a calendar full of annual recurring events (brithdays etc) called yearly
- export the calendar "yearly" to a file
- create a new calendar called test1
- try to import the just created exported calendar file into test1
- You will get pop showing a different number of success out total each time you try!
- Looking in the logfile you will see a load errors
notes: same when trying to import the exported file from current instance into a fresh install of 28.0.1 or fresh install of 27.1.5 as well as using the current version that was updated from 27.1.4.
Expected behavior
you should get popup showing alll the entries where imported, and no errors (I assume) in the logfile
Actual behaviour
You get a pop up the shows only a subset of the entries were inported and you get errors in the log, which changes with each attempt:
Mainly:
OC\DB\Exceptions\DbalException: An exception occurred while executing a query: SQLSTATE[HY000]: General error: 5 database is locked
also:
Sabre\VObject\Recur\MaxInstancesExceededException: Recurring events are only allowed to generate 3500
Also:
Sabre\DAV\Exception\UnsupportedMediaType: Validation error in iCalendar: More than one master object was found for the object with UID 3c1b8ebf-1bf0-4bbe-abdc-8b560e988c4e
Calendar app version
4.6.1
CalDAV-clients used
nextcloud 27.1.5 and 28.0.1
Browser
Firefox 121.0 (64bit)
Client operating system
Windows 11 (patched)
Server operating system
ARM Linux (rpi4 64bit)
Web server
Apache
Database engine version
SQlite
PHP engine version
PHP 8.2
Nextcloud version
27.1.5 and 28.0.1
Updated from an older installed version or fresh install
Fresh install
List of activated apps
root@421b374e64f6:/var/www/nextcloud# sudo -u www-data php occ app:list
Cannot load Zend OPcache - it was already loaded
Enabled:
- activity: 2.19.0
- bookmarks: 13.1.3
- bruteforcesettings: 2.7.0
- calendar: 4.6.1
- circles: 27.0.1
- cloud_federation_api: 1.10.0
- comments: 1.17.0
- contacts: 5.5.0
- contactsinteraction: 1.8.0
- dashboard: 7.7.0
- dav: 1.27.0
- federatedfilesharing: 1.17.0
- federation: 1.17.0
- files: 1.22.0
- files_pdfviewer: 2.8.0
- files_reminders: 1.0.0
- files_rightclick: 1.6.0
- files_sharing: 1.19.0
- files_trashbin: 1.17.0
- files_versions: 1.20.0
- firstrunwizard: 2.16.0
- logreader: 2.12.0
- lookup_server_connector: 1.15.0
- nextcloud_announcements: 1.16.0
- notes: 4.9.1
- notifications: 2.15.0
- oauth2: 1.15.1
- password_policy: 1.17.0
- photos: 2.3.0
- privacy: 1.11.0
- provisioning_api: 1.17.0
- recommendations: 1.6.0
- related_resources: 1.2.0
- serverinfo: 1.17.0
- settings: 1.9.0
- sharebymail: 1.17.0
- support: 1.10.0
- survey_client: 1.15.0
- systemtags: 1.17.0
- tasks: 0.15.0
- text: 3.8.0
- theming: 2.2.0
- twofactor_backupcodes: 1.16.0
- updatenotification: 1.17.0
- user_status: 1.7.0
- viewer: 2.1.0
- weather_status: 1.7.0
- workflowengine: 2.9.0
Disabled:
- admin_audit: 1.17.0
- encryption: 2.15.0
- files_external: 1.19.0
- suspicious_login: 5.0.0
- twofactor_totp: 9.0.0
- user_ldap: 1.17.0
Nextcloud configuration
root@421b374e64f6:/var/www/nextcloud# sudo -u www-data php occ config:list system
Cannot load Zend OPcache - it was already loaded
{
"system": {
"overwrite.cli.url": "https:\/\/xxxxxxx.yyyyyyyy.zzzzzz:2814",
"instanceid": "***REMOVED SENSITIVE VALUE***",
"passwordsalt": "***REMOVED SENSITIVE VALUE***",
"secret": "***REMOVED SENSITIVE VALUE***",
"trusted_domains": [
"xxxxxxx.yyyyyyyy.zzzzzz:2814"
],
"datadirectory": "***REMOVED SENSITIVE VALUE***",
"dbtype": "sqlite3",
"version": "27.1.5.1",
"installed": true
}
}
Web server error log
[Sat Dec 23 23:34:07.042328 2023] [mpm_prefork:notice] [pid 31] AH00163: Apache/2.4.57 (Debian) OpenSSL/3.0.11 configured -- resuming normal operations
[Sat Dec 23 23:34:07.107336 2023] [core:notice] [pid 31] AH00094: Command line: '/usr/sbin/apache2'
Cannot load Zend OPcache - it was already loaded
[Sat Dec 23 23:41:24.625885 2023] [mpm_prefork:notice] [pid 31] AH00169: caught SIGTERM, shutting down
Cannot load Zend OPcache - it was already loaded
[Sat Dec 23 23:43:22.160472 2023] [mpm_prefork:notice] [pid 29] AH00163: Apache/2.4.57 (Debian) OpenSSL/3.0.11 configured -- resuming normal operations
[Sat Dec 23 23:43:22.188066 2023] [core:notice] [pid 29] AH00094: Command line: '/usr/sbin/apache2'
Cannot load Zend OPcache - it was already loaded
Cannot load Zend OPcache - it was already loaded
Log file
{"reqId":"R1UgtO11VYGdDXxNGm03","level":3,"time":"2023-12-27T16:57:17+00:00","remoteAddr":"192.168.20.240","user":"russel2Za","app":"webdav","method":"PUT","url":"/remote .php/dav/calendars/russel2Za/yearly/B3B79815-8189-44E1-AB37-37724915FF43.ics","message":"An exception occurred while executing a query: SQLSTATE[HY000]: General error: 5 database is locked","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:121.0) Gecko/20100101 Firefox/121.0","version":"27.1.5.1","exception":{"Exception":"OC\\DB\\ Exceptions\\DbalException","Message":"An exception occurred while executing a query: SQLSTATE[HY000]: General error: 5 database is locked","Code":5,"Trace":[{"file":"/var /www/nextcloud/lib/private/DB/QueryBuilder/QueryBuilder.php","line":328,"function":"wrap","class":"OC\\DB\\Exceptions\\DbalException","type":"::"},{"file":"/var/www/nextc loud/apps/dav/lib/CalDAV/CalDavBackend.php","line":1280,"function":"executeStatement","class":"OC\\DB\\QueryBuilder\\QueryBuilder","type":"->"},{"file":"/var/www/nextclou d/lib/public/AppFramework/Db/TTransactional.php","line":63,"function":"OCA\\DAV\\CalDAV\\{closure}","class":"OCA\\DAV\\CalDAV\\CalDavBackend","type":"->","args":["*** sen sitive parameters replaced ***"]},{"file":"/var/www/nextcloud/apps/dav/lib/CalDAV/CalDavBackend.php","line":1231,"function":"atomic","class":"OCA\\DAV\\CalDAV\\CalDavBack end","type":"->"},{"file":"/var/www/nextcloud/3rdparty/sabre/dav/lib/CalDAV/Calendar.php","line":199,"function":"createCalendarObject","class":"OCA\\DAV\\CalDAV\\CalDavBa ckend","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"/var/www/nextcloud/3rdparty/sabre/dav/lib/DAV/Server.php","line":1098,"function":"createFile ","class":"Sabre\\CalDAV\\Calendar","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"/var/www/nextcloud/3rdparty/sabre/dav/lib/DAV/CorePlugin.php"," line":504,"function":"createFile","class":"Sabre\\DAV\\Server","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"/var/www/nextcloud/3rdparty/sabre/ev ent/lib/WildcardEmitterTrait.php","line":89,"function":"httpPut","class":"Sabre\\DAV\\CorePlugin","type":"->"},{"file":"/var/www/nextcloud/3rdparty/sabre/dav/lib/DAV/Serv er.php","line":472,"function":"emit","class":"Sabre\\DAV\\Server","type":"->"},{"file":"/var/www/nextcloud/3rdparty/sabre/dav/lib/DAV/Server.php","line":253,"function":"i nvokeMethod","class":"Sabre\\DAV\\Server","type":"->"},{"file":"/var/www/nextcloud/3rdparty/sabre/dav/lib/DAV/Server.php","line":321,"function":"start","class":"Sabre\\DA V\\Server","type":"->"},{"file":"/var/www/nextcloud/apps/dav/lib/Server.php","line":365,"function":"exec","class":"Sabre\\DAV\\Server","type":"->"},{"file":"/var/www/next cloud/apps/dav/appinfo/v2/remote.php","line":35,"function":"exec","class":"OCA\\DAV\\Server","type":"->"},{"file":"/var/www/nextcloud/remote.php","line":172,"args":["/var /www/nextcloud/apps/dav/appinfo/v2/remote.php"],"function":"require_once"}],"File":"/var/www/nextcloud/lib/private/DB/Exceptions/DbalException.php","Line":71,"Previous":{ "Exception":"Doctrine\\DBAL\\Exception\\LockWaitTimeoutException","Message":"An exception occurred while executing a query: SQLSTATE[HY000]: General error: 5 database is locked","Code":5,"Trace":[{"file":"/var/www/nextcloud/3rdparty/doctrine/dbal/src/Connection.php","line":1814,"function":"convert","class":"Doctrine\\DBAL\\Driver\\API\\SQ Lite\\ExceptionConverter","type":"->"},{"file":"/var/www/nextcloud/3rdparty/doctrine/dbal/src/Connection.php","line":1749,"function":"handleDriverException","class":"Doct rine\\DBAL\\Connection","type":"->"},{"file":"/var/www/nextcloud/3rdparty/doctrine/dbal/src/Connection.php","line":1163,"function":"convertExceptionDuringQuery","class":" Doctrine\\DBAL\\Connection","type":"->"},{"file":"/var/www/nextcloud/lib/private/DB/Connection.php","line":295,"function":"executeStatement","class":"Doctrine\\DBAL\\Conn ection","type":"->"},{"file":"/var/www/nextcloud/3rdparty/doctrine/dbal/src/Query/QueryBuilder.php","line":354,"function":"executeStatement","class":"OC\\DB\\Connection", "type":"->"},{"file":"/var/www/nextcloud/lib/private/DB/QueryBuilder/QueryBuilder.php","line":280,"function":"execute","class":"Doctrine\\DBAL\\Query\\QueryBuilder","type ":"->"},{"file":"/var/www/nextcloud/lib/private/DB/QueryBuilder/QueryBuilder.php","line":326,"function":"execute","class":"OC\\DB\\QueryBuilder\\QueryBuilder","type":"->" },{"file":"/var/www/nextcloud/apps/dav/lib/CalDAV/CalDavBackend.php","line":1280,"function":"executeStatement","class":"OC\\DB\\QueryBuilder\\QueryBuilder","type":"->"},{ "file":"/var/www/nextcloud/lib/public/AppFramework/Db/TTransactional.php","line":63,"function":"OCA\\DAV\\CalDAV\\{closure}","class":"OCA\\DAV\\CalDAV\\CalDavBackend","ty pe":"->","args":["*** sensitive parameters replaced ***"]},{"file":"/var/www/nextcloud/apps/dav/lib/CalDAV/CalDavBackend.php","line":1231,"function":"atomic","class":"OCA \\DAV\\CalDAV\\CalDavBackend","type":"->"},{"file":"/var/www/nextcloud/3rdparty/sabre/dav/lib/CalDAV/Calendar.php","line":199,"function":"createCalendarObject","class":"O CA\\DAV\\CalDAV\\CalDavBackend","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"/var/www/nextcloud/3rdparty/sabre/dav/lib/DAV/Server.php","line":10 98,"function":"createFile","class":"Sabre\\CalDAV\\Calendar","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"/var/www/nextcloud/3rdparty/sabre/dav/ lib/DAV/CorePlugin.php","line":504,"function":"createFile","class":"Sabre\\DAV\\Server","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"/var/www/ne xtcloud/3rdparty/sabre/event/lib/WildcardEmitterTrait.php","line":89,"function":"httpPut","class":"Sabre\\DAV\\CorePlugin","type":"->"},{"file":"/var/www/nextcloud/3rdpar ty/sabre/dav/lib/DAV/Server.php","line":472,"function":"emit","class":"Sabre\\DAV\\Server","type":"->"},{"file":"/var/www/nextcloud/3rdparty/sabre/dav/lib/DAV/Server.php" ,"line":253,"function":"invokeMethod","class":"Sabre\\DAV\\Server","type":"->"},{"file":"/var/www/nextcloud/3rdparty/sabre/dav/lib/DAV/Server.php","line":321,"function":" start","class":"Sabre\\DAV\\Server","type":"->"},{"file":"/var/www/nextcloud/apps/dav/lib/Server.php","line":365,"function":"exec","class":"Sabre\\DAV\\Server","type":"-> "},{"file":"/var/www/nextcloud/apps/dav/appinfo/v2/remote.php","line":35,"function":"exec","class":"OCA\\DAV\\Server","type":"->"},{"file":"/var/www/nextcloud/remote.php" ,"line":172,"args":["/var/www/nextcloud/apps/dav/appinfo/v2/remote.php"],"function":"require_once"}],"File":"/var/www/nextcloud/3rdparty/doctrine/dbal/src/Driver/API/SQLi te/ExceptionConverter.php","Line":35,"Previous":{"Exception":"Doctrine\\DBAL\\Driver\\PDO\\Exception","Message":"SQLSTATE[HY000]: General error: 5 database is locked","Co de":5,"Trace":[{"file":"/var/www/nextcloud/3rdparty/doctrine/dbal/src/Driver/PDO/Statement.php","line":103,"function":"new","class":"Doctrine\\DBAL\\Driver\\PDO\\Exceptio n","type":"::"},{"file":"/var/www/nextcloud/3rdparty/doctrine/dbal/src/Connection.php","line":1153,"function":"execute","class":"Doctrine\\DBAL\\Driver\\PDO\\Statement"," type":"->"},{"file":"/var/www/nextcloud/lib/private/DB/Connection.php","line":295,"function":"executeStatement","class":"Doctrine\\DBAL\\Connection","type":"->"},{"file": "/var/www/nextcloud/3rdparty/doctrine/dbal/src/Query/QueryBuilder.php","line":354,"function":"executeStatement","class":"OC\\DB\\Connection","type":"->"},{"file":"/var/ww w/nextcloud/lib/private/DB/QueryBuilder/QueryBuilder.php","line":280,"function":"execute","class":"Doctrine\\DBAL\\Query\\QueryBuilder","type":"->"},{"file":"/var/www/nex tcloud/lib/private/DB/QueryBuilder/QueryBuilder.php","line":326,"function":"execute","class":"OC\\DB\\QueryBuilder\\QueryBuilder","type":"->"},{"file":"/var/www/nextcloud /apps/dav/lib/CalDAV/CalDavBackend.php","line":1280,"function":"executeStatement","class":"OC\\DB\\QueryBuilder\\QueryBuilder","type":"->"},{"file":"/var/www/nextcloud/li b/public/AppFramework/Db/TTransactional.php","line":63,"function":"OCA\\DAV\\CalDAV\\{closure}","class":"OCA\\DAV\\CalDAV\\CalDavBackend","type":"->","args":["*** sensiti ve parameters replaced ***"]},{"file":"/var/www/nextcloud/apps/dav/lib/CalDAV/CalDavBackend.php","line":1231,"function":"atomic","class":"OCA\\DAV\\CalDAV\\CalDavBackend" ,"type":"->"},{"file":"/var/www/nextcloud/3rdparty/sabre/dav/lib/CalDAV/Calendar.php","line":199,"function":"createCalendarObject","class":"OCA\\DAV\\CalDAV\\CalDavBacken d","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"/var/www/nextcloud/3rdparty/sabre/dav/lib/DAV/Server.php","line":1098,"function":"createFile","c lass":"Sabre\\CalDAV\\Calendar","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"/var/www/nextcloud/3rdparty/sabre/dav/lib/DAV/CorePlugin.php","line ":504,"function":"createFile","class":"Sabre\\DAV\\Server","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"/var/www/nextcloud/3rdparty/sabre/event/ lib/WildcardEmitterTrait.php","line":89,"function":"httpPut","class":"Sabre\\DAV\\CorePlugin","type":"->"},{"file":"/var/www/nextcloud/3rdparty/sabre/dav/lib/DAV/Server.p hp","line":472,"function":"emit","class":"Sabre\\DAV\\Server","type":"->"},{"file":"/var/www/nextcloud/3rdparty/sabre/dav/lib/DAV/Server.php","line":253,"function":"invok eMethod","class":"Sabre\\DAV\\Server","type":"->"},{"file":"/var/www/nextcloud/3rdparty/sabre/dav/lib/DAV/Server.php","line":321,"function":"start","class":"Sabre\\DAV\\S erver","type":"->"},{"file":"/var/www/nextcloud/apps/dav/lib/Server.php","line":365,"function":"exec","class":"Sabre\\DAV\\Server","type":"->"},{"file":"/var/www/nextclou d/apps/dav/appinfo/v2/remote.php","line":35,"function":"exec","class":"OCA\\DAV\\Server","type":"->"},{"file":"/var/www/nextcloud/remote.php","line":172,"args":["/var/www /nextcloud/apps/dav/appinfo/v2/remote.php"],"function":"require_once"}],"File":"/var/www/nextcloud/3rdparty/doctrine/dbal/src/Driver/PDO/Exception.php","Line":30,"Previou s":{"Exception":"PDOException","Message":"SQLSTATE[HY000]: General error: 5 database is locked","Code":"HY000","Trace":[{"file":"/var/www/nextcloud/3rdparty/doctrine/dbal /src/Driver/PDO/Statement.php","line":101,"function":"execute","class":"PDOStatement","type":"->"},{"file":"/var/www/nextcloud/3rdparty/doctrine/dbal/src/Connection.php", "line":1153,"function":"execute","class":"Doctrine\\DBAL\\Driver\\PDO\\Statement","type":"->"},{"file":"/var/www/nextcloud/lib/private/DB/Connection.php","line":295,"func tion":"executeStatement","class":"Doctrine\\DBAL\\Connection","type":"->"},{"file":"/var/www/nextcloud/3rdparty/doctrine/dbal/src/Query/QueryBuilder.php","line":354,"func tion":"executeStatement","class":"OC\\DB\\Connection","type":"->"},{"file":"/var/www/nextcloud/lib/private/DB/QueryBuilder/QueryBuilder.php","line":280,"function":"execut e","class":"Doctrine\\DBAL\\Query\\QueryBuilder","type":"->"},{"file":"/var/www/nextcloud/lib/private/DB/QueryBuilder/QueryBuilder.php","line":326,"function":"execute","c lass":"OC\\DB\\QueryBuilder\\QueryBuilder","type":"->"},{"file":"/var/www/nextcloud/apps/dav/lib/CalDAV/CalDavBackend.php","line":1280,"function":"executeStatement","clas s":"OC\\DB\\QueryBuilder\\QueryBuilder","type":"->"},{"file":"/var/www/nextcloud/lib/public/AppFramework/Db/TTransactional.php","line":63,"function":"OCA\\DAV\\CalDAV\\{c losure}","class":"OCA\\DAV\\CalDAV\\CalDavBackend","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"/var/www/nextcloud/apps/dav/lib/CalDAV/CalDavBac kend.php","line":1231,"function":"atomic","class":"OCA\\DAV\\CalDAV\\CalDavBackend","type":"->"},{"file":"/var/www/nextcloud/3rdparty/sabre/dav/lib/CalDAV/Calendar.php"," line":199,"function":"createCalendarObject","class":"OCA\\DAV\\CalDAV\\CalDavBackend","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"/var/www/next cloud/3rdparty/sabre/dav/lib/DAV/Server.php","line":1098,"function":"createFile","class":"Sabre\\CalDAV\\Calendar","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"/var/www/nextcloud/3rdparty/sabre/dav/lib/DAV/CorePlugin.php","line":504,"function":"createFile","class":"Sabre\\DAV\\Server","type":"->","args":["*** sen sitive parameters replaced ***"]},{"file":"/var/www/nextcloud/3rdparty/sabre/event/lib/WildcardEmitterTrait.php","line":89,"function":"httpPut","class":"Sabre\\DAV\\CoreP lugin","type":"->"},{"file":"/var/www/nextcloud/3rdparty/sabre/dav/lib/DAV/Server.php","line":472,"function":"emit","class":"Sabre\\DAV\\Server","type":"->"},{"file":"/va r/www/nextcloud/3rdparty/sabre/dav/lib/DAV/Server.php","line":253,"function":"invokeMethod","class":"Sabre\\DAV\\Server","type":"->"},{"file":"/var/www/nextcloud/3rdparty /sabre/dav/lib/DAV/Server.php","line":321,"function":"start","class":"Sabre\\DAV\\Server","type":"->"},{"file":"/var/www/nextcloud/apps/dav/lib/Server.php","line":365,"fu nction":"exec","class":"Sabre\\DAV\\Server","type":"->"},{"file":"/var/www/nextcloud/apps/dav/appinfo/v2/remote.php","line":35,"function":"exec","class":"OCA\\DAV\\Server ","type":"->"},{"file":"/var/www/nextcloud/remote.php","line":172,"args":["/var/www/nextcloud/apps/dav/appinfo/v2/remote.php"],"function":"require_once"}],"File":"/var/ww w/nextcloud/3rdparty/doctrine/dbal/src/Driver/PDO/Statement.php","Line":101}}},"message":"An exception occurred while executing a query: SQLSTATE[HY000]: General error: 5 database is locked","exception":{},"CustomMessage":"An exception occurred while executing a query: SQLSTATE[HY000]: General error: 5 database is locked"}}
Browser log
No response
Additional info
the log file is massive, so tried to get one of the database is locked examples, if you need more we can arrange a private transfer if you would like. but as this is easily reproducible and to me looks like a race condition as the server is a slow rpi runing sqlite, maybe the exception should be caught and if locked, a delay and retry a few times then hard exception raised? but I am not a php programmer so couldn't really work out quite what was happening, but I have seen such issues in java code before :-)