Synchronization issues with suprsync and E2E testing

[kmharrington]

Every couple hours / maybe once a day the suprsync agent is crashing on the E2E system because of a locked database issue. This tells me there are two writes open at the same time. Right at this moment it's useful for testing finalization down the data flow line, but still probably something to fix.

2023-07-26T02-50-08.457600 run:3 CRASH: [Failure instance: Traceback: <class 'sqlalchemy.exc.OperationalError'>: (sqlite3.OperationalError) database is locked
[SQL: SELECT supersync_v0.id AS supersync_v0_id, supersync_v0.local_path AS supersync_v0_local_path, supersync_v0.local_md5sum AS supersync_v0_local_md5sum, supersync_v0.archive_name AS supersync_v0_archive_name, supersync_v0.remote_path AS supersync_v0_remote_path, supersync_v0.timestamp AS supersync_v0_timestamp, supersync_v0.remote_md5sum AS supersync_v0_remote_md5sum, supersync_v0.copied AS supersync_v0_copied, supersync_v0.removed AS supersync_v0_removed, supersync_v0.failed_copy_attempts AS supersync_v0_failed_copy_attempts, supersync_v0.deletable AS supersync_v0_deletable, supersync_v0."ignore" AS supersync_v0_ignore
FROM supersync_v0
WHERE supersync_v0.removed IS NULL AND supersync_v0.archive_name = ? AND supersync_v0.failed_copy_attempts < ?]
[parameters: ('smurf', 5)]
(Background on this error at: https://sqlalche.me/e/14/e3q8)
/usr/lib/python3.8/threading.py:932:_bootstrap_inner
/usr/lib/python3.8/threading.py:870:run
/home/ocs/.venv/lib/python3.8/site-packages/twisted/_threads/_threadworker.py:47:work
/home/ocs/.venv/lib/python3.8/site-packages/twisted/_threads/_team.py:182:doWork
--- <exception caught here> ---
/home/ocs/.venv/lib/python3.8/site-packages/twisted/python/threadpool.py:244:inContext
/home/ocs/.venv/lib/python3.8/site-packages/twisted/python/threadpool.py:260:<lambda>
/home/ocs/.venv/lib/python3.8/site-packages/twisted/python/context.py:117:callWithContext
/home/ocs/.venv/lib/python3.8/site-packages/twisted/python/context.py:82:callWithContext
/home/ocs/.venv/lib/python3.8/site-packages/socs/agents/suprsync/agent.py:150:run
/home/ocs/.venv/lib/python3.8/site-packages/socs/db/suprsync.py:595:copy_files
/home/ocs/.venv/lib/python3.8/site-packages/socs/db/suprsync.py:351:get_copyable_files
/home/ocs/.venv/lib/python3.8/site-packages/sqlalchemy/orm/query.py:2759:all
/home/ocs/.venv/lib/python3.8/site-packages/sqlalchemy/orm/query.py:2894:_iter
/home/ocs/.venv/lib/python3.8/site-packages/sqlalchemy/orm/session.py:1692:execute
/home/ocs/.venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py:1614:_execute_20
/home/ocs/.venv/lib/python3.8/site-packages/sqlalchemy/sql/elements.py:325:_execute_on_connection
/home/ocs/.venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py:1481:_execute_clauseelement
/home/ocs/.venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py:1845:_execute_context
/home/ocs/.venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py:2026:_handle_dbapi_exception
/home/ocs/.venv/lib/python3.8/site-packages/sqlalchemy/util/compat.py:207:raise_
/home/ocs/.venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py:1802:_execute_context
/home/ocs/.venv/lib/python3.8/site-packages/sqlalchemy/engine/default.py:732:do_execute
]
2023-07-26T02-50-08.460865 run:3 Status is now "done".