synapse crash

[eggy]

My server ran out of space, I have expanded the disk, but now I'm running into this in my logs and I'm unable to sign in

2020-09-19 16:15:44,203 - synapse.metrics.background_process_metrics - 210 - ERROR - user_directory.notify_new_event-5- Background process 'user_directory.notify_new_event' threw an exception
Traceback (most recent call last):
File "/home/eggy/synapse/env/lib/python3.7/site-packages/synapse/metrics/background_process_metrics.py", line 205, in run
result = await result
File "/home/eggy/synapse/env/lib/python3.7/site-packages/synapse/handlers/user_directory.py", line 103, in process
await self._unsafe_process()
File "/home/eggy/synapse/env/lib/python3.7/site-packages/synapse/handlers/user_directory.py", line 154, in _unsafe_process
await self._handle_deltas(deltas)
File "/home/eggy/synapse/env/lib/python3.7/site-packages/synapse/handlers/user_directory.py", line 217, in _handle_deltas
state_key, room_id, prev_event_id, event_id
File "/home/eggy/synapse/env/lib/python3.7/site-packages/synapse/handlers/user_directory.py", line 396, in _handle_profile_change
await self.store.update_profile_in_user_dir(user_id, new_name, new_avatar)
File "/home/eggy/synapse/env/lib/python3.7/site-packages/twisted/internet/defer.py", line 1416, in _inlineCallbacks
result = result.throwExceptionIntoGenerator(g)
File "/home/eggy/synapse/env/lib/python3.7/site-packages/twisted/python/failure.py", line 512, in throwExceptionIntoGenerator
return g.throw(self.type, self.value, self.tb)
File "/home/eggy/synapse/env/lib/python3.7/site-packages/synapse/storage/database.py", line 527, in runInteraction
**kwargs
File "/home/eggy/synapse/env/lib/python3.7/site-packages/twisted/internet/defer.py", line 1416, in _inlineCallbacks
result = result.throwExceptionIntoGenerator(g)
File "/home/eggy/synapse/env/lib/python3.7/site-packages/twisted/python/failure.py", line 512, in throwExceptionIntoGenerator
return g.throw(self.type, self.value, self.tb)
File "/home/eggy/synapse/env/lib/python3.7/site-packages/synapse/storage/database.py", line 575, in runWithConnection
self._db_pool.runWithConnection(inner_func, *args, **kwargs)
File "/home/eggy/synapse/env/lib/python3.7/site-packages/twisted/python/threadpool.py", line 250, in inContext
result = inContext.theWork()
File "/home/eggy/synapse/env/lib/python3.7/site-packages/twisted/python/threadpool.py", line 266, in
inContext.theWork = lambda: context.call(ctx, func, *args, **kw)
File "/home/eggy/synapse/env/lib/python3.7/site-packages/twisted/python/context.py", line 122, in callWithContext
return self.currentContext().callWithContext(ctx, func, *args, **kw)
File "/home/eggy/synapse/env/lib/python3.7/site-packages/twisted/python/context.py", line 85, in callWithContext
return func(*args,**kw)
File "/home/eggy/synapse/env/lib/python3.7/site-packages/twisted/enterprise/adbapi.py", line 306, in _runWithConnection
compat.reraise(excValue, excTraceback)
File "/home/eggy/synapse/env/lib/python3.7/site-packages/twisted/python/compat.py", line 464, in reraise
raise exception.with_traceback(traceback)
File "/home/eggy/synapse/env/lib/python3.7/site-packages/twisted/enterprise/adbapi.py", line 297, in _runWithConnection
result = func(conn, *args, **kw)
File "/home/eggy/synapse/env/lib/python3.7/site-packages/synapse/storage/database.py", line 572, in inner_func
return func(conn, *args, **kwargs)
File "/home/eggy/synapse/env/lib/python3.7/site-packages/synapse/storage/database.py", line 416, in new_transaction
r = func(cursor, *args, **kwargs)
File "/home/eggy/synapse/env/lib/python3.7/site-packages/synapse/storage/databases/main/user_directory.py", line 399, in _update_profile_in_user_dir_txn
display_name,
File "/home/eggy/synapse/env/lib/python3.7/site-packages/synapse/storage/database.py", line 208, in execute
self._do_execute(self.txn.execute, sql, *args)
File "/home/eggy/synapse/env/lib/python3.7/site-packages/synapse/storage/database.py", line 234, in _do_execute
return func(sql, *args)
psycopg2.errors.InvalidTextRepresentation: malformed array literal: ""
LINE 1: ...t(to_tsvector('english', COALESCE(ARRAY['tulir'], '')), 'B')...
^
DETAIL: Array value must start with "{" or dimension information.

2020-09-19 16:15:44,538 - synapse.metrics.background_process_metrics - 210 - ERROR - user_directory.notify_new_event-6- Background process 'user_directory.notify_new_event' threw an exception
Traceback (most recent call last):
File "/home/eggy/synapse/env/lib/python3.7/site-packages/synapse/metrics/background_process_metrics.py", line 205, in run
result = await result
File "/home/eggy/synapse/env/lib/python3.7/site-packages/synapse/handlers/user_directory.py", line 103, in process
await self._unsafe_process()
File "/home/eggy/synapse/env/lib/python3.7/site-packages/synapse/handlers/user_directory.py", line 154, in _unsafe_process
await self._handle_deltas(deltas)
File "/home/eggy/synapse/env/lib/python3.7/site-packages/synapse/handlers/user_directory.py", line 217, in _handle_deltas
state_key, room_id, prev_event_id, event_id
File "/home/eggy/synapse/env/lib/python3.7/site-packages/synapse/handlers/user_directory.py", line 396, in _handle_profile_change
await self.store.update_profile_in_user_dir(user_id, new_name, new_avatar)
File "/home/eggy/synapse/env/lib/python3.7/site-packages/twisted/internet/defer.py", line 1416, in _inlineCallbacks
result = result.throwExceptionIntoGenerator(g)
File "/home/eggy/synapse/env/lib/python3.7/site-packages/twisted/python/failure.py", line 512, in throwExceptionIntoGenerator
return g.throw(self.type, self.value, self.tb)
File "/home/eggy/synapse/env/lib/python3.7/site-packages/synapse/storage/database.py", line 527, in runInteraction
**kwargs
File "/home/eggy/synapse/env/lib/python3.7/site-packages/twisted/internet/defer.py", line 1416, in _inlineCallbacks
result = result.throwExceptionIntoGenerator(g)
File "/home/eggy/synapse/env/lib/python3.7/site-packages/twisted/python/failure.py", line 512, in throwExceptionIntoGenerator
return g.throw(self.type, self.value, self.tb)
File "/home/eggy/synapse/env/lib/python3.7/site-packages/synapse/storage/database.py", line 575, in runWithConnection
self._db_pool.runWithConnection(inner_func, *args, **kwargs)
File "/home/eggy/synapse/env/lib/python3.7/site-packages/twisted/python/threadpool.py", line 250, in inContext
result = inContext.theWork()
File "/home/eggy/synapse/env/lib/python3.7/site-packages/twisted/python/threadpool.py", line 266, in
inContext.theWork = lambda: context.call(ctx, func, *args, **kw)
File "/home/eggy/synapse/env/lib/python3.7/site-packages/twisted/python/context.py", line 122, in callWithContext
return self.currentContext().callWithContext(ctx, func, *args, **kw)
File "/home/eggy/synapse/env/lib/python3.7/site-packages/twisted/python/context.py", line 85, in callWithContext
return func(*args,**kw)
File "/home/eggy/synapse/env/lib/python3.7/site-packages/twisted/enterprise/adbapi.py", line 306, in _runWithConnection
compat.reraise(excValue, excTraceback)
File "/home/eggy/synapse/env/lib/python3.7/site-packages/twisted/python/compat.py", line 464, in reraise
raise exception.with_traceback(traceback)
File "/home/eggy/synapse/env/lib/python3.7/site-packages/twisted/enterprise/adbapi.py", line 297, in _runWithConnection
result = func(conn, *args, **kw)
File "/home/eggy/synapse/env/lib/python3.7/site-packages/synapse/storage/database.py", line 572, in inner_func
return func(conn, *args, **kwargs)
File "/home/eggy/synapse/env/lib/python3.7/site-packages/synapse/storage/database.py", line 416, in new_transaction
r = func(cursor, *args, **kwargs)
File "/home/eggy/synapse/env/lib/python3.7/site-packages/synapse/storage/databases/main/user_directory.py", line 399, in _update_profile_in_user_dir_txn
display_name,
File "/home/eggy/synapse/env/lib/python3.7/site-packages/synapse/storage/database.py", line 208, in execute
self._do_execute(self.txn.execute, sql, *args)
File "/home/eggy/synapse/env/lib/python3.7/site-packages/synapse/storage/database.py", line 234, in _do_execute
return func(sql, *args)
psycopg2.errors.InvalidTextRepresentation: malformed array literal: ""
LINE 1: ...t(to_tsvector('english', COALESCE(ARRAY['tulir'], '')), 'B')...

I'm on synapse 1.19.3, with a very large database at this point (over 90GB).

[anoadragon453]

Hmm, that COALESCE should not be getting an array, but rather a string literal. The fact COALESCE is handed two different types seems to be what's causing the exception.

Looking at the relevant code, it's not expected that display_name would be an array here:

synapse/synapse/storage/databases/main/user_directory.py

Lines 393 to 409 in 8a4a418

	sql = """
	INSERT INTO user_directory_search(user_id, vector)
	VALUES (?,
	setweight(to_tsvector('english', ?), 'A')
	|| setweight(to_tsvector('english', ?), 'D')
	|| setweight(to_tsvector('english', COALESCE(?, '')), 'B')
	) ON CONFLICT (user_id) DO UPDATE SET vector=EXCLUDED.vector
	"""
	txn.execute(
	sql,
	(
	user_id,
	get_localpart_from_id(user_id),
	get_domain_from_id(user_id),
	display_name,
	),
	)

#8223 should hopefully have fixed this, as it the code will now bail out if the displayname is not a string:

synapse/synapse/handlers/user_directory.py

Lines 392 to 393 in 8a4a418

	if not isinstance(new_name, str):
	new_name = prev_name

The fix will be released in v1.20.0. Feel free to update to a v1.20.0 RC if you'd like to check whether your issue is fixed before the release.

Closing this for now. Please reopen if you continue to experience this even after updating.

[clokep]

I think this is a duplicate of #8220.