Unable to connect to homeserver, retrying... - Lost access to homeserver on mobile, web, and desktop apps after logout

[johndball]

Description

Steps to reproduce

This was originally posted in Element Web, but I later discovered this is an issue with Matrix Synapse server. Due to my original post being closed under Elemenet, I am moving this to Synapse.

1. I logged out and am unable to log back in. On my mobile device, I am prompted with a "verify session" prompt, but the client never loads the content.
2. This behavior has started recently after a user logs out of a client on a self-run homeserver. This issue is experienced on the Element Windows desktop app 1.10.13, app.element.io on the web, and Element mobile for iOS and Android.
3. Web versions of Element show a sync job that never ends. I opened developer view on the web and I can see "Number of consecutive failed sync requests: 36" and other issues related to syncing.
4. Clearing the cache on devices will log out and not log back in. In my attempts to regain access to my home server, I cleared cache on mobile and desktop which effectively lost access to the homeserver.
5. Running Matrix Synapse 1.59.1, the user will log out and then is unable to log back in. A rollback to Matrix Synapse version 1.57.0 allows users to log back in and the sync error is gone.

Version information

• Homeserver:
  We are running two custom homeservers - a production version and a federated/development homeserver.

• Version:
  1.59.1 - buggy - users unable to log in after clearing cache or logging out
  1.58.1 - buggy, but some users can log in - the users that can log in have joined 2 or fewer rooms. The users that cannot log in have 3 or more rooms. My account is 15 rooms deep.
  1.57.0 - works fine. All users who previously could not log in can log in.
  1.56.0 - unable to downgrade to this version due to database constraints

• Install method:

Installed via repo/package manager

• Platform:
  Ubuntu 20.04 LTS

• Others:
  One homeserver uses an apache proxy server for communications, the other has no proxy. Both homeservers sit behind Cloudflare. One is on-prem and the other is third-party cloud provider. Both homeservers are federated.
  I do not have verbose logging enabled, but can re-enable if needed.

• Screenshot from Element Web, Element Desktop, and Element Mobile:
  
  
  

[squahtx]

Possibly the same as #12864 but we would very much like Synapse logs to confirm.
Are you able to provide INFO-level logs covering the time period of a failed login?

[johndball]

Absolutely. Bear with me, but do I set the log level in homeserver.yaml and which logs (homeserver.log?) are you looking for?
If there is sensitive information in the log (specifically homeserver URL, etc.) can I DM?

Edit: Looks like I had database set to INFO, but root set to ERROR. I set root to INFO under log.yaml. Just advise on sensitive information and if homeserver.log is the log being requested. Thanks!

[squahtx]

log_config in homeserver.yaml will point to the log config file. Inside the log config file, there'll be a line like:

root:
    level: INFO

which specifies the log level. It's INFO by default, so if you've not changed it, you probably don't need to do anything there.

It's homeserver.log we're after, unless you've got workers set up.

If you want to DM the logs, I'm reachable at @squah:matrix.org.

[johndball]

Didn't take long at all. Upgraded from 1.57.0 to 1.59.1 and it immediately went haywire once I cleared cache and reloaded.

Downgraded and the clients can sign in.

dpkg: warning: downgrading matrix-synapse-py3 from 1.59.1+focal1 to 1.57.0+focal1
(Reading database ... 169057 files and directories currently installed.)
Preparing to unpack matrix-synapse-py3_1.57.0+focal1_amd64.deb ...
Unpacking matrix-synapse-py3 (1.57.0+focal1) over (1.59.1+focal1) ...
Setting up matrix-synapse-py3 (1.57.0+focal1) ...

I will DM you a shared folder to download the logs from both Firefox Element web and homeserver.log in INFO mode.

[johndball]

Hi @squahtx - just wanted to check in and ensure that the logs I uploaded were sufficient. Please let me know if additional details are required and I will be happy to support. The link for the log download will expire in 7 days. Thanks.

[squahtx]

I've received them, thank you! I'm tied up with other things and haven't had the time to look at them yet.

In the mean time, could you try running the following SQL queries from @richvdh in #12864:

SELECT * FROM events e LEFT JOIN rejections r USING (event_id) WHERE event_id IN ('$KuawYhzU8G0qk7zUX2jEI_0PQH1yFnj-2CiiFF4K2UM', '$bEbwpl46iHvYccVAwpkLp6-qzxdeCCmiQTxbo1MObAQ');

SELECT event_id, internal_metadata FROM event_json WHERE event_id IN ('$KuawYhzU8G0qk7zUX2jEI_0PQH1yFnj-2CiiFF4K2UM', '$bEbwpl46iHvYccVAwpkLp6-qzxdeCCmiQTxbo1MObAQ'); 

[johndball]

I attempted but continue to receive a query error due to unexpected syntax. Sorry, PostgreSQL is not my cup of tea.

[squahtx]

Do you mind posting the query error you're seeing?

[johndball]

Do you mind posting the query error you're seeing?

The query is not executing. It is most likely my lack of knowledge of PostgreSQL.

[MRAAGH]

Me and other users are experiencing this issue on both my homeservers (mazie.rocks running synapse 1.58.1 and aagrinder.xyz running synapse 1.59.1). Downgrading synapse 1.59.1 to 1.57.0 did NOT fix it. I tried different clients:

• Element desktop 1.10.13: stuck on infinite sync; upon restarting it says "unable to connect to homeserver. retrying"
• Element web 1.10.13 and 1.10.10: same as above
• Fluffychat 1.4.0 web: "No connection to the server"
• Fluffychat 1.4.0 on Android: able to log in normally
• Moment (using matrix-nio): able to log in normally
• Fractal on Ubuntu: able to log in normally
• NeoChat on Ubuntu: able to log in normally

In all cases, clients that were already logged in, work perfectly fine. The issue only appears at login.

[squahtx]

It's highly likely this is the same issue as #12864, which richvdh has a proposed fix for.

Are either of you willing to run a bleeding edge development branch of Synapse to test if it fixes the issue?:
#12905
https://github.com/matrix-org/synapse/tree/rav/fix_sync_404