/sync hangs on synapse 0.33.3 with Twisted 16.6

[richvdh]

we've had a number of reports of people's clients hanging after upgrading to synapse 0.33.3.

It appears that the body of the response to /sync is not being correctly sent. It is limited to requests with large responses (more than, say, 64K, hence it affecting /sync), and only manifests itself when the client asks for a compressed body.

It also only happens on Twisted versions before 17.1.

[richvdh]

I think it may have been incidentally fixed by twisted/twisted#591 (certainly that seems to have rewritten a bunch of this code).

[richvdh]

For those who are being bitten by this bug: your best bet for now is probably to downgrade.

Note that you will need to manually carry out some database manipulation to downgrade.

The debs for 0.33.2 are still in the repo at https://matrix.org/packages/debian/pool/main/m/matrix-synapse/matrix-synapse_0.33.2-1_all.deb; download it and install with dpkg -i matrix-synapse_0.33.2-1_all.deb.

Before 0,33.2 will start, you will need to roll back the database schema upgrade that will have been done when you upgraded. This can be done by obtaining a sql shell on your database and:

update schema_version set version=50;
delete from applied_schema_deltas where version=51;
drop table monthly_active_users;

[richvdh]

FWIW the incompatibility with Twisted 16.6 was introduced by #3701.

[xsteadfastx]

i think i run into this with my own written bot with matrix-client in python. since 0.33.3 i get broken json:

> Aug 23 09:11:54 foobar pipenv[4205]:     Traceback (most recent call last):
> Aug 23 09:11:54 foobar pipenv[4205]:       File "mossbot.py", line 481, in connect
> Aug 23 09:11:54 foobar pipenv[4205]:         self.password
> Aug 23 09:11:54 foobar pipenv[4205]:       File "/opt/mossbot/.local/share/virtualenvs/mossbot-B1iKwa68/lib/python3.6/site-packages/matrix_client/client.py", line 183, in login_with_password
> Aug 23 09:11:54 foobar pipenv[4205]:         self._sync()
> Aug 23 09:11:54 foobar pipenv[4205]:       File "/opt/mossbot/.local/share/virtualenvs/mossbot-B1iKwa68/lib/python3.6/site-packages/matrix_client/client.py", line 426, in _sync
> Aug 23 09:11:54 foobar pipenv[4205]:         response = self.api.sync(self.sync_token, timeout_ms, filter=self.sync_filter)
> Aug 23 09:11:54 foobar pipenv[4205]:       File "/opt/mossbot/.local/share/virtualenvs/mossbot-B1iKwa68/lib/python3.6/site-packages/matrix_client/api.py", line 92, in sync
> Aug 23 09:11:54 foobar pipenv[4205]:         api_path=MATRIX_V2_API_PATH)
> Aug 23 09:11:54 foobar pipenv[4205]:       File "/opt/mossbot/.local/share/virtualenvs/mossbot-B1iKwa68/lib/python3.6/site-packages/matrix_client/api.py", line 526, in _send
> Aug 23 09:11:54 foobar pipenv[4205]:         verify=self.validate_cert
> Aug 23 09:11:54 foobar pipenv[4205]:       File "/opt/mossbot/.local/share/virtualenvs/mossbot-B1iKwa68/lib/python3.6/site-packages/requests/api.py", line 58, in request
> Aug 23 09:11:54 foobar pipenv[4205]:         return session.request(method=method, url=url, **kwargs)
> Aug 23 09:11:54 foobar pipenv[4205]:       File "/opt/mossbot/.local/share/virtualenvs/mossbot-B1iKwa68/lib/python3.6/site-packages/requests/sessions.py", line 508, in request
> Aug 23 09:11:54 foobar pipenv[4205]:         resp = self.send(prep, **send_kwargs)
> Aug 23 09:11:54 foobar pipenv[4205]:       File "/opt/mossbot/.local/share/virtualenvs/mossbot-B1iKwa68/lib/python3.6/site-packages/requests/sessions.py", line 658, in send
> Aug 23 09:11:54 foobar pipenv[4205]:         r.content
> Aug 23 09:11:54 foobar pipenv[4205]:       File "/opt/mossbot/.local/share/virtualenvs/mossbot-B1iKwa68/lib/python3.6/site-packages/requests/models.py", line 823, in content
> Aug 23 09:11:54 foobar pipenv[4205]:         self._content = bytes().join(self.iter_content(CONTENT_CHUNK_SIZE)) or bytes()
> Aug 23 09:11:54 foobar pipenv[4205]:       File "/opt/mossbot/.local/share/virtualenvs/mossbot-B1iKwa68/lib/python3.6/site-packages/requests/models.py", line 748, in generate
> Aug 23 09:11:54 foobar pipenv[4205]:         raise ChunkedEncodingError(e)
> Aug 23 09:11:54 foobar pipenv[4205]:     requests.exceptions.ChunkedEncodingError: ('Connection broken: IncompleteRead(0 bytes read)', IncompleteRead(0 bytes read))

i would downgrade but this sql fiddling is a little bit too high for me.

[xsteadfastx]

another workaround for me:

set compress: false in homeserver.yaml

  - port: 8008
    tls: false
    bind_address: ''
    type: http

    x_forwarded: true

    resources:
      - names: [client, webclient]
        #compress: true
        compress: false
      - names: [federation]
        compress: false

[andrewshadura]

Actually, since @richvdh refreshed the repos, the proper way to install the old version is now to do: apt install matrix-synapse=0.33.2-1

[Half-Shot]

Seems like we would need to enforce compression: false for older twisted versions ?

[xsteadfastx]

@andrewshadura but this doesnt do the sql rollback or?

[richvdh]

@andrewshadura but this doesnt do the sql rollback or?

correct, it just replaces the downlad and dpkg -i

[richvdh]

ok so the conclusion about what we are going to do here is:

• we are going to ship backported packages of more recent Twisted for Debian Stretch, and Ubuntu Xenial and Artful (Buster and Bionic have recent enough Twisteds already)
• we are going to drop support for Debian Jessie. Sorry to anyone who is still using it, but it's out of support from Debian and you really ought to upgrade...

[richvdh]

In the meantime, @xsteadfastx 's workaround seems like a sensible alternative to downgrading.

[andrewshadura]

I think support for jessie needs to be dropped.

[4nd3r]

we are going to drop support for Debian Jessie. Sorry to anyone who is still using it,
but it's out of support from Debian and you really ought to upgrade...

wrong. debian jessie is supported until 2020: https://wiki.debian.org/LTS

[ptman]

@4nd3r Are all packages supported? Does it make sense for matrix to support Debian jessie? Would jessie be fine with using docker to run matrix?