Twisted has a new version which causes failure

[phreed]

Description

I just started getting this error.

This is due to an update to Twisted from v24.7.0 to v24.10.0 .
Forcing the older version pip install --force-reinstall -v "Twisted==24.7.0" and that corrected the error.
Apparently, _canonicalHeaderName = Headers()._encodeName is no longer valid.

Steps to reproduce

mkdir ~/synapse
python3.12 -m venv ~/synapse/env
source ~/synapse/env/bin/activate

pip install --upgrade pip
pip install --upgrade setuptools
pip install matrix-synapse
cd  ~/synapse
python -m synapse.app.homeserver --server-name matrix.mcp.mil --config-path homeserver.yaml --generate-config

Homeserver

generating a new homeserver.yaml

Synapse Version

1.117.0

Installation Method

pip (from PyPI)

Database

sqlite

Workers

Single process

Platform

Ubuntu 24
fedora 40

Configuration

No response

Relevant log output

Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
 ...
    from synapse.http.site import SynapseRequest
  File "/var/home/me/synapse/env/lib64/python3.12/site-packages/synapse/http/site.py", line 39, in <module>
    from synapse.http.proxy import ProxySite
  File "/var/home/me/synapse/env/lib64/python3.12/site-packages/synapse/http/proxy.py", line 72, in <module>
    _canonicalHeaderName = Headers()._encodeName
                           ^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'Headers' object has no attribute '_encodeName'

Anything else that would be useful to know?

I updated the synapse/http/proxy.py about line 74 to gain some insight.
While this is not a fix, it may be useful.

 if hasattr(Headers, "_canonicalNameCaps"):
    logger.warning(f'you appear to be running an old version of Twisted < 24.7.0')
    _canonicalHeaderName = Headers()._canonicalNameCaps  # type: ignore[attr-defined]
elif hasattr(Headers, "_encodeName"):
    logger.warning(f'you appear to be running an old version of Twisted 24.7.0')
    # But note that `_encodeName` still exists on prior versions,
    # it just encodes differently
    _canonicalHeaderName = Headers()._encodeName
else:
    logger.warning(f'you appear to be at an unsupported version of Twisted')
    _canonicalHeaderName = 'foo.name'

[phreed]

As you can see _encodeName was part of the private API.
https://docs.twisted.org/en/twisted-24.7.0/api/twisted.web.http_headers.Headers.html?private=1

It is removed in the 24.10.0
https://docs.twisted.org/en/twisted-24.10.0/api/twisted.web.http_headers.Headers.html?private=1

I think the intent is to use the str function instead.

[villepeh]

Can confirm the issue. I downgraded Twisted and Synapse works again.

[dillfrescott]

Running into this issue as well, prevents synapse from starting. (Synapse 1.118.0)

[dillfrescott]

Can confirm downgrading to the last version of Twisted remedies the issue

[Josue-T]

Same issue on my side, downgrade also fixed the issue.

[arkamar]

It is currently also related to #17075.

[Midar]

Twisted just got updated on my system and it broke my server. I cannot downgrade Twisted easily as it's coming from the system and not pip. Please fix this urgently (some people will not be able to bring their Matrix server back online until you do a new release with this!) and never use private functions in the future! They're private for a reason (because using them results in breaking).

[Midar]

Ok, this is kinda ridiculous: This method didn't do anything complex, there really was no justification for ever using this private method.

Workaround: Replace this code:

if hasattr(Headers, "_canonicalNameCaps"):                                      
    # Twisted < 24.7.0rc1                                                       
    _canonicalHeaderName = Headers()._canonicalNameCaps  # type: ignore[attr-defined]                                                                           
else:                                                                           
    # Twisted >= 24.7.0rc1                                                      
    # But note that `_encodeName` still exists on prior versions,               
    # it just encodes differently                                               
    _canonicalHeaderName = Headers()._encodeName 

With this:

def _canonicalHeaderName(name):                                                 
    if isinstance(name, unicode):                                               
        return name.lower().encode('iso-8859-1')                                
    return name.lower() 

My Synapse is back up with this.

[brevilo]

What's holding this back? There seems to be a solution (@Midar could you turn this into a PR?) and pinning Twisted to an outdated and potentially insecure version should clearly be fixed ASAP.

Thanks!