improve error handling in /etc/matrix-synapse/conf.d

[pedro-nonfree]

Description

/etc/matrix-synapse/conf.d is an important section because there you can do whatever you want and upgrade debian-based package for matrix in a good manner

Steps to reproduce

cd /etc/matrix-synapse/conf.d
touch test1
systemctl restart matrix-synapse

OK?

cd /etc/matrix-synapse/conf.d
touch test2.yaml
systemctl restart matrix-synapse

FAILS (it also files if all lines in that file are commented), error says

matrix-synapse[23419]: Traceback (most recent call last):
matrix-synapse[23419]:   File "/usr/lib/python3.5/runpy.py", line 193, in _run_module_as_main
matrix-synapse[23419]:     "__main__", mod_spec)
matrix-synapse[23419]:   File "/usr/lib/python3.5/runpy.py", line 85, in _run_code
matrix-synapse[23419]:     exec(code, run_globals)
matrix-synapse[23419]:   File "/opt/venvs/matrix-synapse/lib/python3.5/site-packages/synapse/app/homeserver.py", line 651, in <module>
matrix-synapse[23419]:     main()
matrix-synapse[23419]:   File "/opt/venvs/matrix-synapse/lib/python3.5/site-packages/synapse/app/homeserver.py", line 646, in main
matrix-synapse[23419]:     hs = setup(sys.argv[1:])
matrix-synapse[23419]:   File "/opt/venvs/matrix-synapse/lib/python3.5/site-packages/synapse/app/homeserver.py", line 324, in setup
matrix-synapse[23419]:     "Synapse Homeserver", config_options
matrix-synapse[23419]:   File "/opt/venvs/matrix-synapse/lib/python3.5/site-packages/synapse/config/_base.py", line 623, in load_or_generate_config
matrix-synapse[23419]:     config_dict = read_config_files(config_files)
matrix-synapse[23419]:   File "/opt/venvs/matrix-synapse/lib/python3.5/site-packages/synapse/config/_base.py", line 670, in read_config_files
matrix-synapse[23419]:     specified_config.update(yaml_config)
matrix-synapse[23419]: TypeError: 'NoneType' object is not iterable
systemd[1]: matrix-synapse.service: Control process exited, code=exited, status=1/FAILURE
systemd[1]: matrix-synapse.service: Failed with result 'exit-code'.
systemd[1]: Failed to start Synapse Matrix homeserver.

cd /etc/matrix-synapse/conf.d
touch test3.yml
systemctl restart matrix-synapse

OK? I have a question, how can I verify that the running synapse has my variable enabled? Maybe this file is not taken in account by synapse

Version information

• Homeserver: matrix.guifi.net

• Version:

curl http://localhost:8008/_synapse/admin/v1/server_version
{
    "python_version": "3.5.3",
    "server_version": "1.12.3"
}

• Install method: debian package

• Platform: debian stable 10 buster proxmox VM

[babolivier]

Hi @pedro-nonfree, and thanks for your report! I've opened #7341 to prevent Synapse from hitting a fatal error if it's given an empty or malformed YAML file, instead it will print, e.g.:

File 'conf.d/test2.yaml' is empty or doesn't parse into a key-value map. IGNORING.

when starting up.

how can I verify that the running synapse has my variable enabled?

If a file is ignored, Synapse will say so when starting up, e.g. by reproducing your examples I can see:

Found file in config directory that does not end in '.yaml': 'conf.d/test1'. IGNORING.
Found file in config directory that does not end in '.yaml': 'conf.d/test3.yml'. IGNORING.

in Synapse's output. I just did some testing on a production homeserver I run that's been installed using the Debian packages and it looks like it pipes correctly into journalctl, as I can see the following in the command's output:

Apr 24 14:55:11 [...] matrix-synapse[6510]: Found file in config directory that does not end in '.yaml': '/etc/matrix-synapse/conf.d/test.yml'. IGNORING.

Any file that isn't mentioned in a message ending with "IGNORING" at the start of Synapse should be considered as parsed and its settings processed as long as they're valid Synapse settings (as long as they're provided to Synapse using -c/--config-path obviously).

[pedro-nonfree]

thanks @babolivier , I liked a lot your proposal in #7341 and your solution is neat in my opinion

I had this issue while understanding what was the configuration synapse was using after reading all the files and executing the service. I still have this question. How can ensure what is the value for a key, if that key is in some files.

[babolivier]

How can ensure what is the value for a key, if that key is in some files.

There is no way to do that currently, apart from making sure you don't have duplicates between your configuration files (which you should do anyway).

[babolivier]

#7341 has been merged so I'm closing this issue. If you have more questions about Synapse, feel free to join #synapse:matrix.org and ask them there :)