Adds initial scripts to use rsyslog for logging in workstation #8
Conversation
|
Why introduce Redis? Why not just use syslog there as well? |
Because we want to smoothly split the logs in different files based on which vm they are coming from, and redis helps as it automagically makes them proper even if more than one thread is pushing them into the queue. Tested this approach before while processing jobs across multiple processes over many machines. |
|
rsyslog can route to different destinations based on hostname, application name, and other criteria, and it's pretty good at handling concurrent sources. |
Yup, but, I was unable to see how to use it here. Can you please help in the PR on that front? in the |
|
Of course. I hope to have time later today, will see what's required or what I'm missing. 😉 |
There was a problem hiding this comment.
I looked at just using rsyslog instead of redis within sd-log, and I think getting syslog-formatted messages across the qrexec link in the chain and then having rsyslog in sd-log route them is problematic. We could probably figure out rsyslog templates and parsing, maybe using the mmnormalize plugin, and get the messages routed properly, but frankly it looks more painful than what you've got now. I've spent a few hours experimenting and digging through the rsyslog docs, and I'm ready to be done.
This might be easier if we had rsyslog in the various VMs talking directly to rsyslog in sd-log, but setting up encrypted transport and configuring the firewall or qvm-connect-tcp is a lot of work on top of figuring out the rsyslog configuration, and ultimately less safe than qrexec.
Any other multiprocess-safe Python logging solution looks to be about the same amount of work, still involves a daemon process to collect and write the logs (I haven't found one that tries file locking), and will overall likely be less solid than using Redis.
I think this just needs some more documentation and some operational polishing -- getting Redis installed in sd-log-buster-template and added to the pip requirements here; figuring out how securedrop-log-saver should be run (systemd service as user?) -- and then we should ship it. It works, and if we want to fine-tune it later it should be much easier for any of us to work with securedrop-log-saver than rsyslog configuration.
| f"{vmname}", | ||
| "syslog.log", | ||
| ) | ||
| fh = open(filepath, "a") |
There was a problem hiding this comment.
This bit needs the directory creation logic from securedrop-log.
|
|
||
| import os | ||
| import sys | ||
| import redis |
There was a problem hiding this comment.
will this require a pip install?
There was a problem hiding this comment.
Yeah, which of course will need to be done in the template.
|
@rmol thanks for the feedback. Now, I can work to make it more production ready from that draft state. |
|
Q: If we want to control the python redis package, then we can follow like any other our packages, and install via dh-virtualenv, this will also require to add the @redshiftzero @rmol @emkll Just asking this as it was in mind about this particular code/application packaging. For now I am working to make it a |
kushaldas
force-pushed
the
oh_my_syslog
branch
3 times, most recently
from
afe7803
to
6b8e8ca
Compare
|
I prefer installing the redis Python module via dh-virtualenv, particularly if we're using a version that's already been reviewed. 😉 Better to have all the Python dependencies specified in one place and installed the same way, and to not build the virtualenv with system packages. |
|
I will be submitting the PR tomorrow, I will have to add a wheel for |
kushaldas
force-pushed
the
oh_my_syslog
branch
2 times, most recently
from
ad05c89
to
7c5ab29
Compare
| confirmMessages="on" | ||
| ...) | ||
|
|
||
| Licensed under the Apache License, Version 2.0 (the "License"); |
There was a problem hiding this comment.
should we license (a)GPLv3 here for consistency with both other repos and your setup.py: https://github.com/freedomofpress/securedrop-log/pull/8/files#diff-2eeaed663bd0d25b7e608891384b7298R27
There was a problem hiding this comment.
We can not as this is copied from the official example.
kushaldas
force-pushed
the
oh_my_syslog
branch
from
7c5ab29
to
2ee7c61
Compare
|
Updated with the latest wheel hashes from debian packaging repo for the production beta release. |
kushaldas
force-pushed
the
oh_my_syslog
branch
from
2ee7c61
to
10aae1e
Compare
There was a problem hiding this comment.
Thanks @kushaldas I have gone through the test plan and tried to adapt it to a test plan where, I have some questions, and some files appear to be missing from the packaging, and ran into issues with redis. Perhaps a PR to securedrop-workstation is required to populate the systemd service files? Would it be possible to update the test plan as well, it seems like your test plan is out-of-date, now that you've done packaging work.
- In
dom0, add the following file/etc/qubes-rpc/policy/securedrop.Logand then the text below.
@tag:sd-workstation sd-log allow
@anyvm @anyvm deny
Is this step necessary? We already add those in master: https://github.com/freedomofpress/securedrop-workstation/blob/master/dom0/sd-log.sls#L44-L49
- in
sd-logvm, Add the following content to/etc/qubes-rpc/securedrop.Log
This appears to be already handled by https://github.com/freedomofpress/securedrop-debian-packaging/pull/133/files#diff-c3454cdfb26226700f264ea620b88cc2R1
/usr/sbin/securedrop-log
-
Create a virtualenv at
/opt/venvs/securedrop-log, you may have to apt install python3-venv for this. -
and then place
securedrop-redis-logandsecuredrop-log-saverscripts to the
virtualenv at/opt/venvs/securedrop-logand create links to/usr/sbin/
directory and make sure that they are executable. This step will be automated via
the Debian package.
Same as above, this seems to be handled by the packaging logic -
❓ : Copy
securedrop-log.servicefile to/usr/systemd/systemand then
Is there a reason this isn't dropped into place by https://github.com/freedomofpress/securedrop-debian-packaging/pull/133/files ?
❗ the target folder is/etc/systemd/system
sudo systemctl daemon-reload
sudo systemctl start redis
sudo systemctl start securedrop-log
❗ when running sudo systemctl start redis, i get the following error:
Failed to start redis.service: Unit redis.service not found
I stopped the test plan at this point, please let me know if I missed something. An updated test plan (using the packaging logic) should make things easier and more reproducible for a reviewer, and also allow to review the packaging logic as well.
Also, you are correct that I missed to put the systemd file on place, I was waiting for this review to finish (verifying that the idea is correct). You will have to manually install I have updated the test steps. |
No, but, I just wanted to have clear steps. |
There was a problem hiding this comment.
Thanks @kushaldas for your patience, i have gotten much further this time. A few comments:
- I have opened a ticket to track the provisioning/configuration for all VMs in Install and configure securedrop-log as part of provisioning securedrop-workstation#440, it might be good to tackle this prior to merge of this PR to ensure the configuration is reliable and reproducible. What do you think?
- python3-distutils package was required to get the scripts to work on the sd-log vm see stacktrace in [1]. Do we need to include the system package or a wheel?
- I would recommend stating in the Readme that this package is dual-purpose (provides scripts to both send and receive). I can see some advantages with this approach, but I am curious if you issues in the future? specifically, if we need to automate some configuration changes (in postinst or other), the logic to distinguish a system that is sending logs from one receiving logs is complex.
- Having some issues sending logs over: invoking
sd-rsyslogdirectly allows me to send amessage (and i see in qvm-rpc logs), but I don't see the message being logged/processed onsd-log. The redis server logs don't indicate a connection, but maybe I am doing something wrong. securedrop-log.service seems properly configured [2]. I think this highlights the issues described in 1), where it would be helpful to have a more reliable and reproducible configuration
[1]
Traceback (most recent call last):
File "/opt/venvs/securedrop-log/sbin/securedrop-log-saver", line 5, in <module>
import redis
File "/opt/venvs/securedrop-log/lib/python3.7/site-packages/redis/__init__.py", line 1, in <module>
from redis.client import Redis, StrictRedis
File "/opt/venvs/securedrop-log/lib/python3.7/site-packages/redis/client.py", line 13, in <module>
from redis.connection import (ConnectionPool, UnixDomainSocketConnection,
File "/opt/venvs/securedrop-log/lib/python3.7/site-packages/redis/connection.py", line 2, in <module>
from distutils.version import StrictVersion
File "/opt/venvs/securedrop-log/lib/python3.7/distutils/__init__.py", line 25, in <module>
from distutils import dist, sysconfig
ImportError: cannot import name 'dist' from 'distutils' (/opt/venvs/securedrop-log/lib/python3.7/distutils/__init__.py)
[2]
● securedrop-log.service - securedrop logging Service
Loaded: loaded (/etc/systemd/system/securedrop-log.service; disabled; vendor preset: enabled)
Active: active (running) since Wed 2020-02-05 14:27:48 EST; 48min ago
Main PID: 3828 (securedrop-log-)
Tasks: 1 (limit: 597)
Memory: 8.1M
CGroup: /system.slice/securedrop-log.service
└─3828 /opt/venvs/securedrop-log/bin/python3 /opt/venvs/securedrop-log/sbin/securedrop-log-saver
`sd-rsyslog` is the output plugin of rsyslog to be installed in /usr/sbin `sdlog.conf` is the configuration of rsyslog in /etc/rsyslog.d/ `securedrop-redis-log` is part of the Qrexec service inside of sd-log vm. This will receive the messages from any other vm, and add them into a queue in Redis. `securedrop-log-saver` will be the service inside `sd-log` VM, this will monitor the queue, and save any incoming message to the `syslog.log` file of the respective directory for each VM. It also has the Makefile for the project.
kushaldas
force-pushed
the
oh_my_syslog
branch
from
10aae1e
to
29c1cf7
Compare
|
@emkll Updated the test steps using Debian package only. |
There was a problem hiding this comment.
Thanks for your patience, @kushaldas, this now works as expected 🎉
I am approving this pr but we should not merge it right away. Let's ensure CI passes since nightlies will be built once we merge. I propose we do as follows:
- push a 0.0.4 tag
- restart CI and observe ci passing in Adds changes for updated securedrop-log package securedrop-builder#133
- merge Adds changes for updated securedrop-log package securedrop-builder#133
- Open follow up issue to track configuration Install and configure securedrop-log as part of provisioning securedrop-workstation#440
- merge this PR ❗ @rmol has requested changes, however the 0.0.4 tag and nightlies should unblock work in Install and configure securedrop-log as part of provisioning securedrop-workstation#440 and we can wait a little bit longer to merge this one (we can discuss in standup)
|
I've appended 9fbcae4 to add a build job to CI for this repository |
|
I have tracked the split packaging in freedomofpress/securedrop-client#1721 It seems like most issues @rmol requested changes on were address, with the exception of a follow up tracked in freedomofpress/securedrop-client#1721., in order to unblock nightlies. As i am not an admin in this repo, and because I have added a commit for CI, I cannot dismiss @rmol 's review and should not merge this pr. Could someone else @redshiftzero perhaps? review my commit and dismiss the review? |
redshiftzero
dismissed
rmol’s stale review
dismissing as @rmol is not available and we want to merge this to resolve nightlies and address further comments in follow ups which Mickael has filed
closes #10
sd-rsyslogis the output plugin of rsyslog to be installed in/usr/sbin
sdlog.confis the configuration of rsyslog in /etc/rsyslog.d/securedrop-redis-logis part of the Qrexec service inside ofsd-log vm. This will receive the messages from any other vm,
and add them into a queue in Redis.
securedrop-log-saverwill be the service insidesd-logVM,this will monitor the queue, and save any incoming message to
the
syslog.logfile of the respective directory for each VM.freedomofpress/securedrop-builder#132 needs to be merged first as
we need redis to build the package. You can still test this PR with the steps below.
How to test?
The following are the manual steps.
python3 setup.py sdistThen install the fresh debian package in sd-log vm (it will require network connection to get a dependency), then also install redis-server and redis
securedrop-log, execute the following commands.Next, in
sd-appvmsd-rsyslog-example.conffile to/etc/sd-rsyslog.conf, make sure updateit so that is shows the right localvm name (sd-app).
logger hello, this is sd-apprun this command insd-app, this should up in the/home/user/QubesIncomingLogs/sd-app/syslog.logfile/home/user/QubesIncomingLogs/sd-app/syslog.logfile.