External and userspace configuration environment

[benjaoming]

Main problem

This won't require a big change, but the elaboration is long:

apt install kolibri-server
apt remove kolibri-server

Installing kolibri-server and removing it again will leave behind a broken Kolibri, as it is now depending on for instance a Redis cache configured in ~/.kolibri/options.ini or maybe binding to TCP port 80.

Because the configuration is a bit long-haired, it also takes significant time to sort out what's going on.

The kolibri-server debian package manages its configuration inside userspace. But we already built /etc/kolibri/ for this, so it can actually manage configuration outside of userspace.

uWSGI to read from environment

Currently, kolibri workers started by uWSGI will find their environment via options.ini, and everything from /etc/kolibri is ignored.

It would be a solution to pass the Kolibri environment set via /etc/kolibri to uWSGI by reading environment in uwsgi.ini with $(ENV_VAR) and passing it to kolibri with for instance env = KOLIBRI_CACHE_BACKEND=$(KOLIBRI_CACHE_BACKEND)

kolibri command in user shell (wontfix?)

While kolibri-server is installed and active, we can ask: What about invoking kolibri from the local user's shell environment? In that case, it wouldn't know about the Redis cache, the Nginx port etc. This can be seen as a problem of kolibri-server -- but it can and should refer this issue to be a more general issue with the kolibri Debian package that needs to sort out how it wants to handle local invocation of kolibri outside of the system service's configuration.

We can specify all the configurations we need with for instance /etc/kolibri/conf.d/01-cache, /etc/kolibri/conf.d/01-tcp-port etc. This will reduce a lot of the complexity of kolibri_server_setup.py as it's no longer responsible for updating options.ini but can simply pass on the given environment configured by the system service script.

CHERRYPY_START = True

This is the main issue that I think we should work out on the short term. The shell-invoked kolibri in a system with kolibri-server needs to know that it shouldn't run CherryPy. Otherwise it might occupy the port that Nginx should have.

This is currently managed by kolibri_server_setup.py, so one way to go about with the changes is to keep this in place, until kolibri Debian package knows how to read the /etc/kolibri/ environment for a user account.

options.ini

Here is a full default options.ini written by kolibri-server. All of these can be configured as environment variables, but it is better to keep them as close to userspace as possible.

[Cache]
CACHE_BACKEND = redis
[Database]
[Server]
CHERRYPY_START = True
[Paths]
CONTENT_DIR = /home/test/.kolibri/content
[Urls]
[Deployment]
URL_PATH_PREFIX = /
HTTP_PORT = 80
[Python]

One-directional configuration and ~/.kolibri/nginx.conf

There have previously been discussed how HTTP_PORT or CONTENT_DIR could be changed in options.ini (userspace) and then the Nginx configuration would read them. This behavior can be kept, and ~/.kolibri/nginx.conf can still be written from userspace and included in Nginx.

It's worth noting that ~/.kolibri/nginx.conf still causes occassional issues when .kolibri has been removed and nginx then fails to start. If you work with testing and development, you'll get stuck here often, as ~/.kolibri is deleted for testing purposes.

Another unexpected result is this: If you remove .kolibri and start kolibri-server again, the port changes.

# Select port 80
$ sudo apt install ./kolibri-server_0.4.0-0ubuntu1_all.deb
$ cat .kolibri/nginx.conf | grep listen
listen 80;
# Stop kolibri-server
$ sudo systemctl stop kolibri-server
# Remove userspace data
$ rm -rf .kolibri
# Start a clean kolibri-server
$ sudo systemctl start kolibri-server
# The port is now 8080
$ cat .kolibri/nginx.conf | grep listen
listen 8080;

Longer example

We cannot remove kolibri-server running on port 80 and have the system return in a functional state.

test@kolibri-test:~$ sudo apt remove kolibri-server
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following packages were automatically installed and are no longer required:
  fontconfig-config fonts-dejavu-core geoip-database kolibri libfontconfig1 libgd3 libgeoip1 libhiredis0.14 libjansson4 libjbig0 libjemalloc2 libjpeg-turbo8 libjpeg8 liblua5.1-0
  libnginx-mod-http-auth-pam libnginx-mod-http-dav-ext libnginx-mod-http-echo libnginx-mod-http-geoip libnginx-mod-http-geoip2 libnginx-mod-http-image-filter libnginx-mod-http-subs-filter
  libnginx-mod-http-upstream-fair libnginx-mod-http-xslt-filter libnginx-mod-mail libnginx-mod-stream libnorm1 libpgm-5.2-0 libtiff5 libwebp6 libxpm4 libzmq5 lua-bitop lua-cjson nginx-common nginx-full
  redis-server redis-tools uwsgi uwsgi-core uwsgi-plugin-python3
Use 'sudo apt autoremove' to remove them.
The following packages will be REMOVED:
  kolibri-server
0 upgraded, 0 newly installed, 1 to remove and 6 not upgraded.
After this operation, 455 kB disk space will be freed.
Do you want to continue? [Y/n] 
(Reading database ... 90396 files and directories currently installed.)
Removing kolibri-server (0.4.0-0ubuntu1) ...
WARNING:root:No C Extensions available for this platform.

WARNING  Options file /home/test/.kolibri/options.ini has been updated; server restart is required before change will take effect.
Job for kolibri.service failed because the control process exited with error code.
See "systemctl status kolibri.service" and "journalctl -xe" for details.

A solution

I think it's easier to implement a solution in a PR than to draft it in writing.

It would involve:

• A nice simplification of kolibri_server_setup.py
• Some new environment logic added to uwsgi.ini.
• Currently, --env=KOLIBRI_HOME=/home/test/.kolibri is specified in kolibri-server.init, this pattern could be moved to uwsgi.ini.

Unchanged: The handling of CHERRYPY_START = True should stay as it is until we have a better way of setting the external environment from /etc/kolibri/ for a given userspace shell invocation.

[benjaoming]

This issue was originally supposed to have targeted a few issues with respect to installing and uninstalling. It instead became an issue about where to maintain configuration.

[indirectlylit]

please provide references to examples of this affecting users in forums etc to help with prioritization

[benjaoming]

I haven't come across any reports. We can wait for user reports and prioritize it as that happens.

The configuration is brittle in the sense that either it breaks by uninstalling kolibri-server or by removing .kolibri and restarting the kolibri-server system service.

These are both problematic scenarios:

• It's not possible to try out kolibri-server, uninstall and automatically revert. Configuration has to be manually restored for kolibri to run as it did before kolibri-server was added.
• Removing .kolibri does not provide a clean system for kolibri-server to run in -- instead it forgets its configuration and Nginx momentarily breaks (demonstrated above).

It's not to say that the current state of kolibri-server is brittle, it installs, runs the system-service just fine and accepts upgrades gracefully... it's only if the user decides to do something messy that it can get problematic (hence the long elaboration). All the work can wait for when someone thinks it's necessary: These changes can be added with a fully compatible upgrade path and the solution will also simplify the package.

[jredrejo]

I think uninstalling should be clean for the user, so that's an issue. Not a clear issue because the user can tweak the options.ini , but I'd like to fix it. It is an easy fix and I'll do it asap.
For the nginx restoration, I already began to work on it, but I have switched to other priorities.

However, for the other issues mentioned here I think they are not real issues. Some are surprising to me is because they were proposed by Ben (I had done them other way), like the nginx config in userland. For others I don't understand why they are considered an issue, like uWSGI to read from environment . So, summarizing:

• uninstall/install should be fixed
  For the rest of the things, there are no complains for the user and from my point of view are things that are correct as they are.

[indirectlylit]

just to reiterate an earlier decision regarding configuration:

• For users that need custom configurations, we can provide sample nginx and uwsgi configs in our docs. These users would uninstall kolibri-server and manually configure nginx and uswgi to work with the kolibri package
• Our default advice is for users to install kolibri-server, which provides no manual configuration affordances of nginx/uswgi and abstracts those details away from the user completely

ref: https://docs.google.com/document/d/1poW3owLd_v_1QaQiRCcDJ-Fa3KKf1CYcHNQL67jgjM0/edit#

[benjaoming]

This can be parked under tech debt and isolated incidents can be fixed by patching kolibri_server_setup.py as @jredrejo suggests.

We have a full external environment that's 100% in the control of the debian packages through conventional means. So if we want to, we can have a more clean one-directional configuration path.

Some are surprising to me is because they were proposed by Ben (I had done them other way), like the nginx config in userland.

We went over the fundamental problem of configuration levels, I think these are compromises and decisions that we make together. The Nginx solution isn't ideal, I also had alternative proposals. It is a workaround and your follow-up improvement is most welcome.

[jredrejo]

Closed by #87