Coexistence of pbench-agent and pbench-server

[dbutenhof]

@webbnh and I spent a a fun hour or so debugging a problem he'd encountered in his unit test container experiments where server commands were failing, unable to import pbench.cli.server.user_management (and etc.) modules. (This was actually the second of two "intriguing" issues we discovered.)

The problem turns out to be that he'd installed both pbench-agent and pbench-server. The agent install creates /etc/profile.d/pbench-agent.sh, which constructs for all system logins a PYTHONPATH to run agent commands.

Looking at python3 -m site we can see that these paths are added to the front of the sys.path search path.

The root of the problem appears to be that when we look for pbench.cli.server.user_management, the Python module is pbench ... it doesn't matter that a path under that level (the "sub-module") isn't found: the search stops there. So we're trying to resolve all import pbench.* statements out of the agent pbench module, which obviously doesn't work.

In this case we were able to correct the issue with unset PYTHONPATH so it would use the implicit pbench account sys.path, which correctly finds the files in his local /home/pbench git copy; the legacy unit tests now run correctly.

Now normally we don't expect the agent and server to be installed together, but sharing the same Python module name remains problematic in several odd cases. One is this "one [container] to rule them all" unit testing model, but another is our planned 0.69 passthrough ("PUT shim") server, which requires both a 0.69 server install and a 0.7x agent install, with "server" code able to successfully invoke agent commands (or, should we choose to go that way, agent Python modules directly).

Here's the kicker: that means we need to be able to construct a sys.path which will satisfy both... and this is currently impossible.

1. A simple hack might be for put-shim to re-define PYTHONPATH to the agent configuration (perhaps by running /etc/profile.e/pbench-agent.sh) in the subprocess that invokes the agent command to push the tarball. This is a bit ugly, but might be good enough to get us through this unusual situation.
2. A cleaner (but also ugly) solution would be to completely separate the module names, e.g., pbench_agent.cli.* vs pbench_server.cli.* so the installations become truly independent.
3. A neater (but difficult to manage) solution would be to be able to merge the libraries; for example instead of installing into /opt/pbench-agent/lib and /opt/pbench-server/lib, put them in /opt/pbench (maybe reflecting the git tree, with /opt/pbench/lib, /opt/pbench/server, and /opt/pbench/agent), so they share a single global "pbench" module. (But there's where it gets complicated, since we're talking about different versions in general, so the agent pbench.common may not be the same bits as the server pbench.common.)

It'd be very satisfying to simply say "don't do that" ... but we have motivation to do this for our testing containers, and our product plans include doing this for end-of-life 0.69. So ... we so it seems that we need to stick our collective fingers in that wall outlet and figure out the best way to deal with the pain.

[webbnh]

Wow, that's a lot of labels! (Very colorful!!)

I think there might be a solution along the lines of keeping the respective deployments of the Agent and Server separate. I'm not sure how to do this in a seamless fashion, but, if they are installed in separate directories, then, at least in theory, we ought to be able to use them separately via the appropriate PYTHONPATH (etc.) configurations. Hopefully, there's not much call for a single "user" to run them both. If that's the case, then we can run the Server (including the Put-Shim) in a constrained, carefully constructed environment, while leaving the Agent available to the rest of the users. And, then, in the Put-Shim case, all we have to do is ensure that the subprocess invocation uses the normal user environment instead of its parent's.

(Just for the record, we finesse this situation in the "one container to rule them all case, because we have access to the Git checkout, which has both the Agent and Server components in place, so either of them works, with the exception of pbench-config, which grabs the Agent config file instead of the Server one, if given a choice (and, the server unit tests already deny it the installed Server config file, and my changes also deny it the installed Agent one).)

[dbutenhof]

Note that even with both agent and server install, I'm pretty sure our cron jobs will be started with only "server" PYTHONPATH via the crontab prologue. So, as I said in my "option 1", one possible solution is to simply run with that and make sure that, when we spawn the subprocess to run the agent PUT upload command, it runs with an appropriate agent PYTHONPATH. Everything else should work fine. Maybe we have no need for a longer term or more general solution; I just wanted to get it all written down.