Provide authorisation/security information to kafka producer.

[stmudie]

Expected Behavior

Our Kafka servers are secured using SSL and SASL. We wish to supplying the required parameters to the Queueserver's in built Bluesky-Kafka Publisher instance (via the producer_config) for connecting to our servers.

Current Behavior

Currently there is no way of passing this information through to the Bluesky Kafka Producer instance provided by default within the Queueserver.

Possible Solution

Add a command line argument to start-re-manager for passing a path to a Kafka configuration file then reading said config file and passing through to the Kafka-Bluesky Publisher constructor. For example:
https://github.com/stmudie/bluesky-queueserver/compare/783b950..895d150
I'm happy for this code to form the basis of a PR (after updating to bring in latest commits from upstream).

Context

Without this configuration option (or an alternative method for configuring the subscribed Kafka publisher/producer) we can't use the Queueserver with our standard Kafka deployments.

[dmgav]

While it is possible to extend the set of command line arguments, I don't think it is the best way to go, since it is unlikely we could cover every possible scenario. It is likely that built-in options to create Run Engine and subscribe it to Data Broker or Kafka Publisher are going to be used only for demos and tutorials. In production deployments we create an instance of Run Engine (it must be a variable named RE) in the startup script and add all needed subscriptions. If start-re-manager is passed the parameter --keep-re then no additional instance of RE Engine is created and Run Engine defined in the startup code is used by the Queue Server to run plans.

[tacaswell]

I'm also inclined to think that --keep-re is the right way to go short term. I am coming to think my initial push to have QS create its own RE was a mistake.

Medium term, it might also be worth investigating something like what tiled does with handlers to have a config file that specifies what callbacks to attach the RE that it creates rather than passing this configuration through the command line.

[dmgav]

In production we set all callbacks in the startup code. I think that may be the default mode should be the one that uses Run Engine created in the startup scripts and the other mode should be explicitly activated. Also we don't want to couple Queue Server deployment and configuration of other services not used by the Queue Server directly, such as Kafka.