[WIP] Refs #34239 - [rex_ssh] Install mosquitto with 'pull-mqtt' mode #726

wbclark · 2022-02-09T11:05:10Z

Opening this work in progress for feedback. This requires, at a minimum, tests as well as support for certificate authentication.

ekohl

Something I never got clarity on is whether MQTT mode still supports SSH mode or if you really need to choose between the two.

manifests/plugin/remote_execution/ssh.pp

ekohl · 2022-02-09T12:15:57Z

metadata.json

+    {
+      "name": "puppet/mosquitto",
+      "version_requirement": ">= 1.0.1 < 2.0.0"
+    },


I wonder if this should be a hard dependency or rather a soft one. It's really an edge case so most users won't need it.

If it's a soft dependency, it would need to be included explicitly in the installer Puppetfile?

With the soft dependency approach (what I have now implemented) what is the proper method to test?

add it to .fixtures.yml

ekohl · 2022-02-09T12:16:28Z

templates/plugin/remote_execution_ssh.yml.erb

@@ -7,3 +7,7 @@

 # Whether to run remote execution jobs asynchronously
 :mode: <%= scope.lookupvar("::foreman_proxy::plugin::remote_execution::ssh::mode") %>
+<% if scope.lookupvar("::foreman_proxy::plugin::remote_execution::ssh::mode") == 'pull-mqtt' -%>
+:mqtt_broker: localhost
+:mqtt_port: 1883


It would be nice if this port was somehow extracted from the mosquitto config.

Will add the additional params necessary to make that happen

ekohl · 2022-02-09T12:17:11Z

templates/plugin/remote_execution_ssh.yml.erb

@@ -7,3 +7,7 @@



Are the options above this (like ssh_identify_key_file and kerberos_auth still relevant in mqtt mode?

@adamruzicka Is this something that you can answer for us? See the question from above comment as well:

Something I never got clarity on is whether MQTT mode still supports SSH mode or if you really need to choose between the two.

kerberos_auth is only applicable if one of the ssh modes is used.
ssh_identify_key_file is only applicable for ssh modes and ansible. This is probably something that should be looked into as well

smart_proxy_remote_execution_ssh validates that a private key really exists at the location specified by ssh_identity_key_file, regardless of the selected mode[1].

[1] - https://projects.theforeman.org/issues/34439

Added a comment about it at https://github.com/theforeman/puppet-foreman_proxy/pull/726/files#diff-b2aaacbd8a851b0617daa6eee5a0bb4b0330898d90035b1766117a2012c6a3d9R3

Hmm, now that I'm thinking about it again, the ssh key is used for web console integration no matter what mode is used. So maybe always managing the key isn't a bad idea?

adamruzicka · 2022-02-11T11:15:01Z

whether MQTT mode still supports SSH mode or if you really need to choose between the two.

They are mutually exclusive, you really need to choose between the two, or at least as long as smart_proxy_remote_execution_ssh is concerned. Once you add smart_proxy_ansible into the mix, sp-ansible will always use ssh.

ekohl · 2022-02-11T11:47:14Z

It sounds to me like it should have been implemented as:

smart_proxy_remote_execution is the REX plugin
smart_proxy_remote_execution_ssh is the SSH provider for REX
smart_proxy_remote_execution_mqtt is the MQTT provider for REX

That would have been in line with how the Smart Proxy implements abstraction. The mode parameter would be provider.

However, the Ansible part complicates things.

adamruzicka · 2022-02-11T12:14:03Z

IIRC the plan was to rename "SSH" provider in foreman to "Script" and then having ssh or mqtt as a different transport between the proxy and the target hosts. Once it goes underway, we could rename smart_proxy_remote_execution_ssh to smart_proxy_remote_execution.

However, the Ansible part complicates things.

And salt too

manifests/plugin/remote_execution/ssh.pp

evgeni · 2022-03-22T10:34:49Z

manifests/plugin/remote_execution/ssh.pp

+
+  if $mode == 'pull-mqtt' {
+    class { 'mosquitto':
+      package_name => 'mosquitto',


what's the reason for overriding the package name here?

evgeni · 2022-03-22T11:09:44Z

Is there a way (in Kafo?) to make Enum params show the possible choices in --help?

Right now, the help ends up being:

    --foreman-proxy-plugin-remote-execution-ssh-mode  Operation Mode of the plugin. (current: "ssh")

Which makes it hard for the user to discover they need to pass pull-mqtt here.

(obviously, totally unrelated to the actual change in this PR, but something that jumped at me)

evgeni · 2022-03-22T11:45:18Z

Couple of architectural questions:

Right now, mosquitto binds to 127.0.0.1:1883, which means that neither external proxies, nor clients can actually connect to it?
There is no cert auth (and I also don't see any in theforeman/smart_proxy_remote_execution_ssh@611f315), which means that any user with access to the Foreman machine can inject commands for any client, and any client can get messages for any other client.
The current setup means that every foreman-proxy can connect to (their) mosquitto and inject messages there, but I kinda thought all of these would be coming from the central Foreman instance, and the mosquittos on the proxies would be just that, proxies? Edit: looking at https://community.theforeman.org/t/rfc-rex-pull-provider/21593, the "multi proxy" setup is not very well spellt out, but looks like individual trees.

ehelms · 2022-03-22T12:54:15Z

Right now, mosquitto binds to 127.0.0.1:1883, which means that neither external proxies, nor clients can actually connect to it?

This will need updating as clients need to connect to the mosquitto instance.

There is no cert auth (and I also don't see any in theforeman/smart_proxy_remote_execution_ssh@611f315), which means that any user with access to the Foreman machine can inject commands for any client, and any client can get messages for any other client.

@adamruzicka Can you speak to this? Is auth a TODO for talking to mosquitto from REX plugin?

The current setup means that every foreman-proxy can connect to (their) mosquitto and inject messages there, but I kinda thought all of these would be coming from the central Foreman instance, and the mosquittos on the proxies would be just that, proxies? Edit: looking at https://community.theforeman.org/t/rfc-rex-pull-provider/21593, the "multi proxy" setup is not very well spellt out, but looks like individual trees.

I think you are describing the Qpid model which we were not intending to do with the pull provider. Rather, Foreman tells a Foreman Proxy that there is a job for a host, and the smart-proxy then relays that to the mosquitto running on the proxy itself.

adamruzicka · 2022-03-22T13:58:32Z

Is auth a TODO for talking to mosquitto from REX plugin?

Pretty much yes. What I didn't know back then, and still I'm not sure, is whether we can just reuse the set of certs the proxy already has or if there will have to be another set. Here I'm assuming the certs proxy already has and consumer certs clients use are issued by different CAs which might pose a problem?

evgeni · 2022-03-23T07:27:20Z

Is auth a TODO for talking to mosquitto from REX plugin?

Pretty much yes. What I didn't know back then, and still I'm not sure, is whether we can just reuse the set of certs the proxy already has or if there will have to be another set. Here I'm assuming the certs proxy already has and consumer certs clients use are issued by different CAs which might pose a problem?

In the Katello scenario, all involved parties have certs issued by the Katello CA.
In the Foreman scenario with Puppet, all involved parties have certs issued by the Puppet CA.
Otherwise, there will be dragons, but let's not go there just yet (@ehelms has it on the agenda).

adamruzicka · 2022-03-23T08:32:49Z

A brief recap how the auth was planned:

Mosquitto should use ACLs to grant access to topics based on a CN present in client's (even a smart-proxy is a client in this context) cert. Hosts only need read access to their own topics, smart-proxy needs to be able to write to all of the per-host topics. Per @jlsherrill 's comment this should work, just the topic names will be slightly different.

In the Katello and Foreman with puppet scenarios dealing with certs should be pretty straight forward. In the dragon scenario, both mosquitto and yggdrasil can be configured to trust any cert signed by one of many configured CAs. While it may need special handling, hopefully there won't be that many dragons.

It would be great if in this PR we could:

set up the ACLs
make mosquitto reuse the same certs smart-proxy uses
- but bear in mind we may eventually want to configure multiple CAs

In the meantime I'll take a stab at making sp-rex reuse smart-proxy's certs when connecting to mosquitto.

EDIT: theforeman/smart_proxy_remote_execution_ssh#75

evgeni · 2022-03-23T09:15:37Z

/etc/mosquitto/mosquitto.conf:

# THIS FILE IS MANAGED BY PUPPET
listener 1883

acl_file /etc/mosquitto/foreman.acl

cafile /etc/foreman-proxy/ssl_ca.pem
certfile /etc/foreman-proxy/ssl_cert.pem
keyfile /etc/foreman-proxy/ssl_key.pem

require_certificate true
use_identity_as_username true

/etc/mosquitto/foreman.acl:

pattern read yggdrasil/%u/data/in
pattern write yggdrasil/%u/control/out

user centos8-stream-katello-nightly.tanso.example.com
topic write yggdrasil/+/data/in
topic read yggdrasil/+/control/out

/etc/yggdrasil/config.toml:

# yggdrasil global configuration settings written by katello-pull-transport-migrate

# IMPORTANT: mqtts here, k-p-t-m by default does tcp, which disables cert auth…
broker = ["mqtts://centos8-stream-katello-nightly.tanso.example.com:1883"]
cert-file = "/etc/pki/consumer/cert.pem"
key-file = "/etc/pki/consumer/key.pem"
ca-root = ["/etc/rhsm/ca/katello-server-ca.pem"]
log-level = "info"

subscribing with CLI (on client):

# mosquitto_sub --cafile /etc/rhsm/ca/katello-server-ca.pem --cert /etc/pki/consumer/cert.pem --key /etc/pki/consumer/key.pem --host centos8-stream-katello-nightly.tanso.example.com --port 1883 -t THE_TOPIC

publishing with CLI (on server):

# mosquitto_pub -t THE_TOPIC -m THE_MESSAGE --cafile /etc/foreman-proxy/foreman_ssl_ca.pem --key /etc/foreman-proxy/foreman_ssl_key.pem --cert /etc/foreman-proxy/foreman_ssl_cert.pem -h $HOSTNAME -p 1883

publishing with proxy (on server):
I've edited pull_script.rb to read:

      MQTT::Client.connect(:host => settings.mqtt_broker, :port => settings.mqtt_port, :ssl => true, :cert_file => '/etc/foreman-proxy/foreman_ssl_cert.pem', :key_file => '/etc/foreman-proxy/foreman_ssl_key.pem', :ca_file => '/etc/foreman-proxy/foreman_ssl_ca.pem') do |c|
        c.publish(mqtt_topic, JSON.dump(payload), false, 1)
      end

test REX with:

hammer job-invocation create --job-template 'Run Command - SSH Default' --inputs 'command=uptime' --search-query "name = centos8-stream.tanso.example.com"

Edit: this seems to work now, ACLs are also working properly

ehelms · 2022-03-23T12:55:59Z

Nice walk through of steps and testing! If you have your setup still, do you think you could try out the dragon scenario? aka custom certificates

evgeni · 2022-03-23T13:11:15Z

Nice walk through of steps and testing! If you have your setup still, do you think you could try out the dragon scenario? aka custom certificates

It's a simple Katello without Puppet setup, I guess I could add Puppet to the mix and add another client.

evgeni · 2022-03-23T15:05:26Z

The dragons ate my lunch!

I could get mosquitto to offer both certs with the following config:

listener 1883


acl_file /etc/mosquitto/foreman.acl

#cafile /etc/foreman-proxy/ssl_ca.pem
cafile /etc/mosquitto/ca.pem # this file is cat foreman/ssl_ca and puppet_ca.pems
certfile /etc/foreman-proxy/ssl_cert.pem
keyfile /etc/foreman-proxy/ssl_key.pem

require_certificate true
use_identity_as_username true

listener 2883

cafile /etc/puppetlabs/puppet/ssl/certs/ca.pem
certfile /etc/puppetlabs/puppet/ssl/certs/centos8-stream-katello-nightly.tanso.example.com.pem
keyfile /etc/puppetlabs/puppet/ssl/private_keys/centos8-stream-katello-nightly.tanso.example.com.pem

require_certificate true
use_identity_as_username true

Note: it didn't like me to point at the same acl_file twice, so no idea if ACLs are in place right now.

Note: I tried just listing both CAs in cafile and while this was accepted by mosquitto, yggrasil didn't want to connect (yggdrasild[3111]: cannot connect to broker: network Error : x509: certificate signed by unknown authority). Solvable by adding the Katello CA to the trust of yggdrasild on the client.

however REX didn't work in the end, as even with the Katello CA in the trust store, yggd fails to download the script:

[yggdrasild] 2022/03/23 15:00:44 /builddir/build/BUILD/yggdrasil-0.2.0/cmd/yggd/mqtt.go:18: received a message on topic yggdrasil/centos7.tanso.example.com/data/in
[yggdrasild] 2022/03/23 15:00:44 /builddir/build/BUILD/yggdrasil-0.2.0/cmd/yggd/http.go:38: sending HTTP request: GET https://centos8-stream-katello-nightly.tanso.example.com:9090/ssh/jobs/e72803e1-b93b-489c-85ee-02db1d024c8c
[yggdrasild] 2022/03/23 15:00:44 /builddir/build/BUILD/yggdrasil-0.2.0/cmd/yggd/grpc.go:142: cannot get detached message content: cannot download from URL: Get "https://centos8-stream-katello-nightly.tanso.example.com:9090/ssh/jobs/e72803e1-b93b-489c-85ee-02db1d024c8c": remote error: tls: unknown certificate authority

Probably because AUTH against the proxy on 9090 also needs to happen with the Katello cert…

ehelms · 2022-03-23T15:38:56Z

Solvable by adding the Katello CA to the trust of yggdrasild on the client.

however REX didn't work in the end, as even with the Katello CA in the trust store, yggd fails to download the script:

How'd you do this? by setting multiple CA files in yggdrasil config or another method?

evgeni · 2022-03-23T16:16:33Z

Solvable by adding the Katello CA to the trust of yggdrasild on the client.

however REX didn't work in the end, as even with the Katello CA in the trust store, yggd fails to download the script:

How'd you do this? by setting multiple CA files in yggdrasil config or another method?

Exactly, I added both files to the Yggdrasil config

evgeni · 2022-03-25T12:46:44Z

Just for completeness, for the dragon scenario, to connect to the Katello CA signed Mosquitto with a Puppet CA signed cert (port 1883 above), you need the following config.toml:

# yggdrasil global configuration settings written by katello-pull-transport-migrate

broker = ["mqtts://centos8-stream-katello-nightly.tanso.example.com:1883"]
cert-file = "/etc/puppetlabs/puppet/ssl/certs/centos7.tanso.example.com.pem"
key-file = "/etc/puppetlabs/puppet/ssl/private_keys/centos7.tanso.example.com.pem"
ca-root = ["/etc/puppetlabs/puppet/ssl/certs/ca.pem", "/etc/pki/ca-trust/source/anchors/katello-server-ca.crt"]
log-level = "debug"

ekohl · 2022-03-25T13:15:18Z

Is there a way (in Kafo?) to make Enum params show the possible choices in --help?

I've ran into this in the past. We sort of have this in the to_s definition:
https://github.com/theforeman/kafo/blob/a68e22c8fb5973c53088811db18a8a8a64264fea/lib/kafo/data_types/enum.rb#L8-L10

So it would be a matter of enhancing the help builder to describe the data type. The downside is that it may explode the size when it starts to describe complex types as well (think the regex for paths or hostnames/IPs). That's what I ran into the last time I looked at it.

ehelms · 2022-03-25T18:07:31Z

The dragon scenario I was thinking of and tested was (and seemed to work ok with a few tweaks mentioned below):

Generate custom certificates (using theforeman/forklift#1481):

ansible-playbook playbooks/custom_certificates.yml -l centos8-stream-katello-nightly

Now update foreman with katello install with custom certificates:

foreman-installer --certs-server-cert /root/custom_certificates/certs/centos8-stream-katello-nightly.wareagle.example.com.crt --certs-server-key /root/custom_certificates/private/centos8-stream-katello-nightly.wareagle.example.com.key --certs-server-ca-cert /root/custom_certificates/certs/custom_ca.crt --certs-update-server --certs-update-server-ca --disable-system-checks

Now smart-proxy is running with the custom certificates as it's server certificates. Add back the following config and restart mosquitto:

# THIS FILE IS MANAGED BY PUPPET
listener 1883

acl_file /etc/mosquitto/foreman.acl

cafile /etc/foreman-proxy/combined_ca.pem
certfile /etc/foreman-proxy/ssl_cert.pem
keyfile /etc/foreman-proxy/ssl_key.pem

require_certificate true
use_identity_as_username true

On the client, my yggdrasil is configured as:

# cat /etc/yggdrasil/config.toml
# yggdrasil global configuration settings

broker = ["mqtts://centos8-stream-katello-nightly.wareagle.example.com:1883"]
cert-file = "/etc/pki/consumer/cert.pem"
key-file = "/etc/pki/consumer/key.pem"
ca-root = ["/etc/rhsm/ca/katello-default-ca.pem", "/etc/rhsm/ca/katello-server-ca.pem"]
log-level = "info"

I also had to configure the smart-proxy with a combined CA certificate to get this work (more to come on this fun finding).

ehelms · 2022-03-26T12:04:08Z

~~I think this theforeman/puppet-certs#396 addresses some of the problems we saw with the dragon scenario (at least for Katello scenarios).~~

We will need to deploy some dedicated certificates for mosquitto over in puppet-certs. I am thinking we deploy to: /etc/mosquitto/ssl

And then we have two options for configuration:

Only deploy mosquitto with Katello default CA or Puppet certificates (in Foreman scenario)
Deploy mosquitto with custom certificates from users as we have been doing with public facing services (i.e. the same server certificates for the smart-proxy). This will mean we need to deploy a combined CA, as mosquitto will have smart-proxy and clients talking to it with potentially different certificate setups. The REX smart_proxy plugin could opt to use the Foreman client certificates to talk to mosquitto thereby reducing the difference.

ehelms · 2022-04-13T03:13:35Z

There is a lot of great discussion here which, while I hate to lose, I have taken over this PR with #737

wbclark added the Enhancement label Feb 9, 2022

theforeman-bot added Needs testing Not yet reviewed labels Feb 9, 2022

wbclark marked this pull request as draft February 9, 2022 11:05

ekohl reviewed Feb 9, 2022

View reviewed changes

adamruzicka reviewed Feb 21, 2022

View reviewed changes

manifests/plugin/remote_execution/ssh.pp Outdated Show resolved Hide resolved

ehelms reviewed Mar 10, 2022

View reviewed changes

manifests/plugin/remote_execution/ssh.pp Outdated Show resolved Hide resolved

wbclark force-pushed the 34239_rex_pull_transport branch from 7435214 to 3a6116e Compare March 21, 2022 11:46

evgeni reviewed Mar 22, 2022

View reviewed changes

Refs #34239 - [rex_ssh] Install mosquitto with 'pull-mqtt' mode

56632f1

wbclark force-pushed the 34239_rex_pull_transport branch from 3a6116e to 56632f1 Compare March 23, 2022 13:04

ehelms mentioned this pull request Mar 26, 2022

Deploy server CA certificate for smart-proxy rather than default theforeman/puppet-certs#396

Closed

ehelms closed this Apr 22, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[WIP] Refs #34239 - [rex_ssh] Install mosquitto with 'pull-mqtt' mode #726

[WIP] Refs #34239 - [rex_ssh] Install mosquitto with 'pull-mqtt' mode #726

wbclark commented Feb 9, 2022

ekohl left a comment

ekohl Feb 9, 2022

ehelms Mar 10, 2022

wbclark Mar 21, 2022

evgeni Mar 22, 2022

ekohl Feb 9, 2022

wbclark Mar 21, 2022

ekohl Feb 9, 2022

wbclark Feb 11, 2022

adamruzicka Feb 11, 2022

wbclark Mar 21, 2022

adamruzicka Mar 23, 2022

adamruzicka commented Feb 11, 2022

ekohl commented Feb 11, 2022

adamruzicka commented Feb 11, 2022

evgeni Mar 22, 2022

evgeni commented Mar 22, 2022

evgeni commented Mar 22, 2022 •

edited

Loading

ehelms commented Mar 22, 2022

adamruzicka commented Mar 22, 2022

evgeni commented Mar 23, 2022

adamruzicka commented Mar 23, 2022 •

edited

Loading

evgeni commented Mar 23, 2022 •

edited

Loading

ehelms commented Mar 23, 2022

evgeni commented Mar 23, 2022

evgeni commented Mar 23, 2022

ehelms commented Mar 23, 2022

evgeni commented Mar 23, 2022

evgeni commented Mar 25, 2022

ekohl commented Mar 25, 2022

ehelms commented Mar 25, 2022

ehelms commented Mar 26, 2022 •

edited

Loading

ehelms commented Apr 13, 2022

[WIP] Refs #34239 - [rex_ssh] Install mosquitto with 'pull-mqtt' mode #726

[WIP] Refs #34239 - [rex_ssh] Install mosquitto with 'pull-mqtt' mode #726

Conversation

wbclark commented Feb 9, 2022

ekohl left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

adamruzicka commented Feb 11, 2022

ekohl commented Feb 11, 2022

adamruzicka commented Feb 11, 2022

Choose a reason for hiding this comment

evgeni commented Mar 22, 2022

evgeni commented Mar 22, 2022 • edited Loading

ehelms commented Mar 22, 2022

adamruzicka commented Mar 22, 2022

evgeni commented Mar 23, 2022

adamruzicka commented Mar 23, 2022 • edited Loading

evgeni commented Mar 23, 2022 • edited Loading

ehelms commented Mar 23, 2022

evgeni commented Mar 23, 2022

evgeni commented Mar 23, 2022

ehelms commented Mar 23, 2022

evgeni commented Mar 23, 2022

evgeni commented Mar 25, 2022

ekohl commented Mar 25, 2022

ehelms commented Mar 25, 2022

ehelms commented Mar 26, 2022 • edited Loading

ehelms commented Apr 13, 2022

evgeni commented Mar 22, 2022 •

edited

Loading

adamruzicka commented Mar 23, 2022 •

edited

Loading

evgeni commented Mar 23, 2022 •

edited

Loading

ehelms commented Mar 26, 2022 •

edited

Loading