[Contribution] qvm-expose-port

[niccokunzmann]

Community Dev: @niccokunzmann
PoC: https://github.com/niccokunzmann/qvm-expose-port

---

Qubes OS version:

R3.2

Affected component(s):

dom0

---

Steps to reproduce the behavior:

Expected behavior:

I would like to expose a port from a VM to the outside.

Actual behavior:

I can read a long documentation and may fail.

General notes:

I implemented a script qvm-expose-port to do the work for me and
published it on GitHub.

I am offering it to you and would like to know if it is useful to add this to Qubes.
I am using it because I develop servers I would like to share.

---

Related issues:

[andrewdavidwong]

Thanks for your contribution! Please see the Package Contributions page for the package contribution procedure.

[tasket]

Not sure if its complementary, but @jpouellet also has a port-forwarding script:

"Forwards a specified port to a specified VM, auto-detecting its NetVM chain. (Qubes OS)"
https://gist.github.com/jpouellet/d8cd0eb8589a5b9bf0c53a28fc530369

[niccokunzmann]

@tasket Thanks! Posting this clarifies that I should use qvm-run -u root as pointed out in niccokunzmann/qvm-expose-port#1

Also, I like to point out that they know

• how to recurse
• how to get the IP without grep

[jpouellet]

If people actually want this, then IMO we should integrate it properly rather than as a fragile script in an optional package, and make it portable across different potential NetVMs/ProxyVMs (e.g. not make assumptions about the use or even availability of iptables).

I believe the correct way to implement this would be via a new "forward" action type for firewall rules. We already have infrastructure to handle reading/writing these rules, and the rules' intent are already provided to guests in a mostly-guest-OS-agnostic way via qubesdb, and we already have infrastructure in the linux agent for detecting changes to this exposed information and triggering idempotent iptables updates from them.

Some open questions are then:

• how to handle the case where multiple VMs both have rules requesting the same port be forwarded to them? One possible solution may be for adding a forwarding rule for a VM to remove all rules for the same port on other VMs with the same NetVM, but it's not clear to me that this is the best way.
• how to express mappings from one port to a different one (e.g. 80->8080)? The firewall rules at the moment only have a single port range field (internally referred to as dstports). Should we overload the semantics of a port range to mean lower bound is the port in the ingress NetVM side and the upper bound is the port on the VM providing the service? Should we add another field instead?

[jpouellet]

There is also the issue of direct recursive forwarding up the NetVM chain not being appropriate for all VMs. For example, if some VM is behind a ProxyVM which e.g. transparently tunnels all traffic at layer 2, then a port forward on that ProxyVM's NetVM does not make sense since the ports to be forwarded only exist on the overlay network (inside the proxy tunnel). Some way of denoting that forwarding should stop is required. It seems the introduction of a qvm-features flag would be a good way to express this. ProxyVMs which only act as firewalls (e.g. sys-firewall) could set this flag to mean "establish further forwarding firewall rules in NetVMs upstream of me", or not set it to not do so.

I believe the least-surprising-to-users way to introduce this would be a positive feature saying "do forward upstream" rather than a negative feature saying "do not foward upstream", and setting the feature by default on sys-firewall. This way people with existing ProxyVMs do not suddenly get ports forwarded from their NetVM to their ProxyVM as a surprise, but must be aware of this feature and manually opt in on the ProxyVMs for which it makes sense for them.

Following from this, creating a forwarding rule in a particular VM could automatically create corresponding forwarding rules in upstream NetVMs so long as they have the "auto-port-forward" (or "propagate-port-mappings", or whatever) flag set. This would eliminate the need for a separate port-forwarding tool altogether, and allow easy integration into firewall-management GUIs such as qubes-manager without duplicating netvm-recursion logic, and keeping all semantics contained in the core firewall handling logic.

[jpouellet]

OTOH, I suspect such a mechanism would be R4+ only since in R3.2 the firewall rules were not provided in a guest-agnostic format (although I suppose they could be shoved into the provided iptables rules anyway) and IIRC qube feature flags did not exist. So, I see nothing wrong with wanting to package up a short script without forward-compatibility guarantees for R3.2 users.

[jpouellet]

Shoutout to @Joeviocoe since (although I have not reviewed/audited it) his script is probably a better starting point than mine.

[niccokunzmann]

@jpouellet Since I am new to the code, I would appreciate more links to the source and implementation you are writing about. I can I agree because I do not know what you are talking about but I am never sure even if I found code. Can you create links to these?

• [Contribution] qvm-expose-port #4028 (comment)

  • We already have infrastructure to handle reading/writing these rules

  • qubesdb - (probably a repository)

Answers:

how to handle the case where multiple VMs both have rules requesting the same port be forwarded to them?

I would expect an error with no change being done, if I request to forward a port which is already forwarded.

how to express mappings from one port to a different one (e.g. 80->8080)?

This is not an important use case for now in my opinion. If one desires to re-map, one can get a proxy. The basic use-case is to allow people to expose ports in general. Once there are more people who have port conflicts, they can join in creating this more advanced case. However, if we create database entries and such, we can already keep in mind that the incoming port is not the outgoing port.

[jpouellet]

for qubesdb, see:

• https://www.qubes-os.org/doc/vm-interface/
• https://github.com/QubesOS/qubes-core-qubesdb
• $ qubesdb-list /
• $ qubesdb-read /qubes-ip (as a random example)

It's basically a hierarchical key-value store with roughly equivalent semantics to xenstore. If you're not familiar with it, it's pretty simple. Only thing that may be non-obvious is that updates are passed as messages, so you can observe writes even if they are the same value as was there previously. This behavior can be used for generic notification of events without polling at some interval or relying on another out-of-band mechanism, for example to notify that some other set of non-atomic changes has been completed and state is now consistent and can be acted on, for example in the case of firewall rules spread over multiple key/value pairs in the store.

Other functionality is built on top of this, for example getting the firewall rules from entries under /qubesdb-firewall/${vm_ip}/... in R4 (or /qubes-iptables-domainrules/${domain} in R3, still present for backwards compat in R4 but deprecated) and being notified that the rules have been updated and that the vm should install handle them somehow (e.g. install them into the kernel via iptables) via watching for empty writes to /qubesdb-firewall/${vm_ip} (or write of reload to /qubesdb-iptables in R3).

---

as for infrastructure for handling firewall rules, it's nothing special. I'm just referring to the fact that we already have a qvm-firewall tool, rules accessible under qubesadmin.Qubes().domains[somevm].firewall.rules, stored in /var/lib/qubes/appvms/somevm/firewall.xml (for now, maybe to be merged into qubes.xml at some point), modifyable via the qubes-manager gui (albeit with some limitations), etc.

---

Useful entry-points for reading the relevant code would be:

• the agent (not-dom0) side, responsible for grabbing rules from qubesdb and putting them into iptables/nftables is at https://github.com/QubesOS/qubes-core-agent-linux/blob/master/qubesagent/firewall.py
• the core qubes handling of per-appvm firewall rule state involves many moving parts but is mostly here: https://github.com/QubesOS/qubes-core-admin/blob/master/qubes/firewall.py
• the dom0 (or theoretically other AdminVM) qvm-firewall command line tool is here: https://github.com/QubesOS/qubes-core-admin-client/blob/master/qubesadmin/tools/qvm_firewall.py
• the firewall tab of the Qubes manager gui is here: https://github.com/QubesOS/qubes-manager/blob/master/qubesmanager/firewall.py

Thanks for being interested :)

[tlaurion]

I think the qvm-expose-port command should permit from which netvm (-s) on we want that port and protocol to be available.

The use case I have in mind is for the exposition of services only from a defined VPN netvm or sys-whonix for hidden tor services access, not opening everything up from the firewall-vm up to the destination qube.

example:
qvm-expose-port -a work 8000 -s sys-whonix

@jpouellet @niccokunzmann @tasket @Joeviocoe @daktak: which project aims to go mainstream and be packaged?

[niccokunzmann]

I think, no tool is ready to be packaged as is. Also, they solve our use-cases to I do not think I will put much time into it. If I were to put time into packaging this, I would need (a) to learn a lot about Qubes and (b) help on where to look for firewall configuration, and what it takes to become a package. Currently, I do not know much of any of this nor do I know of a mentor who is willing to give hints and some guidance. What I can help with is bash/Python a little IP-protocol, and maintainence of a collaborative repository with documentation and processes.

…

On 10/18/2018 06:13 PM, tlaurion wrote: I think the |qvm-expose-port|command should permit from which netvm (-s) on we want that port and protocol to be available. The use case I have in mind is for the exposition of services only from a defined VPN netvm or sys-whonix for hidden tor services, not opening everything up from the firewall-vm up to the destination qube. example: qvm-expose-port -a work 8000 -s sys-whonix @jpouellet <https://github.com/jpouellet> @niccokunzmann <https://github.com/niccokunzmann> @tasket <https://github.com/tasket> @Joeviocoe <https://github.com/Joeviocoe> @daktak <https://github.com/daktak>: which project aims to go mainstream and be packaged? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#4028 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAieIBevwNAQlarXbM-SC3KFoAnVxTv9ks5umKivgaJpZM4UzzeT>.

[zby]

I am not a bash programmer - so I am ready to remove this comment - but I think there is a major problem with this script.

There is an error function that is called whenever there is a serious error and it exits the script, but then the functions are called in backticks - so that exit is executed in a subshell (I think - I am not an expert here - but see below) and the main script does not exit on errors but goes on executing the rest possibly doing a lot of harm.

A simplified example:

#!/bin/bash

set -e

function error() {
  echo "ERROR: $@" >&2
  exit 1
  echo "AFTER EXIT"
}

function get_vm_ip() {
  error "SOME ERROR"
}

function expose_port() {
  local TARGET_VM_IP="`get_vm_ip`"
  echo "AFTER ERROR"
}

expose_port

exit 0

The effect of running this is:

user@personal:~/Downloads$ sh ~/QubesIncoming/dom0/aaa.sh 
ERROR: SOME ERROR
AFTER ERROR

Update: added 'set -e' line - but it did not change the output - as explained by marmarek below.

[marmarek]

In theory there is set -e, which should also terminate main script if any command (including backticks) fails, but usage of local bypass this effect.
So, right now it relies on exit_if_empty after each such assignment. One need to be very careful to not miss any such case...

[rustybird]

BTW this can be avoided by decoupling a local variable's declaration from its definition when the definition uses command substitution:

local foo
foo=`bar`

The same goes for export or any other command.

[Osndok]

If anyone is interested, I have started collecting & building upon the work presented here.

Of particular note:

• The exposed port can be made permanent with a "--perm" flag (it modifies the rc.local files in a clean-ish way)
• It's up-to-date (as presented, the other scripts did not work for me due to minor usage drift over time)

https://github.com/Osndok/qvm-expose-port/blob/master/qvm-expose-port

[fepitre]

Maybe we should do some common one with #5693. @marmarek

[Osndok]

Thanks @fepitre I did not see that. I've added those gists to an alt branch to track them, but I'm not a huge fan of putting a glob of unreadable iptables rules into the rc.local... it's one of those things where accidentally deleting a stray character will break things in ways that are nearly-unnoticable and/or hard to diagnose.

[tlaurion]

I would love to bring back to everybody's attention the work done under Qubesos Network server : https://github.com/Rudd-O/qubes-network-server

Having Qubesos firewall frontend to have rules -from, and cascading rules application from netvm to appvm would be totally awesome. Look at the issues there.

This project needs collaboration, which would ease user experience.

[fepitre]

@tlaurion I think the scope of those scripts is mostly for straightforward/temp. VM exposure. IMHO, using https://github.com/Rudd-O/qubes-network-server is much more for advanced/recurrent setup than simply exposing one port. I've encountered the case where I needed to easily forward external to multiples AppVMs and calling the qvm-portfwd-iptables tool was the easiest solution for me. Depending on available features and project status, it's certainly worth to add qubes-network-server to QubesOS-contrib.

[andrewdavidwong]

Please note that https://github.com/Rudd-O/qubes-network-server already has its own open issue: #5088

[Tehvan]

I'm using the 'in.sh' script by unman on R4.1.2.
https://github.com/unman/stuff/

I tried to use the script by fepitre (https://gist.github.com/fepitre/941d7161ae1150d90e15f778027e3248) but I couldn't get it working.

Update: Fepitre's script (with Istador's fix) is working fine and allows persistance. I just had to install net-tools in all the relevant qubes since ip is now the default package instead of net-tools.

[ben-grande]

R4.2 with nftables script based on Fepitre and previous contributors: qvm-port-forward, provided as is.

It could be improved... such as sanitizing and validating data coming from DomU and deal with disposable sys-net combined with disposable sys-firewall, which is not dealt with currently.