Merge #1182: Only onion message channels

5cc1695 Disconnection logic fixes, add btcnet to handshake (Adam Gibson)
830ac22 Allow taker peers to not serve onions + bugfixes. (Adam Gibson)
fd550ee Onion-based message channels with directory nodes (Adam Gibson)

docs/onion-message-channels.md

@@ -0,0 +1,209 @@
+# HOW TO SETUP ONION MESSAGE CHANNELS IN JOINMARKET
+
+### Contents
+
+1. [Overview](#overview)
+
+2. [Testing, configuring for signet](#testing)
+
+4. [Directory nodes](#directory)
+
+<a name="overview" />
+
+## Overview
+
+This is a new way for Joinmarket bots to communicate, namely by serving and connecting to Tor onion services. This does not
+introduce any new requirements to your Joinmarket installation, technically, because the use of Payjoin already required the need
+to run such onion services, and connecting to IRC used a SOCKS5 proxy (used by almost all users) over Tor to
+a remote onion service.
+
+(Note however that taker bots will *not* be required to serve onions; they will only make outbound SOCKS connections, as they currently do on IRC).
+
+The purpose of this new type of message channel is as follows:
+
+* less reliance on any service external to Joinmarket
+* most of the transaction negotiation will be happening directly peer to peer, not passed over a central server (
+albeit it was and remains E2E encrypted data, in either case)
+* the above can lead to better scalability at large numbers
+* a substantial increase in the speed of transaction negotiation; this is mostly related to the throttling of high bursts of traffic on IRC
+
+The configuration for a user is simple; in their `joinmarket.cfg` they will get a new `[MESSAGING]` section like this, if they start from scratch:
+
+```
+[MESSAGING:onion]
+# onion based message channels must have the exact type 'onion'
+# (while the section name above can be MESSAGING:whatever), and there must
+# be only ONE such message channel configured (note the directory servers
+# can be multiple, below):
+type = onion
+
+socks5_host = localhost
+socks5_port = 9050
+
+# the tor control configuration.
+# for most people running the tor daemon
+# on Linux, no changes are required here:
+tor_control_host = localhost
+# or, to use a UNIX socket
+# tor_control_host = unix:/var/run/tor/control
+tor_control_port = 9051
+
+# the host/port actually serving the hidden service
+# (note the *virtual port*, that the client uses,
+# is hardcoded to 80):
+onion_serving_host = 127.0.0.1
+onion_serving_port = 8080
+
+# directory node configuration
+#
+# This is mandatory for directory nodes (who must also set their
+# own *.onion:port as the only directory in directory_nodes, below),
+# but NOT TO BE USED by non-directory nodes (which is you, unless
+# you know otherwise!), as it will greatly degrade your privacy.
+# (note the default is no value, don't replace it with "").
+hidden_service_dir =
+#
+# This is a comma separated list (comma can be omitted if only one item).
+# Each item has format host:port ; both are required, though port will
+# be 80 if created in this code.
+# for MAINNET:
+directory_nodes = 3kxw6lf5vf6y26emzwgibzhrzhmhqiw6ekrek3nqfjjmhwznb2moonad.onion,qqd22cwgygaxcy6vdw6mzwkyaxg5urb4ptbc5d74nrj25phspajxjbqd.onion
+
+# for SIGNET (testing network):
+# directory_nodes = rr6f6qtleiiwic45bby4zwmiwjrj3jsbmcvutwpqxjziaydjydkk5iad.onion:80,k74oyetjqgcamsyhlym2vgbjtvhcrbxr4iowd4nv4zk5sehw4v665jad.onion:80
+
+# This setting is ONLY for developer regtest setups,
+# running multiple bots at once. Don't alter it otherwise
+regtest_count = 0,0
+```
+
+All of these can be left as default for most users - but most importantly, pay attention to:
+
+* The list of `directory_nodes`, which will be comma separated if multiple directory nodes are configured (we expect there will be 2 or 3 as a normal situation). Make sure to choose the ones for your network (mainnet by default, or signet or otherwise); if it's wrong your bot will just get auto-disconnected.
+* The `onion_serving_port` is the port on the local machine on which the onion service is served; you won't usually need to use it, but it mustn't conflict with some other usage (so if you have something running on port 8080, change it).
+The `type` field must always be `onion` in this case, and distinguishes it from IRC message channels and others.
+
+### Can/should I still run IRC message channels?
+
+In short, yes, at least for now, though you are free to disable any message channel you like.
+
+### Do I need to configure Tor, and if so, how?
+
+To make outbound Tor connections to other onions in the network, you will need to configure the
+SOCKS5 proxy settings (so, only directory nodes may *not* need this; everyone else does).
+This is identical to what we already do for IRC, except that in this case, we disallow clearnet connections.
+
+#### Running/testing as a maker
+
+A maker will additionally allow *inbound* connections to an onion service.
+This onion service will be ephemeral, that is, it will have a different onion address every time
+you restart. This should work automatically, using your existing Tor daemon (here, we are using
+the same code as we use when running the `receive-payjoin` script, essentially).
+
+#### Running/testing as other bots (taker, ob-watcher)
+
+A taker will not attempt to serve an onion; it will only use outbound connections, first to directory
+nodes and then, as according to need, to individual makers, also.
+
+As previously mentioned, both of these features - inbound and outbound, to onion, Tor connections - were already in use in Joinmarket. If you want to run/test as a maker bot, but never served an onion service before, it should work fine as long as you have the Tor service running in the background,
+and the default control port 9051 (if not, change that value in the `joinmarket.cfg`, see above).
+
+#### Why not use Lightning based onions?
+
+(*Feel free to skip this section if you don't know what "Lightning based onions" refers to!*). The reason this architecture is
+proposed as an alternative to the previously suggested Lightning-node-based network (see
+[this PR](https://github.com/JoinMarket-Org/joinmarket-clientserver/pull/1000)), is mostly that:
+
+* the latter has a bunch of extra installation and maintenance dependencies (just one example: pyln-client requires coincurve, which we just
+removed)
+* the latter requires establishing a new node "identity" which can be refreshed, but that creates more concern
+* longer term ideas to integrate Lightning payments to the coinjoin workflow (and vice versa!) are not realizable yet
+* using multi-hop onion messaging in the LN network itself is also a way off, and a bit problematic
+
+So the short version is: the Lightning based alternative is certainly feasible, but has a lot more baggage that can't really be justified
+unless we're actually using it for something.
+
+
+<a name="testing" />
+
+## Testing, and configuring for signet.
+
+This testing section focuses on signet since that will be the less troublesome way of getting involved in tests for
+the non-hardcore JM developer :)
+
+(For the latter, please use the regtest setup by running `test/e2e-coinjoin-test.py` under `pytest`,
+and pay attention to the settings in `regtest_joinmarket.cfg`.)
+
+There is no separate/special configuration for signet other than the configuration that is already needed for running
+Joinmarket against a signet backend (so e.g. RPC port of 38332).
+
+You can just uncomment the `directory_nodes` entry listed as SIGNET, and comment out the one for MAINNET.
+
+Then just make sure your bot has some signet coins and try running as maker or taker or both.
+
+<a name="directory" />
+
+## Directory nodes
+
+**This last section is for people with a lot of technical knowledge in this area,
+who would like to help by running a directory node. You can ignore it if that does not apply.**.
+
+This requires a long running bot. It should be on a server you can keep running permanently, so perhaps a VPS,
+but in any case, very high uptime. For reliability it also makes sense to configure to run as a systemd service.
+
+The currently suggested way to run a directory node is to use the script found [here](https://github.com/JoinMarket-Org/custom-scripts/blob/0eda6154265e012b907c43e2ecdacb895aa9e3ab/start-dn.py); you can place it in your `joinmarket-clientserver/scripts` directory and run it *without* arguments, but with one option flag: `--datadir=/your/chosen/datadir` (as you'll see below).
+
+This slightly unobvious approach is based on the following ideas: we run a Joinmarket script, with a Joinmarket python virtualenv, so that we are able to parse messages; this means that the directory node *can* be a bot, e.g. a maker bot, but need not be - and here it is basically a "crippled" maker bot that cannot do anything. This 'crippling' is actually very useful because (a) we use the `no-blockchain` argument (it is forced in-code; you don't need to set it) so we don't need a running Bitcoin node (of whatever flavour), and (b) we don't need a wallet either.
+
+#### Joinmarket-specific configuration
+
+Add a non-empty `hidden_service_dir` entry to your `[MESSAGING:onion]` with a directory accessible to your user. You may want to lock this down
+a bit, but be careful changing permissions from what is created by the script, because Tor is very finicky about this.
+
+The hostname for your onion service will not change and will be stored permanently in that directory.
+
+The point to understand is: Joinmarket's `jmbase.JMHiddenService` will, if configured with a non-empty `hidden_service_dir`
+field, actually start an *independent* instance of Tor specifically for serving this, under the current user.
+(our Tor interface library `txtorcon` needs read access to the Tor HS dir, so it's troublesome to do this another way).
+
+##### Question: How to configure the `directory-nodes` list in our `joinmarket.cfg` for this directory node bot?
+
+Answer: **you must only enter your own node in this list!**. This way your bot will recognize that it is a directory node and it avoids weird edge case behaviour (so don't add *other* known directory nodes; you won't be talking to them).
+
+A natural retort is: but I don't know my own node's onion service hostname before I start it the first time. Indeed. So, just run it once with the default `directory_nodes` entries, then note down the new onion service hostname you created, and insert that as the only entry in the list.
+
+
+#### Suggested setup of a systemd service:
+
+The most basic bare-bones service seems to work fine here:
+
+```
+[Unit]
+Description=My JM signet directory node
+Requires=network-online.target
+After=network-online.target
+
+[Service]
+Type=simple
+ExecStart=/bin/bash -c 'cd /path/to/joinmarket-clientserver && source jmvenv/bin/activate && cd scripts && python start-dn.py --datadir=/path/to/chosen/datadir'
+User=user
+
+[Install]
+WantedBy=multi-user.target
+```
+
+... however, you need to kind of 'bootstrap' it the first time. For example:
+
+* run once with systemctl start
+
+* look at log with `journalctl`, service fails due to default `joinmarket.cfg` and quit.
+* go to that cfg file. Remove the IRC settings, they serve no purpose here. Change the `hidden_service_dir` to `/yourlocation/hidserv` (the actual directory need not exist, it's better if it doesn't, this first time). Edit the `network` field in `BLOCKCHAIN` to whatever network (mainnet, signet) you intend to support - it can be only one for one directory node, for now.
+
+* `systemctl start` again, now note the onion hostname created from the log or the directory
+
+* set that hostname in `directory_nodes` in `joinmarket.cfg`
+
+* now the service should start correctly
+
+TODO: add some material on network hardening/firewalls here, I guess.
+

jmbase/jmbase/commands.py

@@ -27,11 +27,11 @@ class JMInit(JMCommand):
     """Communicates the client's required setup
     configuration.
     Blockchain source is communicated only as a naming
-    tag for messagechannels (currently IRC 'realname' field).
+    tag for messagechannels (for IRC, 'realname' field).
     """
     arguments = [(b'bcsource', Unicode()),
                  (b'network', Unicode()),
-                 (b'irc_configs', JsonEncodable()),
+                 (b'chan_configs', JsonEncodable()),
                  (b'minmakers', Integer()),
                  (b'maker_timeout_sec', Integer()),
                  (b'dust_threshold', Integer()),

jmbase/jmbase/twisted_utils.py

@@ -128,16 +128,23 @@ def config_to_hs_ports(virtual_port, host, port):
 class JMHiddenService(object):
     """ Wrapper class around the actions needed to
     create and serve on a hidden service; an object of
-    type Resource must be provided in the constructor,
-    which does the HTTP serving actions (GET, POST serving).
+    type either Resource or server.ProtocolFactory must
+    be provided in the constructor, which does the HTTP
+    (GET, POST) or other protocol serving actions.
     """
-    def __init__(self, resource, info_callback, error_callback,
-                 onion_hostname_callback, tor_control_host,
+    def __init__(self, proto_factory_or_resource, info_callback,
+                 error_callback, onion_hostname_callback, tor_control_host,
                  tor_control_port, serving_host, serving_port,
-                 virtual_port = None,
-                 shutdown_callback = None):
-        self.site = Site(resource)
-        self.site.displayTracebacks = False
+                 virtual_port=None,
+                 shutdown_callback=None,
+                 hidden_service_dir=""):
+        if isinstance(proto_factory_or_resource, Resource):
+            # TODO bad naming, in this case it doesn't start
+            # out as a protocol factory; a Site is one, a Resource isn't.
+            self.proto_factory = Site(proto_factory_or_resource)
+            self.proto_factory.displayTracebacks = False
+        else:
+            self.proto_factory = proto_factory_or_resource
         self.info_callback = info_callback
         self.error_callback = error_callback
         # this has a separate callback for convenience, it should
@@ -155,26 +162,45 @@ def __init__(self, resource, info_callback, error_callback,
         # config object, so no default here:
         self.serving_host = serving_host
         self.serving_port = serving_port
+        # this is used to serve an onion from the filesystem,
+        # NB: Because of how txtorcon is set up, this option
+        # uses a *separate tor instance* owned by the owner of
+        # this script (because txtorcon needs to read the
+        # HS dir), whereas if this option is "", we set up
+        # an ephemeral HS on the global or pre-existing tor.
+        self.hidden_service_dir = hidden_service_dir
 
     def start_tor(self):
         """ This function executes the workflow
         of starting the hidden service and returning its hostname
         """
         self.info_callback("Attempting to start onion service on port: {} "
                            "...".format(self.virtual_port))
-        if str(self.tor_control_host).startswith('unix:'):
-            control_endpoint = UNIXClientEndpoint(reactor,
-                                    self.tor_control_host[5:])
+        if self.hidden_service_dir == "":
+            if str(self.tor_control_host).startswith('unix:'):
+                control_endpoint = UNIXClientEndpoint(reactor,
+                                        self.tor_control_host[5:])
+            else:
+                control_endpoint = TCP4ClientEndpoint(reactor,
+                                self.tor_control_host, self.tor_control_port)
+            d = txtorcon.connect(reactor, control_endpoint)
+            d.addCallback(self.create_onion_ep)
+            d.addErrback(self.setup_failed)
+            # TODO: add errbacks to the next two calls in
+            # the chain:
+            d.addCallback(self.onion_listen)
+            d.addCallback(self.print_host)
         else:
-            control_endpoint = TCP4ClientEndpoint(reactor,
-                            self.tor_control_host, self.tor_control_port)
-        d = txtorcon.connect(reactor, control_endpoint)
-        d.addCallback(self.create_onion_ep)
-        d.addErrback(self.setup_failed)
-        # TODO: add errbacks to the next two calls in
-        # the chain:
-        d.addCallback(self.onion_listen)
-        d.addCallback(self.print_host)
+            ep = "onion:" + str(self.virtual_port) + ":localPort="
+            ep += str(self.serving_port)
+            # endpoints.TCPHiddenServiceEndpoint creates version 2 by
+            # default for backwards compat (err, txtorcon needs to update that ...)
+            ep += ":version=3"
+            ep += ":hiddenServiceDir="+self.hidden_service_dir
+            onion_endpoint = serverFromString(reactor, ep)
+            d = onion_endpoint.listen(self.proto_factory)
+            d.addCallback(self.print_host_filesystem)
+
 
     def setup_failed(self, arg):
         # Note that actions based on this failure are deferred to callers:
@@ -195,7 +221,8 @@ def onion_listen(self, onion):
         serverstring = "tcp:{}:interface={}".format(self.serving_port,
                                                     self.serving_host)
         onion_endpoint = serverFromString(reactor, serverstring)
-        return onion_endpoint.listen(self.site)
+        print("created the onion endpoint, now calling listen")
+        return onion_endpoint.listen(self.proto_factory)
 
     def print_host(self, ep):
         """ Callback fired once the HS is available
@@ -204,6 +231,14 @@ def print_host(self, ep):
         """
         self.onion_hostname_callback(self.onion.hostname)
 
+    def print_host_filesystem(self, port):
+        """ As above but needed to respect slightly different
+        callback chain for this case (where we start our own tor
+        instance for the filesystem-based onion).
+        """
+        self.onion = port.onion_service
+        self.onion_hostname_callback(self.onion.hostname)
+
     def shutdown(self):
         self.tor_connection.protocol.transport.loseConnection()
         self.info_callback("Hidden service shutdown complete")

jmbase/test/test_commands.py

@@ -43,9 +43,9 @@ def end_test():
 class JMTestServerProtocol(JMBaseProtocol):
 
     @JMInit.responder
-    def on_JM_INIT(self, bcsource, network, irc_configs, minmakers,
+    def on_JM_INIT(self, bcsource, network, chan_configs, minmakers,
                    maker_timeout_sec, dust_threshold, blacklist_location):
-        show_receipt("JMINIT", bcsource, network, irc_configs, minmakers,
+        show_receipt("JMINIT", bcsource, network, chan_configs, minmakers,
                      maker_timeout_sec, dust_threshold, blacklist_location)
         d = self.callRemote(JMInitProto,
                             nick_hash_length=1,
@@ -137,7 +137,7 @@ def clientStart(self):
         d = self.callRemote(JMInit,
                             bcsource="dummyblockchain",
                             network="dummynetwork",
-                            irc_configs=['dummy', 'irc', 'config'],
+                            chan_configs=['dummy', 'irc', 'config'],
                             minmakers=7,
                             maker_timeout_sec=8,
                             dust_threshold=1500,

jmclient/jmclient/__init__.py

@@ -24,7 +24,7 @@
                            TYPE_P2PKH, TYPE_P2SH_P2WPKH, TYPE_P2WPKH, detect_script_type)
 from .configure import (load_test_config, process_shutdown,
     load_program_config, jm_single, get_network, update_persist_config,
-    validate_address, is_burn_destination, get_irc_mchannels,
+    validate_address, is_burn_destination, get_mchannels,
     get_blockchain_interface_instance, set_config, is_segwit_mode,
     is_native_segwit_mode, JMPluginService, get_interest_rate,
     get_bondless_makers_allowance, check_and_start_tor)
@@ -33,7 +33,7 @@
 from .snicker_receiver import SNICKERError, SNICKERReceiver
 from .client_protocol import (JMTakerClientProtocol, JMClientProtocolFactory,
                               start_reactor, SNICKERClientProtocolFactory,
-                              BIP78ClientProtocolFactory,
+                              BIP78ClientProtocolFactory, JMMakerClientProtocol,
                               get_daemon_serving_params)
 from .podle import (set_commitment_file, get_commitment_file,
                     add_external_commitments,