Merge #251: release notes and docs for 0.5.0

a475b3b release notes and docs for 0.5.0 (AdamISZ)

README.md

@@ -2,14 +2,20 @@
 
 Joinmarket refactored to separate client and backend operations
 
-**The [latest release](https://github.com/Joinmarket-Org/joinmarket-clientserver/releases) uses segwit wallets by default,
-you are strongly recommended not to change this, but you can use a non-segwit wallet by setting `segwit = false` in joinmarket.cfg.**
+Joinmarket is coinjoin software, and includes a wallet, which requires Bitcoin Core as backend (version 0.16.3, 0.17.0 or later).
 
-### Quickstart
+### Wallet features
 
-**For good quality privacy, performance and reliability, Bitcoin Core is required for Makers (yield generators), and recommended for takers (doing coinjoins). Use version 0.15+ ideally; 0.13.1+ required.**
+* Segwit addresses in the backward compatible form (start with `3`)
+* Multiple "mixdepths" or pockets (by default 5) for better coin isolation
+* Ability to spend directly, or with coinjoin; export private keys; BIP49 compatible seed (Trezor, Samourai etc.) and mnemonic extension option
+* Fine-grained control over bitcoin transaction fees
+* Can run sequence of coinjoins in automated form, either auto-generated (see `tumbler.py`) or self-generated sequence.
+* Can specify exact amount of coinjoin (figures from 0.01 to 30.0 btc and higher are practical), can choose time and number of counterparties
+* Can run passively to receive small payouts for taking part in coinjoins (see "Maker" and "yield-generator" in docs)
+* GUI to support Taker role, including tumbler/automated coinjoin sequence.
 
-For doing coinjoins (Taker side), you can also run without Bitcoin Core, connecting to Electrum servers.
+### Quickstart
 
 Once you've downloaded this repo, either as a zip file, and extracted it, or via `git clone`:
 
@@ -18,23 +24,20 @@ Once you've downloaded this repo, either as a zip file, and extracted it, or via
     source jmvenv/bin/activate
     cd scripts
 
-(You can omit `-p python3` if you want to use Python2)
+(You can omit `-p python3` if you want to use Python2. You can also add `--develop` as an extra flag to `install.sh` to make the Joinmarket code editable in-place.)
 
 You should now be able to run the scripts like `python wallet-tool.py` etc., just as you did in the previous Joinmarket version.
 
 Alternative to this "quickstart" (including for MacOS): follow the [install guide](docs/INSTALL.md).
 
-### Upgrade for segwit
-
-See the [segwit upgrade guide](docs/SEGWIT-UPGRADE.md) if you need to update your wallet.
 
 ### Usage
 
 If you are new, follow and read the links in the [usage guide](docs/USAGE.md).
 
 If you are running Joinmarket-Qt, you can instead use the [walkthrough](docs/JOINMARKET-QT-GUIDE.md) to start.
 
-If you are not new to Joinmarket, the notes in the [scripts readme](scripts/README.md) help to understand what has and hasn't changed about the scripts.
+If you used the old version of Joinmarket, the notes in the [scripts readme](scripts/README.md) help to understand what has and hasn't changed about the scripts.
 
 ### Joinmarket-Qt
 

docs/TODO.md

@@ -5,9 +5,9 @@ The Issues list is for specific bugs or feature requests.
 
 * PEP8 compliance.
 * Details which may or may not be included in PEP8 might be, consistent variable naming conventions, and use of single/double quotes.
-* Porting to Python 3.
+* ~~Porting to Python 3~~. This is done in that we are now Py2 and Py3 compatible as of 0.5.0; but we may deprecate Py2 soon.
 
-A note on the above - took a look at it last December, but had problems in particular with some twisted elements, specifically `txsocksx`
+~~A note on the above - took a look at it last December, but had problems in particular with some twisted elements, specifically `txsocksx`~~ Done as of 0.4.2, now switched to txtorcon.
 
 * Twisted related: there are cases where much better use of deferreds should be possible.
 
@@ -38,7 +38,8 @@ concern (there is already no bitcoin security concern even without it).
 * We should look into lite-client modes, in particular client-side filtering as used by Neutrino and others,
 and described [here](https://github.com/Roasbeef/bips/blob/master/gcs_light_client.mediawiki}.
 
-* ~~Re-work the existing electrum code so it works reliably and with some decent performance (easier short term goal).~~ (Done)
+* Re-work the existing electrum code so it works reliably and with some decent performance (easier short term goal).
+This was previously marked 'done' but is now very much "un-done" since the code has not been updated and doesn't work; it's debatable what should be done, if anything, about it. It's certainly useful to have Electrum for testing/testnet.
 
 ### Joinmarket protocol
 
@@ -50,14 +51,14 @@ and described [here](https://github.com/Roasbeef/bips/blob/master/gcs_light_clie
 
 There is no current process for building binaries on Mac or Windows (theoretically the latter is possible but a mess, so I'm not doing it).
 
-* Qt5 support, as Qt4 is being deprecated (see [issue204](https://github.com/JoinMarket-Org/joinmarket-clientserver/issues/204)).
+* ~~Qt5 support, as Qt4 is being deprecated (see [issue204](https://github.com/JoinMarket-Org/joinmarket-clientserver/issues/204)).~~ Done as of 0.5.0.
 
 ### Alternative implementations
 
 * Build an alternative client implementation in Java or Javascript for example, using some existing Bitcoin library in that language, perhaps using some lite client solution e.g. SPV, and then connecting to the daemon (executable or Python script).
 
 ### Bitcoin
 
-* Several related concepts here: 1/ switching to Peter Todd's python-bitcoinlib, but want good backend i.e. libsecp256k1 binding.
+* ~~Several related concepts here: 1/ switching to Peter Todd's python-bitcoinlib, but want good backend i.e. libsecp256k1 binding.
 2/ coincurve of Ofek looks like a better (and better-maintained) option than secp256k1-py, so ideally put that behind python-bitcoinlib.
-3/ Also doing all this without Python3 usage doesn't seem to make much sense.
+3/ Also doing all this without Python3 usage doesn't seem to make much sense.~~ The story is probably not over yet, but we now use coincurve and have made some alterations to use the same logic as the now un-maintained python-bitcoinlib. There will be more changes to Bitcoin including better segwit support. There is still much to do here but it'll probably be patchwork changes rather than a further complete rewrite.

docs/release-notes/release-notes-0.5.0.md

@@ -0,0 +1,92 @@
+Joinmarket-clientserver 0.5.0:
+=================
+
+<https://github.com/joinmarket-org/joinmarket-clientserver/releases/tag/v0.5.0>
+
+This release is mostly about "modernisation" - as of this release, Joinmarket **can be run under Python3** (including the GUI JoinmarketQt), as well as updating the binding to libsecp256k1. Big thanks to @jameshilliard for taking the initiative on getting this working.
+
+However there also some bugfixes which prevent possible crash conditions, so you're recommended to upgrade immediately. Use of Python3 is optional; see the "Upgrading" note below on how to choose it.
+
+Please note that although there is no current fixed timeline to deprecate Python2 entirely, it will be done at some point.
+
+If you are upgrading from pre-0.4.0 you **must** read the [release notes for 0.4.0](https://github.com/JoinMarket-Org/joinmarket-clientserver/blob/master/docs/release-notes/release-notes-0.4.0.md) and follow
+the relevant upgrade instructions, which apply here exactly the same.
+
+Please report bugs using the issue tracker at github:
+
+<https://github.com/joinmarket-org/joinmarket-clientserver/issues>
+
+Upgrading 
+=========
+
+To upgrade (but: see note above if from pre-0.4.0):
+
+First run the `install.sh` script as mentioned in the README. When prompted to overwrite the directory `jmvenv`, accept.
+To install using Python3, use `./install.sh -p python3` ; for now the default is still Python2.
+
+If you are running JoinmarketQt, you *must* choose this Python3 option; see the README for the extra steps required.
+
+Notable changes
+===============
+
+### Making the codebase compatible with Python3 usage
+
+Here the python `future` package (see [here](https://python-future.org/)) has been used to allow the code to be compatible with Python 2.7 (in all earlier releases, only this version of Python was supported) and Python 3 (at least, 3.5 and 3.6, other versions of Py3 not tested). See above for the installation flags needed, and to repeat the previous note, Py3 is, from this release, required for the JoinmarketQt gui.
+
+`3537fb4` Convert jmbitcoin to py3 style
+`214dbf3` override system libsecp256k1 for coincurve
+`6dc3504` Convert jmclient to py3 style
+`8e5826d` Convert JoinmarketQt to PySide2
+`9ce9ee5` Fix encode crash on IRC notice
+
+### Fix crash conditions that may create DOS vectors for Makers/Takers
+
+These edge conditions were discovered by @undeath and myself during the process of auditing the code that was necessary for the Python3 compatibility upgrade described above. They are all cases where some kinds of crafted input from a counterparty could cause the raise of an un-caught exception, resulting in the Maker or Taker process crashing (hence DOS vectors). They are not coin security risks. Nevertheless immediate upgrade, by Makers particularly (Taker crashes aren't really an issue), is recommended to make these impossible.
+
+`979480e` fix possible crash on tx with odd scripts
+`b112576` Validate maker destination addresses
+`be6735d` Handle invalid utxos in query_utxo_set
+
+### Use coincurve as new binding to libsecp256k1
+
+Previously we were using the Python binding to [libsecp256k1](https://github.com/bitcoin-core/secp256k1) located [here](https://github.com/ludbb/secp256k1-py).
+Now we have switched to [coincurve](https://github.com/ofek/coincurve) because it is actively maintained.
+The changes to Joinmarket (as you can see in the below commit) are extremely minimal, as the interface is very close to the same.
+We have audited and tested this change extensively, since it affects the core Bitcoin signing operations.
+
+`abad597` Replace secp256k1-py with coincurve
+
+
+### Minor technical changes
+
+Will not be of interest to non-developers (these are mostly changes to tests/very minor refactoring).
+
+`d353b5f` remove gpg verify disable flag from docker scripts
+`fdc6687` remove gpg signature validation from install.sh
+`5a0d5fe` Revert "Revert "Python 3 style conversion""
+`8bcaf36` add full coinjoin test
+`647335e` add undeath's pgp key
+`249d547` clean up test_configure
+`25ca6d2` testing: allow auditing maker wallets in manual test
+`99ac1ca` update TESTING.md with latest test syntax
+`512cddd` fix test_net_byte
+`46e61f3` Fix bugs in bigstring and test_commands
+`a43eceb` add very basic JMMakerClientProtocol test, fixes: 3ae5bff, 12705b5 
+`d33cc5c` clean up Maker interface
+`42060db` remove slowaes
+`19fbef1` Fix nondeterministic failure in test tumbler tweak
+`1033b64` remove unused locks from global config
+
+Credits
+=======
+
+Thanks to everyone who directly contributed to this release -
+
+- @jameshilliard
+- @undeath
+- @AdamISZ
+- @fivepiece
+
+And thanks also to those who submitted bug reports, tested and otherwise helped out.
+
+

jmbase/setup.py

@@ -2,7 +2,7 @@
 
 
 setup(name='joinmarketbase',
-      version='0.4.2',
+      version='0.5.0',
       description='Joinmarket client library for Bitcoin coinjoins',
       url='http://github.com/Joinmarket-Org/joinmarket-clientserver/jmbase',
       author='',

jmbitcoin/setup.py

@@ -2,7 +2,7 @@
 
 
 setup(name='joinmarketbitcoin',
-      version='0.4.2',
+      version='0.5.0',
       description='Joinmarket client library for Bitcoin coinjoins',
       url='http://github.com/Joinmarket-Org/joinmarket-clientserver/jmbitcoin',
       author='',

jmclient/setup.py

@@ -2,14 +2,14 @@
 
 
 setup(name='joinmarketclient',
-      version='0.4.2',
+      version='0.5.0',
       description='Joinmarket client library for Bitcoin coinjoins',
       url='http://github.com/Joinmarket-Org/joinmarket-clientserver/jmclient',
       author='',
       author_email='',
       license='GPL',
       packages=['jmclient'],
       install_requires=['future', 'configparser;python_version<"3.2"',
-                        'joinmarketbase==0.4.2', 'mnemonic', 'argon2_cffi',
+                        'joinmarketbase==0.5.0', 'mnemonic', 'argon2_cffi',
                         'bencoder.pyx', 'pyaes'],
       zip_safe=False)

jmdaemon/setup.py

@@ -2,12 +2,12 @@
 
 
 setup(name='joinmarketdaemon',
-      version='0.4.2',
+      version='0.5.0',
       description='Joinmarket client library for Bitcoin coinjoins',
       url='http://github.com/Joinmarket-Org/joinmarket-clientserver/jmdaemon',
       author='',
       author_email='',
       license='GPL',
       packages=['jmdaemon'],
-      install_requires=['future', 'txtorcon', 'pyopenssl', 'libnacl', 'joinmarketbase==0.4.2'],
+      install_requires=['future', 'txtorcon', 'pyopenssl', 'libnacl', 'joinmarketbase==0.5.0'],
       zip_safe=False)

scripts/joinmarket-qt.py

@@ -60,9 +60,9 @@
 donation_address = "1AZgQZWYRteh6UyF87hwuvyWj73NvWKpL"
 
 #Underlying joinmarket code version (as per setup.py etc.)
-JM_CORE_VERSION = '0.4.2'
+JM_CORE_VERSION = '0.5.0'
 #Version of this Qt script specifically
-JM_GUI_VERSION = '7'
+JM_GUI_VERSION = '8'
 
 from jmclient import load_program_config, get_network,\
     open_test_wallet_maybe, get_wallet_path, get_p2sh_vbyte, get_p2pk_vbyte,\