Merge branch 'master' of github.com:hyperledger/indy-node into docs-f…

…eature

Jenkinsfile.cd

@@ -27,29 +27,6 @@ def nodeTestUbuntu = {
     }
 }
 
-def clientTestUbuntu = {
-    try {
-        echo 'Ubuntu Test: Checkout csm'
-        checkout scm
-
-        echo 'Ubuntu Test: Build docker image'
-        def testEnv = dockerHelpers.build(name)
-
-        testEnv.inside('--network host') {
-            echo 'Ubuntu Test: Install dependencies'
-            testHelpers.install()
-
-            echo 'Ubuntu Test: Test'
-            testHelpers.testRunner([resFile: "test-result-client.${NODE_NAME}.txt", testDir: 'indy_client'])
-            //testHelpers.testJUnit(resFile: "test-result-client.${NODE_NAME}.xml")
-        }
-    }
-    finally {
-        echo 'Ubuntu Test: Cleanup'
-        step([$class: 'WsCleanup'])
-    }
-}
-
 def commonTestUbuntu = {
     try {
         echo 'Ubuntu Test: Checkout csm'
@@ -91,4 +68,4 @@ options = new TestAndPublishOptions()
 options.enable([StagesEnum.PACK_RELEASE_COPY, StagesEnum.PACK_RELEASE_COPY_ST])
 options.setCopyWithDeps(true)
 options.setPrContexts([env.INDY_GITHUB_PR_REQUIRED_CONTEXT ?: "ci/hyperledger-jenkins/pr-merge"])
-testAndPublish(name, [ubuntu: [node: nodeTestUbuntu, client: clientTestUbuntu, common: commonTestUbuntu]], true, options, [ubuntu: buildDebUbuntu])
+testAndPublish(name, [ubuntu: [node: nodeTestUbuntu, common: commonTestUbuntu]], true, options, [ubuntu: buildDebUbuntu])

Jenkinsfile.ci

@@ -142,14 +142,6 @@ def tests = [
             python: python
         )
     },
-    client: { python ->
-        test(
-            resFile: "test-result-client.${NODE_NAME}.txt",
-            testDir: 'indy_client',
-            python: python,
-            useRunner: true
-        )
-    },
     node: { python ->
         test(
             resFile: "test-result-node.${NODE_NAME}.txt",

README.md

@@ -69,15 +69,10 @@ Indy Node repo consists of the following parts:
       So, if you want to work with Indy Node, you will need to have the Plenum code as well in most of the cases
       and work with two projects at the same time 
       (see [How to Start Working with the Code](#how-to-start-working-with-the-code) below).
-- [indy-anoncreds](https://github.com/hyperledger/indy-anoncreds) 
-    - A python implementation of the anonymous credentials ideas developed by IBM Research.
-    - This is quite independent from indy-node/plenum. So, in most cases you don't need this code to contribute to Indy-Node.
-    - It will be deprecated soon in favor of anoncreds implementation in indy-sdk (see below). 
 - [indy-sdk](https://github.com/hyperledger/indy-sdk)
     - An official SDK for Indy.
     - it contains client and anoncreds implementation
     - You don't need it to contribute to Indy-Node. But please use indy-sdk for your own applications dealing with Indy ecosystem.
-    - It will replace indy-client and indy-anoncreds parsts soon.
 - [indy-crypto](https://github.com/hyperledger/indy-crypto)
     - A shared crypto library 
     - It's based on [AMCL](https://github.com/milagro-crypto/amcl)
@@ -119,14 +114,6 @@ The described process is automated in one of the ways below (it allow to install
 Please have a look at [Dev Setup](docs/setup-dev.md)
 
 
-## How to Start Indy Client CLI
-Once installed, you can play with the command-line interface by running Indy from a terminal.
-
-Note: For Windows, we recommended using either [cmder](http://cmder.net/) or [conemu](https://conemu.github.io/).
-```
-indy
-```
-
 ## Continuous Integration and Delivery
 
 Please have a look at [Continuous integration/delivery](docs/ci-cd.md)
@@ -167,10 +154,12 @@ If you made changes in both indy-plenum and indy-node, you need to do the follow
 - Please have a look at documents and diagrams in [docs](docs) folder
 - Please have a look at documents and diagrams in Plenum's [docs](https://github.com/hyperledger/indy-plenum/tree/master/docs) folder:
     - [Technical Overview of Plenum](https://github.com/hyperledger/indy-plenum/blob/master/docs/main.md)
+    - [Plenum Consensus Algorithm Diagram](https://github.com/hyperledger/indy-plenum/blob/master/docs/diagrams/consensus-protocol.png)
     - [Glossary](https://github.com/hyperledger/indy-plenum/blob/master/docs/glossary.md)
     - [Storages](https://github.com/hyperledger/indy-plenum/blob/master/docs/storage.md)
     - [Request Handling](https://github.com/hyperledger/indy-plenum/blob/master/docs/request_handling.md)
     - [Catchup](https://github.com/hyperledger/indy-plenum/blob/master/docs/catchup.md)
+    - [Catchup Diagram](https://github.com/hyperledger/indy-plenum/blob/master/docs/diagrams/catchup-procedure.png)
     - [Plugins](https://github.com/hyperledger/indy-plenum/blob/master/docs/plugins.md)
 - Relationship between Entities and Transactions: [relationship diagram](docs/relationship-diagram.png)
 - Supported transactions and their format: [transactions](docs/transactions.md)

build-scripts/ubuntu-1604/postinst_node

@@ -15,7 +15,7 @@ INSTALL_DIR='/usr/local/lib/python3.5/dist-packages'
 NOFILES_SOFT_LIMIT=65536
 NOFILES_HARD_LIMIT=131072
 
-CLIENT_CONNECTIONS_LIMIT=10000
+CLIENT_CONNECTIONS_LIMIT=500
 
 
 # workaround when .indy become regular file

build-scripts/ubuntu-1604/prepare-package.sh

@@ -35,7 +35,6 @@ fi
 echo -e "\n\nReplace postfixes"
 sed -i -r "s~indy-node-[a-z]+~indy-node~" "$repo/setup.py"
 sed -i -r "s~indy-plenum-[a-z]+~indy-plenum~" "$repo/setup.py"
-sed -i -r "s~indy-anoncreds-[a-z]+~indy-anoncreds~" "$repo/setup.py"
 
 echo -e "Adapt the dependencies for the Canonical archive"
 sed -i "s~python-dateutil~python3-dateutil~" "$repo/setup.py"

ci/ubuntu.dockerfile

@@ -1,4 +1,4 @@
-FROM hyperledger/indy-core-baseci:0.0.1
+FROM hyperledger/indy-core-baseci:0.0.2
 LABEL maintainer="Hyperledger <hyperledger-indy@lists.hyperledger.org>"
 
 ARG uid=1000
@@ -8,16 +8,15 @@ ARG venv=venv
 RUN apt-get update -y && apt-get install -y \
     python3-nacl \
     libindy-crypto=0.4.3 \
-    libindy=1.6.1~683 \
+    libindy=1.6.8 \
 # rocksdb python wrapper
     libbz2-dev \
     zlib1g-dev \
     liblz4-dev \
     libsnappy-dev \
     rocksdb=5.8.8
 
-RUN indy_ci_add_user $uid $user $venv && \
-    indy_ci_charm_crypto $user $venv
+RUN indy_ci_add_user $uid $user $venv
 
 RUN indy_image_clean
 

data/migrations/deb/1_6_703_to_1_6_704.py

@@ -0,0 +1,15 @@
+#!/usr/bin/python3.5
+
+from indy_common.migration.helper import update_indy_env
+
+from stp_core.common.log import getlogger
+
+logger = getlogger()
+
+CLIENT_CONNECTIONS_LIMIT_KEY = "CLIENT_CONNECTIONS_LIMIT"
+CLIENT_CONNECTIONS_LIMIT_VAL = 500
+
+
+logger.info("Going to update clients connections limit.")
+update_indy_env(CLIENT_CONNECTIONS_LIMIT_KEY, CLIENT_CONNECTIONS_LIMIT_VAL)
+logger.info("Done.")

dev-setup/osx/setup-dev-dependencies.sh

@@ -19,22 +19,6 @@ echo 'Installing RocksDB 5.8.8...'
 brew install https://gist.githubusercontent.com/faisal00813/4059a5b41c10aa87270351c4795af752/raw/551d4de01a83f884c798ec5c2cb28a1b15d04db8/rocksdb.rb
 echo 'Installing RocksDB...'
 
-echo 'Installing Charm Crypto...'
-xcode-select --install
-brew install gmp
-brew install pbc
-pushd /tmp
-git clone https://github.com/JHUISI/charm.git
-pushd charm
-./configure.sh --enable-darwin
-make
-make install
-make test
-popd
-rm -rf charm
-popd
-echo 'Installed Charm Crypto'
-
 echo 'Installing libindy...'
 brew install pkg-config
 brew install automake 

dev-setup/ubuntu/init-dev-project.sh

@@ -19,11 +19,6 @@ mkvirtualenv -p python3.5 ${virtualenv_name}
 workon ${virtualenv_name}
 echo "Created virtual environment"
 
-echo "Installing Charm Crypto..."
-cp -r /usr/local/lib/python3.5/dist-packages/Charm_Crypto-0.0.0.egg-info ~/.virtualenvs/${virtualenv_name}/lib/python3.5/site-packages/Charm_Crypto-0.0.0.egg-info
-cp -r /usr/local/lib/python3.5/dist-packages/charm ~/.virtualenvs/${virtualenv_name}/lib/python3.5/site-packages/charm
-echo "Installed Charm Crypto..."
-
 echo "Installing indy-node..."
 pushd indy-node
 pip install -e .[tests]

dev-setup/ubuntu/setup-dev-depend-ubuntu14.sh

@@ -23,12 +23,6 @@ echo 'Installing libsodium...'
 sudo apt-get install -y libsodium13
 echo 'Installed libsodium'
 
-
-echo 'Installing Charm Crypto...'
-sudo apt-get install -y python3-charm-crypto
-echo 'Installed Charm Crypto'
-
-
 echo 'Installing Libindy and Libindy Crypto...'
 sudo apt-get install -y libindy libindy-crypto
 echo 'Installed Libindy and Libindy Crypto'

dev-setup/ubuntu/setup-dev-depend-ubuntu16.sh

@@ -30,11 +30,6 @@ sudo apt-get install -y libbz2-dev \
 echo 'Installed RocksDB'
 
 
-echo 'Installing Charm Crypto...'
-sudo apt-get install -y python3-charm-crypto
-echo 'Installed Charm Crypto'
-
-
 echo 'Installing Libindy and Libindy Crypto...'
 sudo apt-get install -y libindy libindy-crypto
 echo 'Installed Libindy and Libindy Crypto'

docs/source/_build/html/_sources/ci-cd.md.txt

@@ -78,12 +78,10 @@ Use cases for artifacts
 We use [fpm](https://github.com/jordansissel/fpm) for packaging python code into deb packages. Build scripts are placed in `build-scripts` folders:
 - https://github.com/hyperledger/indy-node/blob/master/build-scripts
 - https://github.com/hyperledger/indy-plenum/blob/master/build-scripts
-- https://github.com/hyperledger/indy-anoncreds/blob/master/build-scripts
 
 We also pack some 3rd parties dependencies which are not presented in canonical ubuntu repositories:
 - https://github.com/hyperledger/indy-node/blob/master/build-scripts/ubuntu-1604/build-3rd-parties.sh
 - https://github.com/hyperledger/indy-plenum/blob/master/build-scripts/ubuntu-1604/build-3rd-parties.sh
-- https://github.com/hyperledger/indy-anoncreds/blob/master/build-scripts/ubuntu-1604/build-3rd-parties.sh
 
 Each `build-scripts` folder includes `Readme.md`. Please check them for more details.
 
@@ -92,7 +90,7 @@ Each `build-scripts` folder includes `Readme.md`. Please check them for more det
 - Please note, that we are using semver-like approach for versioning (major, minor, build) for each of the components.
 - Major and minor parts are set in the code (see [\_\_metadata\_\_.py](https://github.com/hyperledger/indy-node/blob/master/indy_node/__metadata__.py)). They must be incremented for new releases manually from code if needed.
 - Build part is incremented with each build on Jenkins (so it always increases, but may be not sequentially)
-- Each dependency (including indy-plenum and indy-anoncreds) has a strict version (see [setup.py](https://github.com/hyperledger/indy-node/blob/master/setup.py))
+- Each dependency (including indy-plenum) has a strict version (see [setup.py](https://github.com/hyperledger/indy-node/blob/master/setup.py))
 - If you install indy-node (either from pypi, or from deb package), the specified in setup.py version of indy-plenum is installed.
 - Master and Stable builds usually have different versions.
 - Differences in master and stable code:

docs/source/_build/html/_sources/setup-dev.md.txt

@@ -18,7 +18,7 @@ You can also have a look at the scripts mentioned below to follow them and perfo
 1. Get scripts from [dev-setup-ubuntu](https://github.com/hyperledger/indy-node/tree/master/dev-setup/ubuntu)
 1. Run `setup-dev-python.sh` to setup Python3.5, pip and virtualenv
 1. Run `source ~/.bashrc` to apply virtual environment wrapper installation
-1. Run `setup-dev-depend-ubuntu16.sh` to setup dependencies (charm-crypto, libindy, libindy-crypto, libsodium)
+1. Run `setup-dev-depend-ubuntu16.sh` to setup dependencies (libindy, libindy-crypto, libsodium)
 1. Fork [indy-plenum](https://github.com/hyperledger/indy-plenum) and [indy-node](https://github.com/hyperledger/indy-node)
 1. Go to the destination folder for the project
 1. Run `init-dev-project.sh <github-name> <new-virtualenv-name>` to clone indy-plenum and indy-node projects and
@@ -81,13 +81,6 @@ Run ```sudo yum install python3.5```
 
 Download the latest build (pywin32-220.win-amd64-py3.5.exe is the latest build as of this writing) from  [here](https://sourceforge.net/projects/pywin32/files/pywin32/Build%20220/) and run the downloaded executable.
 
-### Setup Charm-Crypto
-
-Indy-client requires anonymous credentials library which requires a cryptographic library.
-The default configuration includes an example that uses Charm-Crypto framework.
-You can install it as described in [Anonymous Credentials](https://github.com/evernym/anoncreds) repository 
-(in particular, running `setup-charm.sh`).
-
 ### Setup Libsodium
 
 Indy also depends on libsodium, an awesome crypto library. These need to be installed separately.

docs/source/_build/html/_sources/setup-iptables.md.txt

@@ -1,7 +1,15 @@
 # Setup iptables rules (recommended)
 
-In order to prevent the indy-node process from reaching of open file descriptors limit caused by clients connections it is strongly
-recommended to add iptables rule that limits the number of simultaneous clients connections for client port.
+It is strongly recommended to add iptables (or some other firewall) rule that limits the number of simultaneous clients
+connections for client port.
+There are at least two important reasons for this:
+ - preventing the indy-node process from reaching of open file descriptors limit caused by clients connections
+ - preventing the indy-node process from large memory usage as ZeroMQ creates the separate queue for each TCP connection.
+
+NOTE: limitation of the number of *simultaneous clients connections* does not mean that we limit the
+number of *simultaneous clients* the indy-node works with in any time. The IndySDK client does not keep
+connection infinitely, it uses the same connection for request-response session with some optimisations,
+so it's just about **connections**, **not** about **clients**.
 
 Also iptables can be used to deal with various DoS attacks (e.g. syn flood) but rules' parameters are not estimated yet.
 
@@ -17,7 +25,7 @@ This environment file contains client port (NODE_CLIENT_PORT) and recommended cl
 This parameters can be used to add the iptables rule for chain INPUT:
 
 ```
-# iptables -I INPUT -p tcp --syn --dport 9702 -m connlimit --connlimit-above 15360 --connlimit-mask 0 -j REJECT --reject-with tcp-reset
+# iptables -I INPUT -p tcp --syn --dport 9702 -m connlimit --connlimit-above 500 --connlimit-mask 0 -j REJECT --reject-with tcp-reset
 ```
 Some key options:
  - --dport - a port for which limit is set
@@ -55,8 +63,8 @@ NOTE: this script should be called *after* `init_indy_node` script.
 
 ###### For pip installation
 The `setup_indy_node_iptables` script can not be used in case of pip installation as indy-node environment file does not exist,
-use the `setup_iptables` script instead (9702 is a client port, 15360 is recommended limit for now)
+use the `setup_iptables` script instead (9702 is a client port, 500 is recommended limit for now)
 ```
-# setup_iptables 9702 15360
+# setup_iptables 9702 500
 ```
 In fact, the `setup_indy_node_iptables` script is just a wrapper for the `setup_iptables` script.

docs/source/_build/html/_sources/start-nodes.md.txt

@@ -75,6 +75,12 @@ We can run the script multiple times for different networks.
 
 #### Setup iptables (recommended)
 
+It is strongly recommended to add iptables (or some other firewall) rule that limits the number of simultaneous clients
+connections for client port.
+There are at least two important reasons for this:
+ - preventing the indy-node process from reaching of open file descriptors limit caused by clients connections
+ - preventing the indy-node process from large memory usage as ZeroMQ creates the separate queue for each TCP connection.
+
 Instructions related to iptables setup can be found [here](https://github.com/hyperledger/indy-node/blob/master/docs/setup-iptables.md).
 
 #### Running Node

docs/source/conf.py

@@ -204,8 +204,6 @@ def run_role(self, name, options=None, content=None):
     return orig_run_role(self, name, options, content)
 DummyStateMachine.run_role = run_role
 
-# -------------- end hack ----------------------------
-
 def setup(app):
     app.add_config_value('recommonmark_config', {
             'auto_toc_tree_section': 'Contents',

environment/docker/baseimage/README.md

@@ -26,11 +26,9 @@ Based on [indy-baseimage](#indy-baseimage).
 Adds [Indy Core apt repository](https://repo.sovrin.org/deb) to apt sources.list. Also it adds two scripts into system $PATH available directory that could be run by child images to perform common setup routine:
 
  - `indy_ci_add_user` creates user with python virtualenv configured
- - `indy_ci_charm_crypto` installs Charm Crypto
 
 ### indy-core-baseci
 Base image for images that provide CI testing environment for Indy core projects
-([indy-anoncreds](https://github.com/hyperledger/indy-anoncreds),
 [indy-plenum](https://github.com/hyperledger/indy-plenum),
 [indy-node](https://github.com/hyperledger/indy-node)).
 Based on  [indy-baseci](#indy-baseci).

environment/docker/baseimage/indy-baseci.ubuntu.dockerfile

@@ -7,8 +7,7 @@ RUN apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 68DB5E88 && \
     apt-get update
 
 COPY scripts/user.sh /usr/local/bin/indy_ci_add_user
-COPY scripts/charm_crypto.sh /usr/local/bin/indy_ci_charm_crypto
-RUN bash -c "chmod 755 /usr/local/bin/indy_ci_{add_user,charm_crypto}"
+RUN bash -c "chmod 755 /usr/local/bin/indy_ci_add_user"
 
 COPY __VERSION_FILE__ /