Indy Node system tests depend on the Sovrin package

[askolesov]

That's bad because if someone makes breaking changes in Node, he has to release an updated version of Sovrin in order to fix and run tests.

Example of depending code:
https://github.com/hyperledger/indy-test-automation/blob/master/system/docker/node/Dockerfile#L65

Proposal:

• Make a Sovrin fork of this repository.
• Get rid of Sovrin dependencies and tests in this repository.
• When new changes are added to this repository, merge them into the fork.
• Keep everything Sovrin specific in the fork.

This PR was intended to fix it but I didn't have time to accomplish it so it's closed:
#101

This proposal is the result of the previous research.

[WadeBarnes]

Additional details on the circular dependency:

• indy-test-automation references sovrin package dependencies which in turn reference a specific indy-node package. Therefore, currently, to test a specific indy-node release a sovrin package referencing that specific release needs to be made.
• This circular dependency needs to be broken. The code needs to be updated in such a way that indy-node releases and indy-node based releases (such as sovrin) can be tested separately.

The way forward:

• Although forking the repo and making updates to the two forks would work, I don't think it's the best approach. I'm of the opinion indy-test-automation should be updated to generically support any indy-node based release.
  • Both @askolesov and @devinleighsmith started down this path with the following PRs:
    • [WIP] Sovrin and Node GHA CD #101
    • Correct dockerfiles to run the tests successfully again. split prepa… #103 - Based on Alex's PRs
    • update prepare.sh to reflect correct prepare_client and prepare_node … #104
    • Manually include environment variables into node docker build. #106

Unfortunately this code does not work to conditionally install sovrin:

indy-test-automation/system/docker/node/Dockerfile

Lines 65 to 67 in 3b44feb

	RUN if [ "$SOVRIN_INSTALL" = "yes" ]; then \
	dpkg -i "sovrin_${SOVRIN_VERSION}_amd64.deb"; \
	fi

and which had yet to also address conditional installation of the token-plugin which also references a specific version of indy-node:

indy-test-automation/system/docker/node/Dockerfile

Lines 69 to 78 in 3b44feb

	ARG SOVTOKEN_VERSION
	ARG SOVTOKENFEES_VERSION
	ENV SOVTOKEN_VERSION=${SOVTOKEN_VERSION:-1.0.8~dev153}
	ENV SOVTOKENFEES_VERSION=${SOVTOKENFEES_VERSION:-${SOVTOKEN_VERSION}}
	RUN \
	set -ex; \
	apt-get update && apt-get install -y \
	sovtoken=${SOVTOKEN_VERSION} \
	sovtokenfees=${SOVTOKENFEES_VERSION}; \
	rm -rf /var/lib/apt/lists/*;

Important details:

• indy-test-automation relies on docker-in-docker to spin up the nodes it uses for testing. If that's not configured properly things don't work. Currently there are some issues that need to be resolved to support Windows, such as adding leading / to correctly reference the docker socket. Some reference material can be found here; Running Docker in Docker on Windows (Linux containers).
• When investigating issues with the automated tests it's important to note both indy-node and sovrin CD pipelines use tagged versions of indy-test-automation for performing automated systems tests. For example:
  • https://github.com/hyperledger/indy-node/blob/b5183ac88c7002931dedabba88d37e39d7ee845f/Jenkinsfile.cd#L96-L97
  • https://github.com/hyperledger/indy-node/blob/b5183ac88c7002931dedabba88d37e39d7ee845f/Jenkinsfile.nightly#L69-L70
  • https://github.com/sovrin-foundation/sovrin/blob/31bba83edc6edbd884bbddaf90201522ff61c2f8/Jenkinsfile#L88-L89
  • https://github.com/sovrin-foundation/sovrin/blob/31bba83edc6edbd884bbddaf90201522ff61c2f8/Jenkinsfile.nightly#L78-L79

Next Steps (as I see them):

1. Getting indy-test-automation to build/run locally using the system/docker/prepare.sh and system/docker/run.sh scripts, and the documentation in the system/docker/README.md. This will enable local development to be performed on indy-test-automation and help speed up the development process.
2. Update the associated Dockerfile(s) to support conditional installation of the sovrin packages; sovrin, sovtoken, and sovtokenfees for the node, and libsovtoken for the client. This may require separate docker files in order to perform properly.
3. Determine which unit tests in indy-node rely on either the sovrin or the token-plugin packages and move those to the appropriate repo leaving only indy-node specific tests.
4. Later, as a separate task, refactor indy-test-automation to generically support testing of indy-node based packages.

cc @udosson, @pSchlarb

[WadeBarnes]

The main concern expressed by this issue has been resolved by #109