Provide a way to verify the authenticity of a Zowe driver as being original, untampered, release

[Joe-Winchester]

Is your feature request related to a problem? Please describe.
The Zowe release pipeline creates builds, e.g. 1.5, 1.9, 1....
1.9 represents the Zowe SMP/E FMID AZWE001 - the A letter was chosen to make it vendor neutral - other letters are reserved for specific IBM z/OS vendors.
The Zowe drivers can be redistributed as-is by vendors offering enterprise support for Zowe through their own channels. These are enterprise "trusted" SMP/E receive order sites.
The Zowe SMP/E FMID installs into the USS location /usr/lpp/zowe. The design is for a single destination which offers beneifts to users and extenders that a single desktop and single APIML can have app to app communication and single sign on and MFA ... across all products that extend Zowe from all vendors, whether from the original vendor that delivered the SMP/E package or not.
Traditionally a vendor offering support may only do this if the software they are supporting was obtained by the customer through their channel.

Describe the solution you'd like
Provide assistance for a vendor support team to be able to trust a Zowe driver irrespective of where it came from, so the source is based on a customer preference for who their trusted partner for support is but they may choose to install extensions from other vendors into the same base (CLI, APIML, Desktop).

Additional information
Node.js has already solved this. This section is pasted from the README.MD that is delivered with Node.js.

Verifying Binaries

Download directories contain a `SHASUMS256.txt` file with SHA checksums for the
files.

To download `SHASUMS256.txt` using `curl`:

console
$ curl -O https://nodejs.org/dist/vx.y.z/SHASUMS256.txt

To check that a downloaded file matches the checksum, run
it through `sha256sum` with a command such as:

console
$ grep node-vx.y.z.tar.gz SHASUMS256.txt | sha256sum -c -

For Current and LTS, the GPG detached signature of `SHASUMS256.txt` is in
`SHASUMS256.txt.sig`. You can use it with `gpg` to verify the integrity of
`SHASUM256.txt`. You will first need to import
[the GPG keys of individuals authorized to create releases](#release-keys). To
import the keys:

console
$ gpg --keyserver pool.sks-keyservers.net --recv-keys DD8F2338BAE7501E3DD5AC78C273792F7D83545D

See the bottom of this README for a full script to import active release keys.

Next, download the `SHASUMS256.txt.sig` for the release:

console
$ curl -O https://nodejs.org/dist/vx.y.z/SHASUMS256.txt.sig

Then use `gpg --verify SHASUMS256.txt.sig SHASUMS256.txt` to verify
the file's signature.

[John-A-Davies]

There is already the provision in Zowe to verify that a downloaded file is an original one.

Files and their keys are in Jfrog https://zowe.jfrog.io/zowe/webapp/#/artifacts/browse/tree/General/libs-release-local/org/zowe/1.10.0

Instructions for their use are here on the Zowe download site: https://d1xozlojgf8voe.cloudfront.net/post_download.html?version=1.10.0
This is documented in the user guide; step beginning … “Verify the integrity of the PAX file”.

However, the post-download doc from cloudfront.net does not describe how to verify the driver authenticity for any files except the convenience build .PAX file. I've raised issue zowe/docs-site#1126 for this.

[John-A-Davies]

Support staff will want to verify that the user's runtime directory content is unmodified from the official, released version. The first effort is to verify the USS runtime - verification of the shipped MVS runtime datasets will come later.

One solution is to create a key based on the contents of the USS runtime created during zowe-install-test. The key should ignore data about the file permissions, ownership, date, and any other attributes apart from file name and contents. The key will be published on JFrog like the current ones.

The support person will download the key for the customer's release, and run a script to walk the customer's runtime directory tree and run sha256sum on the aggregated files. Then use gpg --verify ... to verify the aggregated file's signature. A further, simple check will verify that the directories and file names are correct.

[John-A-Davies]

Program md5deep looks promising http://md5deep.sourceforge.net/
It's for Windows but they say you can compile it on Cygwin, Linux, FreeBSD, OpenBSD, Mac OS X, OpenSolaris, HP/UX, etc.

[John-A-Davies]

Suggested solutions

Phase 1 – compare filenames and file sizes
Issues:

1. SMP/E runtime has an extra SMPE directory with 10 PAX files in it.

2. Runtime directory can have javacore, heap and snap dumps in it.

These can be excluded from the list to be compared.

Phase 2 – real checksum of all files
Method:
Step 1 – create a checksum for each file, and aggregate them into a single file
Step 2 – create a checksum of the aggregate

Issues:

1. The system holding the runtime directory to be verified needs Gnu Privacy Guard installed. You can download from here https://gnupg.org/download/index.html

[John-A-Davies]

The hash file could be stored in the runtime directory itself, thus rendering the hash check self-contained.
A script e.g. zowe-verify-runtime-integrity.sh could be provided to L2/L3. The script's sole parameter would be the runtime pathname.

[John-A-Davies]

@Joe-Winchester
Phase 2 – real checksum of all files
Generating checksums takes some time. Current hash set of zowe runtime will take an estimated 100 minutes to generate.

[John-A-Davies]

Example on a real z/OS system

real    40m49.03s
user    21m39.56s
sys     7m13.19s

[John-A-Davies]

An improved approach reduced the total time to generate the local hash set to 17 seconds.

[John-A-Davies]

Adapted from slack with @jackjia-ibm

The checksum generation is better to stay in zowe-install-packaging build, not zowe-install-test. The zowe-install-test branch may not match the zowe build you are testing. For example, you can use zowe-install-test staging branch to test a zowe build generated by a developer’s own branch.

we actually have expanded the full file system tree with installation when we build SMPE

i’m not sure if that can be the best place to generate checksums

we definitely will build SMPE for each RC and formal releases, so this script will be executed, and can be easily associated so the zowe build matches the checksum

if we build checksums along with zowe-install-packaging, we can also promote the checksum file and put into https://zowe.jfrog.io/zowe/webapp/#/artifacts/browse/tree/General/libs-release-local/org/zowe/1.11.0-RC1 folder

[John-A-Davies]

Current proposal.

The zowe-install-packaging SMP/E build will run zowe-checksum-runtime.sh on the runtime directory it has just created. This will generate a RefRuntimeHash.txt file (and the hash program executable) which will be published in JFrog artifactory. The customer can download this RefRuntimeHash.txt file and verify their own installed runtime directory by running zowe-verify-authenticity.sh.

[John-A-Davies]

The zowe-install-packaging SMP/E build will also run zowe-verify-authenticity.sh to check itself.

[John-A-Davies]

This test run of those changes was successful
https://wash.zowe.org:8443/job/zowe-install-packaging/view/change-requests/job/PR-1316/28/consoleFull
Search for

zowe-checksum-runtime.sh started

and

[consumer_2] Deploying artifact: https://zowe.jfrog.io/zowe/libs-snapshot-local/org/zowe/1.11.0-PR-1316/RefRuntimeHash-1.11.0-pr-1316-28-20200505111936.txt
[consumer_2] Deploying artifact: https://zowe.jfrog.io/zowe/libs-snapshot-local/org/zowe/1.11.0-PR-1316/HashFiles-1.11.0-pr-1316-28-20200505111936.class

[jackjia-ibm]

@John-A-Davies I saw you also publish HashFiles to Artifactory. I think it makes sense if this file changes a lot, might be different from build to build, release to release. Otherwise, if it's quite stable, storing in repository could be a better place. For example, we stored signing keys in repository https://github.com/zowe/zowe-install-packaging/tree/master/signing_keys because they don't change.

[John-A-Davies]

@jackjia-ibm Yes, it's as stable as the rest of the source files. The RefRuntimeHash.txt file must change with each release, but HashFiles.* probably won't. I'm happy to follow your guidance on how best to store these items.

We also need to store the RefRuntimeHash.txt file for each release and make them all available for download, as well as versions of that file for releases prior to v1.12.0

[John-A-Davies]

Feedback from my presentation of this topic to the Zowe team

• Parameters are better when they are keyword-value pairs, e.g. script.sh -a abc -b def rather than positional, e.g. script.sh abc def

• Could use java to download the RefRuntimeHash.txt file

• Ideally, zowe-verify-authenticity.sh would need no parameters

• Optimise the user experience for >= v1.12.0 at the expense of prior releases

• Is SMP/E the only alternative build that creates /usr/lpp/zowe ?

[John-A-Davies]

A self-contained version of zowe-verify-authenticity.sh

Either

• Dowload the file RefRuntimeHash.txt with curl, or
• The file RefRuntimeHash.txt could be kept in a separate directory, e.g. /tmp/usr/lpp/zowe/fingerprint, which is excluded from the hash process

Proposed new parm list format:

zowe-verify-authenticity.sh [-r <runtime-dir>] [-h <HashFiles-dir>] [-t <output-dir>]

-r runtime directory path. Defaults to CWD
-h hash path, where HashFiles.class and RefRuntimeHash.txt reside. Defaults to -r/fingerprint
-t directory for output files. Defaults to /tmp

For example:

cd downloads/hash
zowe-verify-authenticity.sh -r /tmp/usr/lpp/zowe -h . -t ~/hash

Or you can just type

cd /tmp/usr/lpp/zowe/bin 
zowe-verify-authenticity.sh

with no parameters. The zowe-support.sh script could run it like this.

This check could be falsified, but L2 can download these files to downloads/hash

zowe-verify-authenticity.sh 
HashFiles.class
RefRuntimeHash.txt

and run

cd downloads/hash
zowe-verify-authenticity.sh -r /tmp/usr/lpp/zowe -h . 

[stevenhorsman]

-r parameter shouldn't default to CWD - this script is in <runtime>/<sub-path>/zowe-verify-authenticity.sh, so we should work out the runtime directory by using dirname $0
-t shouldn't default to /tmp - at least we should default to ${TMPDIR:-/tmp} as some customers have re-defined TMPDIR to point elsewhere. However I think there is an argument that the output should go to the log directory so that it can be picked up by the support script.

[Joe-Winchester]

<ROOT_DIR>/bin/zowe-verify-authenticity.sh should run without any args
points to
<ROOT_DIR>/bin/internal/HashFiles.class
The HashFiles.txt goes into
<ROOT_DIR>/fingerprint/RefRuntimeHash_1.12.0.txt
For out directory write to -l argument that defaults similar to zowe-install.sh -i defaulting and uses a new directory, e.g. /fingerprint
see the script setup-log-dir.sh for defaulting as this is used by other creation scripts, see...
https://docs.zowe.org/stable/user-guide/install-zowe-zos-convenience-build.html#step-3-choose-a-dataset-hlq-for-the-samplib-and-loadlib

[nkocsis]

Can this be closed ?

[stevenhorsman]

Can this be closed ?

No - it's not done

[Joe-Winchester]

@nkocsis This issue will be closed when the pull request #1316 with the code gets reviewed, approved and merged. It was demo'd on a few calls recently to the community, but there is work to do be done to get it into 1.12 and also the have it documented. Once the pull request #1316 that contains the code is closed it gets associated with the issue which will be closed automatically. At the same time the CHANGELOG.md file for the release will be updated to reference that the feature is included in a release, which we're hoping will be 1.12.