Skip to content

QA Procedures

Jump to bottom
Cory Francis Myers edited this page Mar 10, 2022 · 24 revisions

Release QA

RC QA

Hardware installation from scratch (clean install scenario):

  1. (Optional): Take a backup of your current instance.

  2. (Optional, if reusing existing Admin workstation): Copy the following config and pubkey files from your existing Admin Workstation setup to a new folder in ~/Persistent (eg ~/Persistent/sdfiles):

  • ~/Persistent/securedrop/install_files/ansible-base/SecureDrop.asc (or your instance's Submission Public Key)
  • ~/Persistent/securedrop/install_files/ansible-base/journalist.asc (or your instance's journalist pubkey)
  • ~/Persistent/securedrop/install_files/ansible-base/alerts.asc (or your instance's OSSEC alerts pubkey)
  • ~/Persistent/securedrop/install_files/ansible-base/group_vars/all/site-specific

Then

  • Delete the old securedrop repo from this Admin USB
  • Delete ~/.ssh/known_hosts (assuming this Admin USB's SSH key is not being used for anything else)

  1. Download and verify Ubuntu Server and perform a clean Ubuntu install (http://releases.ubuntu.com/20.04/ubuntu-20.04.3-live-server-amd64.iso) on hardware; configure Ubuntu Server as per the standard SecureDrop install process.

cd ~/Persistent
git clone https://github.com/freedomofpress/securedrop

Once cloned, git fetch --tags, verify the tag for the release branch you are testing, and check out that tag.

  1. Manually edit install_files/ansible_base/roles/install-fpf-repo/defaults/main.yml to use:
apt_repo_url: https://apt-test.freedom.press
...
apt_repo_pubkey_files:
    - apt-test-signing-key.pub
  1. cd ~/Persistent/securedrop/install_files/ansible-base/ and copy in the required .asc files, and optionally, copy the site-specific file into ./group_vars/all/ (this will pre-populate your previous configuration during setup, and save you some time and typing).

To configure and install SecureDrop, run:

cd ~/Persistent/securedrop
./securedrop-admin setup
./securedrop-admin sdconfig
./securedrop-admin install
./securedrop-admin tailsconfig

  1. Kernel Testing. On app server:

    a. install paxtest (available via apt) and run paxtest blackhat The desired results are:

    • strcpy and strcpy, PIE will be paxtest: return address contains a NULL byte
    • Randomization tests: bit guesses in the high 20s or higher (there may be one or two in the mid-20s, this is still okay).
    • memcpy and memcpy, PIE will show Vulnerable: this is expected on Ubuntu 20.04
    • Everything else: "Killed"

    b. curl -L https://meltdown.ovh -o meltdown.sh to make sure you have the latest meltdown script. Then run sudo bash meltdown.sh -v so that you have the script on the app server. Note: this should never be done on production hardware--this is for QA-only hardware. You should not see any tests that come back as STATUS:VULNERABLE.

    ATTENTION: The NUC7i5BNH, NUC7i7DNHEs, NUC8s and NUC10s and generate a false-positive on one particular test (Foreshadow L1TF). This has been noted by our Security Engineer. See https://github.com/freedomofpress/securedrop/issues/5040#issuecomment-559597643.

    Note: you will generate OSSEC alerts about grsec when you run these two tests--this is normal/expected.

  2. Follow the QA matrix and the Github release ticket for further tests (end-to-end, acceptance tests, release-specific tests, etc).

Upgrade hardware installation (unattended-upgrades scenario)

  1. Take a backup of your existing SecureDrop installation using the securedrop-admin backup command. You will need enough free space on your Admin Workstation USB to complete this backup.

  2. Either run the ansible QA playbook:

cd ~/Persistent/securedrop
source admin/.venv3/bin/activate 
cd install_files/ansible-base
ansible-playbook -vv --diff securedrop-qa.yml

OR

Connect to each of the application and monitor servers via SSH and follow these steps:

  • Modify /etc/apt/sources.list.d/apt_freedom_press.list and /etc/apt/security.list to use the apt-test.freedom.press repo instead of the apt.freedom.press repo, ensure the apt-test public key is present, and update the apt cache.
  • Run sudo unattended-upgrades -d to upgrade your instance. Your SSH session may freeze or drop, but you should be able to reconnect thanks to tmux. You may use apt-cache policy <packagename> to check a given package's version.
  1. Complete steps 6 and 7 of clean install QA instructions (kernel testing, QA Matrix and Github release ticket testing).

Pull Request QA

  1. Perform a visual review of the code changes.

  2. Check out the pull request branch on your own machine using git fetch <remotename> pull/ID/head:BRANCHNAME (If your freedomofpress/securedrop repo remote is named upstream, and you want to review PR 1337 on a local branch called newfeature, run git fetch upstream pull/1337/head:newfeature, then git checkout newfeature.)

  3. Follow the test plan as documented in the PR, and run the appropriate unit/functional/configuration/linting tests as relevant.

More tips

  • Save a folder in your Tails Persistent directory called something like sdconf that contains: install_files/ansible_base/group_vars/all/site-specific, your Submission Public Key, and your admin/journalist email public key, and copy those into place (the latter two just to the ansible_base directory) on a new QA run to avoid retyping everything during sdconfig.
  • AppArmor can be checked with sudo aa-status
  • Kernel can be checked with uname -r
  • The submissions table is at /var/lib/securedrop/db.sqlite. Use sudo sqlite3 /var/lib/securedrop/db.sqlite to open, select * from submissions to see submission hashes, and so on.
  • There are some utilities (such as qa_loader.py) that may help you with testing; if their use is required, they should be documented in the test plan

Preflight testing

The goal of preflight testing is ensure that the release has been properly deployed, and is ready for final deployment to production instances. For preflight testing, we host the new packages at https://apt-qa.freedom.press/, rather than https://apt.freedom.press/. The exact same packages, Release file, and corresponding detached signature, generated with the prod key, are used. We deploy those artifacts to apt-qa to provide final verification that the release will function well for real-world prod instances.

With more automation around the release process, we should be able to do away with the need to perform preflight testing. For now, it's done in support of the RM as one of the final steps in preparing a release.

Pre-release prep

On release day, prepare either hardware or prod VMs suitable for installation. VMs are usually preferable, since any hardware-specific changes will have been tested thoroughly as part of normal QA. Choose whether you'll be performing a clean install, or testing an upgrade.

For upgrade, it's important to perform these steps ahead of time, because it can take ~1h to install SD before it's ready for preflight. Here are the steps:

# on dev workstation:
molecule create -s libvirt-prod-focal
# then log into Tails Admin VM, and run:
./securedrop-admin install
./securedrop-admin tailsconfig
# then "ssh app" and make an account, log in, make sure it works

Now you're prepped. To test, log into both servers, and edit /etc/apt/sources.list.d/apt_freedom_press.list, replacing apt.freedom.press with apt-qa.freedom.press. No other changes are required.

Once the release is available on apt-qa, run sudo unattended-upgrades -d to upgrade to the new release. Perform functional testing, and report any problems.

For clean install preflight testing, the procedure is similar to the above:

# on dev workstation:
molecule create -s libvirt-prod-focal
# then log into Tails Admin VM
# edit ~/Persistent/securedrop/install_files/ansible-base/roles/install-fpf-repo/defaults/main.yml 
# and change "apt.freedom.press" to "apt-qa.freedom.press".
# wait for the go-ahead from the RM, then proceed with install:
./securedrop-admin install
./securedrop-admin tailsconfig
# then "ssh app" and make an account, log in, make sure it works
Clone this wiki locally