test_tor_mirror_repo_declared_only_once failing on release/0.9

[redshiftzero]

Description

CI is failing on the latest release branch

Steps to Reproduce

Look at all CI builds on release/0.9: https://circleci.com/gh/freedomofpress/securedrop/tree/release%2F0%2E9

Expected Behavior

Builds are green

Actual Behavior

The staging CI job is consistently failing:

    ________ test_tor_mirror_repo_declared_only_once[ansible://app-staging] ________
    [gw19] linux2 -- Python 2.7.13 /usr/bin/python
    
    host = <testinfra.host.Host object at 0x7fa730986650>
    
        def test_tor_mirror_repo_declared_only_once(host):
            """
            The apt repo config is written both via the `securedrop-config`
            package and via Ansible at install time. The filename logic
            was slightly different for both between 0.5 and 0.6, so let's
            ensure that we've cleaned up adequately, and the tor-apt mirror
            is declared on the system only once.
            """
            c = host.command("grep -rl tor-apt /etc/apt/")
            assert c.rc == 0
            files_found = c.stdout.rstrip("\n").split("\n")
            # Two files are expected, cron-apt and one apt repo
    >       assert len(files_found) == 2
    E       AssertionError: assert 1 == 2
    E        +  where 1 = len(['/etc/apt/security.list'])
    
    ../testinfra/staging/common/test_tor_mirror.py:95: AssertionError
    ________ test_tor_mirror_repo_declared_only_once[ansible://mon-staging] ________
    [gw30] linux2 -- Python 2.7.13 /usr/bin/python
    
    host = <testinfra.host.Host object at 0x7f3d39d09410>
    
        def test_tor_mirror_repo_declared_only_once(host):
            """
            The apt repo config is written both via the `securedrop-config`
            package and via Ansible at install time. The filename logic
            was slightly different for both between 0.5 and 0.6, so let's
            ensure that we've cleaned up adequately, and the tor-apt mirror
            is declared on the system only once.
            """
            c = host.command("grep -rl tor-apt /etc/apt/")
            assert c.rc == 0
            files_found = c.stdout.rstrip("\n").split("\n")
            # Two files are expected, cron-apt and one apt repo
    >       assert len(files_found) == 2
    E       AssertionError: assert 1 == 2
    E        +  where 1 = len(['/etc/apt/security.list'])
    
    ../testinfra/staging/common/test_tor_mirror.py:95: AssertionError

[emkll]

In previous release cycles, CI didn't run on the release branch, only for PRs to the release branch:

• https://github.com/freedomofpress/securedrop/commits/release/0.8
• https://github.com/freedomofpress/securedrop/commits/release/0.7

I have run infra tests locally on staging VM provisioned on 0.9.0-rc2 using molecule verify -s libvirt-staging and did not observe the error, and all tests are passing:

========= 422 passes, 13 skipped, 2 xfailed, 2 xpassed in 170.52 seconds ==========

Perhaps something to do with the CI rebase script https://github.com/freedomofpress/securedrop/blob/develop/devops/scripts/rebase-ci.sh#L18 that is skipped for master and release branches ?

[msheiny]

So this test should have two files as the grep output:

root@app-staging:/home/vagrant# grep -rl tor-apt /etc/apt/                                               
/etc/apt/security.list
/etc/apt/sources.list.d/tor_apt_freedom_press.list

I relaunched the failing job, ssh'd in, and stopped it from tearing down. I confirm both files are in-place. This is really really perplexing. Trying to target just that testinfra test now

[msheiny]

IN BEFORE @conorsch its molecule/aws/tor_apt_test.yml

... this seems like it would be the culprit but i'm logged into the servers and not seeing the result I would expect from grep ... 😕

[msheiny]

dang it okay.. the very last thing i did was..

• re-run job with ssh
• modify the molecule.yml to target the test and disable capturing output ( add a k: test_tor_mirror_repo_declared_only_once and s: true ) oh yeah and delete n: auto to reduce crazy number of threads launching
• made some debug lines in the test directly
• re-run with molecule verify -s aws

In particular the very last thing i did was echo the ip info ip a .. and to my surprise the system did not match up to what i was expecting.. I have no idea where these are being run (maybe the local system). I dunno, I was really really perplexed here. More digging is necessary.

[conorsch]

Had some trouble accessing the remote EC2 hosts, since the IdentityFile was missing from /tmp/molecule/sd/aws/. As such, wasn't able to run the testinfra suite, nor log in to the hosts manually to inspect the files on disk. Will revisit later on, tabling for now.

[redshiftzero]

On our failing release/0.9 builds we are seeing the following:

TASK [Switch apt repo URLs to staging.] ****************************************
    fatal: [mon-staging]: FAILED! => {"changed": false, "msg": "Path /etc/apt/sources.list.d/tor.apt.freedom.press.list does not exist !", "rc": 257}
    ...ignoring
    fatal: [app-staging]: FAILED! => {"changed": false, "msg": "Path /etc/apt/sources.list.d/tor.apt.freedom.press.list does not exist !", "rc": 257}
    ...ignoring

The logic in molecule/aws/tor_apt_test.yml is attempting to replace tor-apt.freedom.press with a test repo only for branches beginning in release (i.e. why we're only seeing issues on the release branch). I don't think we are actually using this test repo we are switching to as @emkll informs me that test tor packages are being pushed to apt-test.freedom.press, so that's something we should modify (ref: 732c8da for the backstory on that).

But the fact that tor-apt.freedom.press is apparently not configured means that the failing test makes sense - /etc/apt/sources.list.d/tor_apt_freedom_press.list does not exist, so there should only be one matching result.

What's unclear to me is why tor-apt.freedom.press would not be configured on a staging machine on the release branch...

[msheiny]

just to clarify, I did SSH into the machine during a CI run .. copied the grep test command exactly and got two files listed. it was really really really weird

[redshiftzero]

So in #3189 we renamed /etc/apt/sources.list.d/tor.apt.freedom.press.list to /etc/apt/sources.list.d/tor_apt_freedom_press.list, which was done shortly before #2941 (merged into release branch and then back into develop) - hence the confusion. The fix here is to:

1. Remove the task that removes /etc/apt/sources.list.d/tor_apt_freedom_press.list
2. Remove the task that switches the tor repo to tor-apt-test.freedom.press since we're not using that test repo. Instead we're using apt-test.freedom.press, which is all kosher with these changes as the latest Tor packages will be installed anyway in staging:

root@app-staging:/etc/apt/sources.list.d# apt-cache policy tor
tor:
  Installed: 0.3.3.9-1~trusty+1
  Candidate: 0.3.3.9-1~trusty+1
  Version table:
 *** 0.3.3.9-1~trusty+1 0
        500 https://apt-test.freedom.press/ trusty/main amd64 Packages
        100 /var/lib/dpkg/status
     0.3.3.8-1~trusty+1 0
        500 https://apt-test.freedom.press/ trusty/main amd64 Packages
     0.3.3.7-1~trusty+1 0
        500 https://apt-test.freedom.press/ trusty/main amd64 Packages
        500 https://tor-apt.freedom.press/ trusty/main amd64 Packages
     0.2.4.27-1build0.14.04.1 0
        500 http://archive.ubuntu.com/ubuntu/ trusty-updates/universe amd64 Packages
        500 http://security.ubuntu.com/ubuntu/ trusty-security/universe amd64 Packages
     0.2.4.20-1 0
        500 http://archive.ubuntu.com/ubuntu/ trusty/universe amd64 Packages

which is what we want (so we're QAing new Tor packages in CI)