Added blackbox integration test.

[AdamSaleh]

This test uses an old fedora 34 repo supplied by @tdawson and simple environment, view based on eln and workload based on bash package as defined in test_input. It runs all of the parts of feedback pipeline as defined in it's __main__ entrypoint. Then it compares the subset of results that are in json to ones already stored in test_ouptut

[asamalik]

Hey, thanks! Couple things:

Any reason you're not using the existing test_configs folder? You don't need to add test_input/configs/*.

You probably didn't want to include all the test_output/*?

[AdamSaleh]

Good point on the config. And I thought I did have everything I wanted in the test_output, meaning all of the jsons to compare :)

[asamalik]

Ah, I see. You remove all _static,*.html, *.txt and compare it to your directory.

That's a very interesting approach I'd advise against, because it feels very fragile and many changes will require changing all of the files in test_output. Even adding one data file or adding one test config will break the tests.

What I was hoping for was a test targetting specific pieces of the output such as

• a list of packages in a workload
• a list of packages in a view, including buildroot
  ... and just those, as data extracted from the output files, with a reference. That way adding anything to the output or changing the format of the files slightly won't fail the tests.

Using Troy's repo with Fedora 24 (a dead release which no longer gets updates) ensures we get consistent data over time.

Does that make sense?

[asamalik]

Also Pytest is great for unit testing, not really for blackbox testing. I'd run it in a command line as described in README and then had something look at the output. This way you won't have to basically rewrite the main function to make it work within Pytest.

[asamalik]

To be clear, I'm happy you're contributing, please don't feel discouraged, I'm here to support/help you.

A few pointers... if you look at the github workflow yaml, you'll see it:

• runs on pull-requests
• executes Content Resolver with the test configs - this needs to be updated with --dev-buildroot
• runs pytest

So when writing tests, you already start with a directory (output) full of results. So all you need to do is to look at the results and test those.

You could add this to your tests:

def load_data(path):
    with open(path, 'r') as file:
        data = json.load(file)
    return data

And then load one workload:

workload_data = load_data("workload--bash--base-eln--repo-eln--aarch64.json")

... then, for example, get a list of added packages:

added_packages = workload_data["data"]["pkg_added_ids"]

... and compare that with a reference which you can just have in the script itself. It's not that long. This specific workload has:

...
"pkg_added_ids": [
      "ncurses-base-6.3-3.20220501.eln120.noarch",
      "ncurses-libs-6.3-3.20220501.eln120.aarch64",
      "glibc-2.36.9000-8.fc38.aarch64",
      "glibc-common-2.36.9000-8.fc38.aarch64",
      "basesystem-11-14.eln120.noarch",
      "bash-5.1.16-4.eln121.aarch64",
      "glibc-minimal-langpack-2.36.9000-8.fc38.aarch64",
      "tzdata-2022d-1.eln121.noarch",
      "filesystem-3.18-2.eln121.aarch64",
      "setup-2.14.2-1.eln121.noarch",
      "libgcc-12.2.1-2.eln121.aarch64"
    ],
...

This way you can target specific things, which:

• makes the tests more focused
• makes the tests less fragile, because it won't fail with irrelevant changes in the output

If you use the repo Troy added (Fedora 34), the results will be always the same. Let me know if you want me to walk you through the config structure, it's not complicated but can be complex if you look at it the first time.

And thre's no stupid questions, so please don't hesitate to ask anything!

Oh and to run the same thing locally, you can just add test.sh with something like:

./feedback_pipeline.py --dev-buildroot --dnf-cache-dir /dnf_cachedir test_configs output
pytest

[AdamSaleh]

@asamalik I still really like the idea of thorough comparisons of json files.
I.e. one of the things a good test should do is to fail once we start changing things.

If the repo that Troy added is as stable as we want it to be, this shouldn't be a problem.
I looked the files and I don't see a place where irrelevant changes in the output would be happening.
Ok, apart from the fact the lists are unsorted, but I account for that.

Other thing, if we are running tests, they should be self-contained. I really don't want to rely that
there exists a specific folder with contents of a specific previous run.

But I agree, that with a test like this, it shouldn't fail if we add unrelated new files,
so I only check that the files I have in test_data/*/output have counterparts in the output
with same content.

[asamalik]

In my opinion tests should fail when we break things not when we change things. For example adding information to a file shouldn't fail a test.

Please don't re-run Content Resolver for every single test. For example, it pulls hundreds of MB of repodata with every run. If we need to run it with different sets of configs to test different things, then sure. But not as a rule of thumb.