Skip to content

Latest commit

 

History

History
356 lines (252 loc) · 16.7 KB

README.md

File metadata and controls

356 lines (252 loc) · 16.7 KB
Raw

Testing istio.io Content

This folder contains framework utilies and instructions for testing the content on istio.io. More specifically, these tests confirm that the example, task, and other documents, which contain instructions in the form of bash commands and expected output, are working as documented.

Generated bash scripts, containing the set of commands and expected output for corresponding istio.io markdown files, are used by test programs to invoke the commands and verify the output. This means that we extract and test the exact same commands that are published in the documents.

These tests use the framework defined in the istioio package, which is a thin wrapper around the Istio test framework.

Run the following command to see the current test coverage, including the list of tested documents and those that are in need of a test:

make test_status

Test Authoring Overview

To write an istio.io test, follow these steps:

  1. In the metadata at the top of the index.md file to be tested, change the field test: no to test: yes. This field is used to indicate that the markdown file will be tested and therefore requires a generated bash script containing the commands described in the document.

  2. Run make snips to generate the bash script. After the command completes, you should see a new file, snips.sh, next to the index.md file that you modified in the previous step.

    Each bash command in index.md (i.e., {{< text bash >}} code block) will produce a bash function in snips.sh containing the same command(s) as in the document. Other types of code blocks, e.g., {{< text yaml >}}, will produce a bash variable containing the block content.

    By default, the bash function or variable will be named snip_<section>_<code block number>. For example, the first {{< text bash >}} code block in a section titled ## Apply weight-based routing will generate a bash function named snip_apply_weightbased_routing_1().

    You can override the default name by adding snip_id=<some name> to the corresponding text block attributes. For example {{< text syntax=bash snip_id=config_all_v1 >}} will generate snip_config_all_v1().

    You can also entirely supress generation of a snip function by setting snip_id=none. This is useful for commands that are not intended to be directly executable (e.g., kubectl get pod <your pod name>) and are causing lint errors (see next step, below).

    If a bash code block contains both commands and output, the snips.sh script will include both a bash function and a variable containing the expected output. The name of the variable will be the same as the function, only with _out appended.

  3. Run make lint-fast to check for script errors.

    If there are any lint errors in the generated snips.sh file, it means that a command in the index.md file is not following bash best practices. Because we are extracting the commands from the markdown file into a script file, we get the added benefit of lint checking of the commands that appear in the docs.

    Fix the errors, if any, by updating the corresponding command (or set snip_id=none) in the index.md file and then regenerate the snips.

  4. Create a test bash script named test.sh next to the snips.sh you have just generated.

    If your document is very large and you want to break it into multiple tests, create multiple scripts with the suffix test.sh (e.g., part1_test.sh, part2_test.sh), instead.

    Other scripts in the directory will be ignored.

Test Bash Script

Your bash script will consist of a series of test steps that call the commands in your generated snips.sh file.

Your script can invoke the commands by simply calling snip functions:

snip_config_50_v3 # Step 3: switch 50% traffic to v3

For commands that produce output, pass the snip and expected output to an appropriate _verify_ function. For example:

_verify_same snip_set_up_the_cluster_3 "$snip_set_up_the_cluster_3_out"

This will run the function snip_set_up_the_cluster_3 and confirm that the output is exactly the same as specified in the variable snip_set_up_the_cluster_3_out.

Snip functions often update Istio configuration (e.g., virtual services, destination rules, etc.). Use the _wait_for_istio function to allow the change to propogate to the Istio sidecars before proceeding with the next step of the test:

snip_config_50_v3 # Step 3: switch 50% traffic to v3
_wait_for_istio virtualservice default reviews # wait for routing change to propagate

For snips that deploy Kubernetes services (e.g., kubectl apply -f samples/httpbin/httpbin.yaml), use the _wait_for_deployment function to wait for the deployment to roll out:

_wait_for_deployment default httpbin

You can also use this function to wait for installation changes resulting from istioctl install commands:

_wait_for_deployment istio-system istiod

Test Setup and Cleanup

Before the test steps, there must be one line that specifies the istio setup configuration for the test:

# @setup <setup_config>

Currently supported setup configurations include: profile=default to install the default profile, profile=demo to install the demo profile, profile=ambient to install the ambient profile, and profile=none to not install istio at all.

Choose the setup configuration that best matches the document prerequisites. For example, if the document being tested includes snips with explicit install commands (e.g., setup docs), use:

# @setup profile=none

This will start the test using a clean Kubernetes cluster without Istio installed.

If, on the other hand, the doc's Before you begin section refers the user to the standard Istio installation instructions, chose the profile specified in the doc or default if there is no specific profile mentioned in the instructions.

After all test steps are complete, add the following line to indicate the start of the cleanup steps:

# @cleanup

All steps after this line will be run by the framework, even if the test fails and prematurely exits. The cleanup steps must remove all resources and reverse configuration changes made during the test steps.

Many documents have cleanup instuctions in them, so simply calling the cleanup snip functions will usually reverse all changes made during the test steps. However, extra care should be taken to ensure that the cleanup steps are complete so that after running them, the cluster will be left in the exact same state that it started in. This is important because the test framework runs all tests that specify the same # @setup using the same Kubernetes cluster, so any remaining config changes after the cleanup steps are run, will potentially break a following test.

The framework compares the before and after cluster state and will fail tests that it detects are not properly cleaning up. This comparison, however, is currently not a complete verification, so tests that pass this check may still not be cleaning up completely.

Include Files

The framework automatically includes several bash scripts into your test.sh file, so you don't have to source them yourself. This includes your generated snips.sh file as well as some scripts containing framework utility functions:

You can directly call any function defined in them.

Other optional include files need to be explicitly sourced. For example, tests that use the standard Istio sample services, will typically want to leverage some of the functions in tests/util/samples.sh:

source "tests/util/samples.sh"

startup_bookinfo_sample  # from tests/util/samples.sh
snip_config_50_v3        # from ./snips.sh

Verify Functions

The verify functions first run the snip function and then compare the result to the expected output. The framework includes the following built-in verify functions:

  1. _verify_same func expected

    Runs func and compares the output with expected. If they are not the same, wait a second and try again, up to two minutes by default. The retry behavior can be changed by setting the VERIFY_TIMEOUT and VERIFY_DELAY environment variables. You can also specify the expected number of consecutive successes by setting the VERIFY_CONSECUTIVE environment variable.

  2. _verify_contains func expected

    Runs func and compares the output with expected. If the output does not contain the substring expected, wait a second and try again, up to two minutes by default. The retry behavior can be changed by setting the VERIFY_TIMEOUT and VERIFY_DELAY environment variables. You can also specify the expected number of consecutive successes by setting the VERIFY_CONSECUTIVE environment variable.

  3. _verify_not_contains func expected

    Runs func and compares the output with expected. If the command execution fails or the output contains the substring expected, wait a second and try again, up to two minutes by default. The retry behavior can be changed by setting the VERIFY_TIMEOUT and VERIFY_DELAY environment variables. You can also specify the expected number of consecutive successes by setting the VERIFY_CONSECUTIVE environment variable.

  4. _verify_elided func expected

    Runs func and compares the output with expected. If the output does not contain the lines in expected where "..." on a line matches one or more lines containing any text, wait a second and try again, up to two minutes by default. The retry behavior can be changed by setting the VERIFY_TIMEOUT and VERIFY_DELAY environment variables. You can also specify the expected number of consecutive successes by setting the VERIFY_CONSECUTIVE environment variable.

  5. _verify_regex func expected

    Runs func and compares the output with the regex string expected. If the output does not match with the regex string expected, wait a second and try again, up to two minutes by default. The retry behavior can be changed by setting the VERIFY_TIMEOUT and VERIFY_DELAY environment variables. You can also specify the expected number of consecutive successes by setting the VERIFY_CONSECUTIVE environment variable.

  6. _verify_like func expected

    Runs func and compares the output with expected. If the output is not "like" expected, wait a second and try again, up to two minutes by default. The retry behavior can be changed by setting the VERIFY_TIMEOUT and VERIFY_DELAY environment variables. You can also specify the expected number of consecutive successes by setting the VERIFY_CONSECUTIVE environment variable.

    Like implies:

    • Same number of lines

    • Same number of whitespace-seperated tokens per line

    • Tokens can only differ in the following ways:

      1. different elapsed time values (e.g., 30s is like 5m)
      2. different ip values (e.g., 172.21.0.1 is like 10.0.0.31). Disallows <none> and <pending> by default. This can be customized by setting the CMP_MATCH_IP_NONE and CMP_MATCH_IP_PENDING environment variables, respectively.
      3. prefix match ending with a dash character (e.g., reviews-v1-12345... is like reviews-v1-67890...)
      4. expected ... is a wildcard token, matches anything
      5. different dates in YYYY-MM-DD (e.g. 2024-04-17)
      6. different times in HH:MM:SS.MS (e.g. 22:14:45.964722028)

    This function is useful for comparing the output of commands that include some run-specific values in the output (e.g., kubectl get pods), or when whitespace in the output may be different.

  7. _verify_lines func expected

    Runs func and compares the output with expected. If the output does not "conform to" the specification in expected, wait a second and try again, up to two minutes by default. The retry behavior can be changed by setting the VERIFY_TIMEOUT and VERIFY_DELAY environment variables. You can also specify the expected number of consecutive successes by setting the VERIFY_CONSECUTIVE environment variable.

    Conformance implies:

    1. For each line in expected with the prefix "+ " there must be at least one line in the output containing the following string.
    2. For each line in expected with the prefix "- " there must be no line in the output containing the following string.

  8. _verify_failure func

    Runs func and confirms that it fails (i.e., non-zero return code). This function is useful for testing commands that demonstrate configurations that are expected to fail.

Running the Tests Locally

make doc.test will stand up a kube cluster named istio-testing with KinD, and run the tests in it:

TEST_ENV=kind ADDITIONAL_CONTAINER_OPTIONS="--network host" make doc.test

The make doc.test target can be passed two optional environment variables: TEST and TIMEOUT.

TEST specifies a directory relative to content/en/docs/ containing the tests to run. For example, the following command will only run the tests under content/en/docs/tasks/traffic-management:

TEST_ENV=kind ADDITIONAL_CONTAINER_OPTIONS="--network host" make doc.test TEST=tasks/traffic-management

You can also run one or more individual test by listing the full test names separated by commas. For example:

TEST_ENV=kind ADDITIONAL_CONTAINER_OPTIONS="--network host" make doc.test TEST=tasks/traffic-management/request-routing,tasks/traffic-management/fault-injection

TIMEOUT specifies a time limit exceeding which all tests will halt, and the default value is 30 minutes (30m).

You can also find this information by running make doc.test.help.

Running multicluster Tests

We can leverage KinD to verify multicluster tests locally, using the same script as CI to start our clusters and/or tests: integ-suite-kind.sh can spin up KinD cluster(s) on your local machine as well as trigger tests.

You can run the setup script in make shell to ensure you have all the required tools/packages versions as in CI:

ADDITIONAL_CONTAINER_OPTIONS="--network host" ADDITIONAL_CONDITIONAL_HOST_MOUNTS="--mount type=bind,source=${HOME}/.ssh,destination=/home/user/.ssh,readonly " make shell

To spin up multiple clusters (using the default topology):

mkdir artifacts # create a directory for kubeconfigs and logs
ARTIFACTS=$PWD/artifacts ./prow/integ-suite-kind.sh --topology MULTICLUSTER --skip-cleanup

The topology file is a copy of the multicluster.json updated with pointer to the kubeconfig metadata. For example this is added for the primary and similarly for the others:

    "network": "network-1",
    "meta": {
      "kubeconfig": "/work/artifacts/kubeconfig/primary"
    }

Then to run tests, trigger the desired suite(s) via the command-line or your IDE:

HUB=gcr.io/istio-testing TAG=latest DOCTEST_KUBECONFIG='/work/artifacts/kubeconfig/primary,/work/artifacts/kubeconfig/remote,/work/artifacts/kubeconfig/cross-network-primary' make doc.test.multicluster TEST=./setup/install/external-controlplane/gtwapi_test.sh

Relation to istio/istio repository

When running the tests locally, the version of istio used is inferred from the go.mod istio.io/istio vx.x.x reference - in other words, if that reference is istio.io/istio v0.0.0-20241002191830-e579679ea0ea, the test scripts will clone and use istio/istio branch master@e579679ea0ea for all the doc tests.

Note that you may also set the HUB and TAG environment variables to use a particular Istio build when running tests. If unset, their default values will be inferred from the go.mod reference.

Notes

  1. The tests/util/debug.sh script is automatically included in every test.sh script to enable bash tracing. The bash tracing output can be found in out/<test_path>_[test|cleanup]_debug.txt.

  2. When using kind clusters, you may notice a Exiting due to setup failure: failed waiting for istio-eastwestgateway to become ready: timeout while waiting error as the Istio control plane is being started. Adding a config when creating your kind cluster should fix the issue:

    kind create cluster --name istio-test --config prow/config/default.yaml

  3. For help debugging, you can enable script output to the stdout with the command-line flag --log_output_level=script:debug. This is useful when you're running in an IDE and don't want to find and tail the test output files.