[TT-748] TOML config for integration tests

[Tofel]

TOML is king!

Intro

Compared to CLIP by @kalverra there are some small changes in the implementation:

• TEST_LOG_LEVEL has been kept as env var for now (waiting for v2)
• TEST_TYPE env vars was kept as some tests use to select configuration dynamically
• Chainlink node config was not made configurable yet (waiting for v2)
• override order has been slightly simplified

By default all test configurations should live inside testconfig package in the folder specific for each application, but code is able to find them inside any folder in integration-tests subfolder. First file that's found will be used. To see which configuration files were used run tests with debug log level.

testconfig package provides a single point of entry for reading config for all products and a common part (which apart from logging and network settings provides also initial CL nodes funding). If a product config is not nil it will be validated. I based validations on common sense and values encountered in the code, but it's best if code owners have an extra look. TestConfig struct also has a Save() method in case you want to persist the final config used by tests once all overrides were applied.

Configuration and Overrides

Overrides in precedence order:

• BASE64_CONFIG_OVERRIDE env var
• overrides.toml
• [product_name].toml
• default.toml

BASE64_CONFIG_OVERRIDE is designed to be used in CI (together with ::add-mask::) to override default values with secrets or user/trigger inputs. Example:

  env:
    ENV_JOB_IMAGE: ***-tests:2f75bde4b9f6e51f45411fd85b29f75f5da8ffea
    CHAINLINK_IMAGE: ***
    TEST_SUITE: smoke
    CHAINLINK_ENV_USER: Tofel
    BASE64_CONFIG_OVERRIDE: ***

For tests running in k8s it will be automatically forwarded there, so there's no need to add any manual handling in individual tests.

overrides.toml should be used on local to provide dynamic/run-time variables or to override defaults stored in other files.

[product_name].toml is where most configurations (default and named) should be kept. It supports product-default config like this in case of log_poller.toml:

# product defaults
[LogPoller]
[LogPoller.General]
generator = "looped"
contracts = 2

And named configurations (which are processed as overrides) based on unique configurationName which can be any string, for example test name:

# test-specific
[TestLogManyFiltersPollerFinalityTag]
[TestLogManyFiltersPollerFinalityTag.LogPoller]
[TestLogManyFiltersPollerFinalityTag.LogPoller.General]
contracts = 300

or test type (in vrfv2.toml):

# soak test specific config
[Soak.VRFv2]
[VRFv2.Common]
cancel_subs_after_test_run = true

When each TOML is read the code first looks for the default/unnamed configuration and then for named configuration, which for each file can override default configuration.

Finally default.toml was thought as one that should have the most basic and common settings, like logging configuration.

I also removed all old configs and code that read it, so that there's just one single point of entry for everything.

Local/k8s usage

I have modified all GH workflows present in this repo, so that a base64-ed TOML config is built dynamically from user input or env vars and then sent to GITHUB_ENV for tests to pick up. When you want to run tests in k8s using remote runner or on your local you will need to provide some of these values yourself as they cannot be hardcoded in configuration files.

These required variables might include elements like:

• CL image and version

[ChainlinkImage]
image="public.ecr.aws/chainlink/chainlink"
version="2.8.0"

• test duration (for load, soak, etc):

[Automation]
[Automation.General]
duration=200

• loki-specific configuration (required for these test types/configurations: "Load", "Soak", "Stress", "Spike", "Volume")

[Logging]
[Logging.Loki]
tenant_id="xxxx"
url="https://loki.somewhere.com"
basic_auth="you will never guess"

• or grafana dashboard url

[Logging.Grafana]
url="https://grafana.somewhere.com/chainlink-automation-v2-load-test"

For local execution it's best to put these variables in overrides.toml file.

For k8s/remote runner you need to:

1. Create TOML file with these values
2. Base64 it: cat your.toml | base64
3. Set the base64 result as BASE64_CONFIG_OVERRIDE environment variable.

Known issues/limitations

• if there are two files with the same name (e.g. overrides.toml) in different locations we don't know which one will be picked up
• partial configs will cause validation failures. E.g. if you have present some values in overrides.toml for [VRFv2] config, that overrides 1 key, then we will validate whole VRFv2 config, since it's not nil and thus should be valid (and in this case validation will of course fail)
• that also means that if you didn't set any config values for product or logging, then validation won't run and you might get nil pointer exception, when trying to access a config property further downstream, so it's up to the user to either make sure that the config they need is always set or to validate it's presence before accessing any of the properties
• TestVRFv2Plus/Oracle_Withdraw is failing when run in parallel, but when run in isolation it always passes, in CI it always fails and there's a lot of errors in CL node visible in Grafana
• some of the TOML elements are not saved to file, even though they are set in the actual config. For example for vrfv2plus nothing that's inherited from vrfv2 is ever saved, for the latter no funding-related keys are saved (it seems somehow related to composition)
• overriding of automation's []Load config is not implemented (currently it's an append)
• Slack configuration wasn't moved to TOML
• Since all fields in the config are pointers (to handle optionality) you must take special care when modifying the config programmatically. Making a shallow copy of it won't work, since it will copy addresses of pointers and not their values. It's best to use MustCopy() function of testconfig.TestConfig that performs a deep copy and returns a new copy that can be freely modified without impacting the original. It is especially advised to use copy of config in nested tests to avoid unintended modifications of original that might be time-consuming to detect.

GHA workflows

I've adjusted all of them, but one: integration-staging-tests.yml as it was disabled anyway. All the others but automation-ondemand-tests.yml now accept only 1 input parameter - base64Config - which is what the name suggests. To get that value all you need to do is to execute this cat your.toml | base64 or you could trigger the test from CLI the way @skudasov proposed here.

So when tests expect a base64-ed config it should contain all information that's not present in GHA as secrets (which in case of all these on-demand workflows means Loki specifics). For workflows that use Loki and user inputs I am merging user-provided base64 config with Loki config created on flight, then base64-ed and saved in the env var.

Requirements

It requires this CTF version.

[github-actions]

I see that you haven't updated any CHANGELOG files. Would it make sense to do so?

[github-actions]

I see that you haven't updated any README files. Would it make sense to do so?

[cl-sonarqube-production]

SonarQube Quality Gate

0 Bugs
0 Vulnerabilities
0 Security Hotspots
24 Code Smells

No Coverage information
1.5% Duplication