Open
Description
openedon Sep 25, 2022
Currently elastic-package test builds do not explicitly render the package, stack and elastic-package executable versions into the test report. Having these available can be very helpful when attempting to debug issues relating to changes in the versions of system components.
When elastic-package test is run the package version is incidentally rendered into the test output in the asset tests in both the human and XML report outputs, but the stack version and elastic-stack executable versions are not. See details (all run with elastic-package as at ba7e680 with the default stack version 8.4.1 in this version of ep.
Human output
$ elastic-package test 2022/09/25 10:10:19 WARN CommitHash is undefined, in both .../.elastic-package/version and the compiled binary, config may be out of date. Run test suite for the package Run system tests for the package 2022/09/25 10:11:10 INFO Write container logs to file: .../go/src/github.com/elastic/integrations/build/container-logs/meraki-webhook-https-1664066470434862000.log 2022/09/25 10:11:57 INFO Write container logs to file: .../go/src/github.com/elastic/integrations/build/container-logs/meraki-webhook-http-1664066517438873000.log 2022/09/25 10:12:36 INFO Write container logs to file: .../go/src/github.com/elastic/integrations/build/container-logs/cisco_meraki-log-logfile-1664066556401479000.log 2022/09/25 10:13:23 INFO Write container logs to file: .../go/src/github.com/elastic/integrations/build/container-logs/cisco_meraki-log-tcp-1664066603425999000.log 2022/09/25 10:14:06 INFO Write container logs to file: .../go/src/github.com/elastic/integrations/build/container-logs/cisco_meraki-log-udp-1664066646381401000.log --- Test results for package: cisco_meraki - START --- ╭──────────────┬─────────────┬───────────┬──────────────┬────────┬───────────────╮ │ PACKAGE │ DATA STREAM │ TEST TYPE │ TEST NAME │ RESULT │ TIME ELAPSED │ ├──────────────┼─────────────┼───────────┼──────────────┼────────┼───────────────┤ │ cisco_meraki │ events │ system │ meraki-https │ PASS │ 35.901381537s │ │ cisco_meraki │ events │ system │ meraki-http │ PASS │ 30.872360032s │ │ cisco_meraki │ log │ system │ logfile │ PASS │ 23.100860977s │ │ cisco_meraki │ log │ system │ tcp │ PASS │ 30.951037031s │ │ cisco_meraki │ log │ system │ udp │ PASS │ 27.989145104s │ ╰──────────────┴─────────────┴───────────┴──────────────┴────────┴───────────────╯ --- Test results for package: cisco_meraki - END --- Done Run asset tests for the package --- Test results for package: cisco_meraki - START --- ╭──────────────┬─────────────┬───────────┬───────────────────────────────────────────────────────────────────────┬────────┬──────────────╮ │ PACKAGE │ DATA STREAM │ TEST TYPE │ TEST NAME │ RESULT │ TIME ELAPSED │ ├──────────────┼─────────────┼───────────┼───────────────────────────────────────────────────────────────────────┼────────┼──────────────┤ │ cisco_meraki │ │ asset │ dashboard cisco_meraki-4832a430-af22-11ec-a899-6f7e676e0fb4 is loaded │ PASS │ 722ns │ │ cisco_meraki │ events │ asset │ index_template logs-cisco_meraki.events is loaded │ PASS │ 152ns │ │ cisco_meraki │ events │ asset │ ingest_pipeline logs-cisco_meraki.events-1.2.0 is loaded │ PASS │ 159ns │ │ cisco_meraki │ log │ asset │ index_template logs-cisco_meraki.log is loaded │ PASS │ 171ns │ │ cisco_meraki │ log │ asset │ ingest_pipeline logs-cisco_meraki.log-1.2.0 is loaded │ PASS │ 125ns │ ╰──────────────┴─────────────┴───────────┴───────────────────────────────────────────────────────────────────────┴────────┴──────────────╯ --- Test results for package: cisco_meraki - END --- Done Run pipeline tests for the package --- Test results for package: cisco_meraki - START --- ╭──────────────┬─────────────┬───────────┬────────────────────────────┬────────┬──────────────╮ │ PACKAGE │ DATA STREAM │ TEST TYPE │ TEST NAME │ RESULT │ TIME ELAPSED │ ├──────────────┼─────────────┼───────────┼────────────────────────────┼────────┼──────────────┤ │ cisco_meraki │ events │ pipeline │ test-mx-events.json │ PASS │ 2.33674ms │ │ cisco_meraki │ log │ pipeline │ test-airmarshal-events.log │ PASS │ 26.484352ms │ │ cisco_meraki │ log │ pipeline │ test-events.log │ PASS │ 8.890943ms │ │ cisco_meraki │ log │ pipeline │ test-flows.log │ PASS │ 5.378436ms │ │ cisco_meraki │ log │ pipeline │ test-ip-flow.log │ PASS │ 3.913639ms │ │ cisco_meraki │ log │ pipeline │ test-security-events.log │ PASS │ 3.371085ms │ │ cisco_meraki │ log │ pipeline │ test-urls.log │ PASS │ 3.926439ms │ ╰──────────────┴─────────────┴───────────┴────────────────────────────┴────────┴──────────────╯ --- Test results for package: cisco_meraki - END --- Done Run static tests for the package --- Test results for package: cisco_meraki - START --- ╭──────────────┬─────────────┬───────────┬──────────────────────────┬────────┬──────────────╮ │ PACKAGE │ DATA STREAM │ TEST TYPE │ TEST NAME │ RESULT │ TIME ELAPSED │ ├──────────────┼─────────────┼───────────┼──────────────────────────┼────────┼──────────────┤ │ cisco_meraki │ events │ static │ Verify sample_event.json │ PASS │ 47.401594ms │ │ cisco_meraki │ log │ static │ Verify sample_event.json │ PASS │ 48.464882ms │ ╰──────────────┴─────────────┴───────────┴──────────────────────────┴────────┴──────────────╯ --- Test results for package: cisco_meraki - END --- Done
XML output
$ elastic-package test --report-format xUnit
2022/09/25 10:15:32 WARN CommitHash is undefined, in both .../.elastic-package/version and the compiled binary, config may be out of date.
Run test suite for the package
Run system tests for the package
2022/09/25 10:16:21 INFO Write container logs to file: .../go/src/github.com/elastic/integrations/build/container-logs/meraki-webhook-https-1664066781392044000.log
2022/09/25 10:17:06 INFO Write container logs to file: .../go/src/github.com/elastic/integrations/build/container-logs/meraki-webhook-http-1664066826421459000.log
2022/09/25 10:17:44 INFO Write container logs to file: .../go/src/github.com/elastic/integrations/build/container-logs/cisco_meraki-log-logfile-1664066864397218000.log
2022/09/25 10:18:28 INFO Write container logs to file: .../go/src/github.com/elastic/integrations/build/container-logs/cisco_meraki-log-tcp-1664066908373918000.log
2022/09/25 10:19:14 INFO Write container logs to file: .../go/src/github.com/elastic/integrations/build/container-logs/cisco_meraki-log-udp-1664066954414983000.log
--- Test results for package: cisco_meraki - START ---
<?xml version="1.0" encoding="UTF-8"?>
<testsuites>
<testsuite name="system" tests="5">
<!--test suite for system tests-->
<testcase name="system test: meraki-https" classname="cisco_meraki.events" time="34.309694582"></testcase>
<testcase name="system test: meraki-http" classname="cisco_meraki.events" time="30.706257083"></testcase>
<testcase name="system test: logfile" classname="cisco_meraki.log" time="23.083842672"></testcase>
<testcase name="system test: tcp" classname="cisco_meraki.log" time="30.049010339"></testcase>
<testcase name="system test: udp" classname="cisco_meraki.log" time="29.92422814"></testcase>
</testsuite>
</testsuites>
--- Test results for package: cisco_meraki - END ---
Done
Run asset tests for the package
--- Test results for package: cisco_meraki - START ---
<?xml version="1.0" encoding="UTF-8"?>
<testsuites>
<testsuite name="asset" tests="5">
<!--test suite for asset tests-->
<testcase name="asset test: dashboard cisco_meraki-4832a430-af22-11ec-a899-6f7e676e0fb4 is loaded" classname="cisco_meraki." time="7.9e-07"></testcase>
<testcase name="asset test: index_template logs-cisco_meraki.events is loaded" classname="cisco_meraki.events" time="2.07e-07"></testcase>
<testcase name="asset test: ingest_pipeline logs-cisco_meraki.events-1.2.0 is loaded" classname="cisco_meraki.events" time="1.2e-07"></testcase>
<testcase name="asset test: index_template logs-cisco_meraki.log is loaded" classname="cisco_meraki.log" time="1.47e-07"></testcase>
<testcase name="asset test: ingest_pipeline logs-cisco_meraki.log-1.2.0 is loaded" classname="cisco_meraki.log" time="1.4e-07"></testcase>
</testsuite>
</testsuites>
--- Test results for package: cisco_meraki - END ---
Done
Run pipeline tests for the package
--- Test results for package: cisco_meraki - START ---
<?xml version="1.0" encoding="UTF-8"?>
<testsuites>
<testsuite name="pipeline" tests="7">
<!--test suite for pipeline tests-->
<testcase name="pipeline test: test-mx-events.json" classname="cisco_meraki.events" time="0.002933014"></testcase>
<testcase name="pipeline test: test-airmarshal-events.log" classname="cisco_meraki.log" time="0.022282735"></testcase>
<testcase name="pipeline test: test-events.log" classname="cisco_meraki.log" time="0.006442181"></testcase>
<testcase name="pipeline test: test-flows.log" classname="cisco_meraki.log" time="0.003557785"></testcase>
<testcase name="pipeline test: test-ip-flow.log" classname="cisco_meraki.log" time="0.003611609"></testcase>
<testcase name="pipeline test: test-security-events.log" classname="cisco_meraki.log" time="0.003393975"></testcase>
<testcase name="pipeline test: test-urls.log" classname="cisco_meraki.log" time="0.003288367"></testcase>
</testsuite>
</testsuites>
--- Test results for package: cisco_meraki - END ---
Done
Run static tests for the package
--- Test results for package: cisco_meraki - START ---
<?xml version="1.0" encoding="UTF-8"?>
<testsuites>
<testsuite name="static" tests="2">
<!--test suite for static tests-->
<testcase name="static test: Verify sample_event.json" classname="cisco_meraki.events" time="0.050286524"></testcase>
<testcase name="static test: Verify sample_event.json" classname="cisco_meraki.log" time="0.0515532"></testcase>
</testsuite>
</testsuites>
--- Test results for package: cisco_meraki - END ---
Done
(Verbose output is similar, but is too long to be included due to GitHub limits).
However, the stack version is never emitted, nor is the elastic-package executable version. Locally, these can be found via other means.
On CI, the current recommended approach to finding these versions is to navigate to the build logs and find the phrase "boot up the stack" for the stack version and to look at the go.mod file associated with the state of the tree in the PR. These approaches do no allow the versions to be persisted with the test report and do not scale, particularly in the case where the PR involves updating multiple packages as may be required during stack-wide updates and the stack versions used for different packages in the build are not uniform (as an example a test updating ECS across the SEI packages sees this phrase 182 times in 627311 log lines, needing to be correlated with the failing packages — the signal to noise ratio when a single package is a similar ~1/5000 line/line).
It would be helpful to render these version numbers into the test report so that they can be retained in conjunction with the test results and to allow quick correlation between test outcomes as versions change.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment