From f17c2a10c6f19459615533403478ba8cb478c6b9 Mon Sep 17 00:00:00 2001 From: szatmz <39386237+szatmz@users.noreply.github.com> Date: Mon, 29 Jun 2020 15:02:49 -0400 Subject: [PATCH] Requirements document for on-device testing (#1161) (#1283) * Requirements document for on-device testing (#1161) Test framework, dispatching, device interaction, result collection are discussed. * Status: first draft * Restyled by prettier-markdown Co-authored-by: Restyled.io --- docs/api/device_runner.md | 103 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 103 insertions(+) create mode 100644 docs/api/device_runner.md diff --git a/docs/api/device_runner.md b/docs/api/device_runner.md new file mode 100644 index 00000000000000..3b621eb8e70675 --- /dev/null +++ b/docs/api/device_runner.md @@ -0,0 +1,103 @@ +# CHIP on-device testing + +_Requirements and high-level design_ + +## Background + +The ability to run tests on actual and emulated hardware is paramount in +embedded projects. CHIP is no exception. We want on-device testing to be a first +class goal of CHIP architecture. On-device testing requirements apply both to +Continuous Integration testing for main CHIP software stack development and to +eventual CHIP product certification. This document explores the requirements and +evaluates potential solutions. + +## Overview of requirements + +A good device test infrastructure is built on four pillars. + +### Pillar 1: Using a test framework + +A test framework provides a testing structure that developers can follow and +potentially reduces some of the burden of test setup and teardown (less +boilerplate). Support for state-oriented and asynchronous structuring of tests +would be beneficial. Many test frameworks leverage scripting languages such as +Python to simplify the quick development of tests and to leverage rich sets of +libraries for device/systems access and results generation. + +### Pillar 2: Dispatching tests + +Tests can run on lab machines or on the developer's local workstation. Tests can +be triggered manually by the developer or as a result of completion of a +changeset built on a continuous integration (CI) server. CHIP involves multiple +stakeholders, many of which will want to contribute to the testing efforts with +lab capacity. The infrastructure therefore must be prepared for +cross-organization test dispatch. + +To facilitate uniform dispatch of tests we will probably need a simple +request/response protocol. Potentially HTTPS based and RESTful. Due to the long +running nature of device tests the response for a test scheduling request could +be a test ID, not the test result. That ID could be used to query the test +status, subscribe for notifications on status changes and to pull the test +results. Core aspects of such a scheme include the conventions for request +artifacts contents and minimum expected results contents once the run is +complete. + +### Pillar 3: Interacting with devices + +The test host environment has to reset devices, flash images on them, issue +commands, monitor status and collect test results. It may also need to integrate +both virtual (simulated) and real devices together. This can at first be done in +an ad-hoc way per platform but eventually we can go into device access +abstraction, i.e. define a common device testing interface which CHIP-compliant +devices can expose. The test host has to be prepared for driving multiple +devices at the same time for a single test, e.g. for tests that check +communication between multiple devices. + +### Pillar 4: Collecting results + +Ideally, test results are output in standard formats and similar or analogous +results between different devices and tests are output the same way. This +ensures reusability of code that processes similar data while allowing +aggregation of results across different dimensions. Failed tests must propagate +errors from device platform layers all the way to the CHIP stack and present +errors and potential stack traces in a standard result format. As the purpose of +on-device tests is to capture bugs, it is important that the test outputs +highlight the failure reason(s) and developers don't have to browse through +thousands of lines of logs to find the one line that sheds light on why a test +failed. + +## Priorities + +In the spirit of CHIP's charter, it would be great to see something taking-off +as soon as possible, to support continuous testing of the evolving CHIP stack. +We could then improve on that first iteration, even if we have to throw away +some temporary concepts and code. + +Test dispatch (Pillar 2) arises as the highest priority, because all other +pillars can have ad-hoc solutions. The first need is an interface between a +CircleCI job and a test execution host at a participating organization. This +would enable dispatching tests to a variety of existing in-house infrastructure, +while retaining common request/response protocols to shield the CI system from +implementation details of each lab. + +The next most important goal is to provide a test framework (Pillar 1). With a +standard framework developers can start writing tests, even if those tests will +be device specific and of ad-hoc input and output format. The general structure +of tests will however be present and later the tests can be adapted to standard +interactions (Pillar 3) and result formats (Pillar 4). + +Specifying result formats (Pillar 4) for the most common outputs +(success/failure, failure reason, stack trace, memory and CPU usage time series, +pcaps of network traffic, etc.) will be an ongoing effort. The simplest output +formats can be specified together with the test framework. + +Lastly, we want to look into a common device interaction interface that would +enable reusing tests between different devices. + +## Baseline hardware platforms for CHIP + +The TSG is targeting the following platforms/boards for early bringup: + +- Nordic nRF52 board +- SiLabs XXXX board +- Espressif ESP32 XXXX board