Skip to content

Commit 381fd43

Browse files
CelendTJT
authored andcommitted
Create EIP-010-integration-test-framework.md
1 parent f585f25 commit 381fd43

File tree

1 file changed

+67
-0
lines changed

1 file changed

+67
-0
lines changed
Lines changed: 67 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,67 @@
1+
# Integration test framework
2+
3+
## Purpose
4+
5+
Extism have many language's SDk and PDK, which is excellent! but we're lack of tests to make sure all SDK and PDK are working expectedly.
6+
7+
for example: we made a change to GO-PDK, tested with GO-SDK, Rust-SDK but breaks JS-SDK.
8+
9+
So we need a integration test framework to test over all SDK/PDK permutations.
10+
11+
## Solution
12+
13+
### Key concepts
14+
15+
#### TestTarget
16+
Represents the test running environment,
17+
Includes env(different toolchains), PDK(and/or version), SDK(and/or version).
18+
19+
example:
20+
```yaml
21+
env:
22+
- tiny-go=23.0.1
23+
sdk: js-sdk=2.0.0-rc
24+
pdk: go-pdk=latest
25+
```
26+
27+
env feature is optional.
28+
29+
#### TestSuite
30+
Represents a group of TestCases with options.
31+
32+
options:
33+
- targets: target matchers, use to skip unimplemented/unsupported target or env
34+
35+
#### TestCase
36+
- inside a TestSuite
37+
- have an expected output to validate
38+
39+
40+
### Workflow
41+
42+
1. design TestSuite and TestCases for ALL SDK/PDK
43+
2. implement TestSuites
44+
- each PDK/SDK to test should implement the TestSuite code, follow the expected output
45+
- store in a standardlong repo?
46+
3. define TestTargets(optional)
47+
- can define additaional TestTargets to run test suites
48+
4. trigger runner by new version released or by manual
49+
50+
51+
### Runflow
52+
Use your favorite build tools to implement, recomment base on Docker.
53+
54+
1. scan TestSuite and TestCases to generate inital TestTargets
55+
2. mix inital TestTargets with additaional TestTargets
56+
3. prepare environment
57+
- compile wasm binaries with specified toolchain and PDK
58+
- prepare SDKs
59+
4. run all TestTargets
60+
5. check output is expected and generate test report
61+
62+
63+
## Considerations
64+
- need a Runner to run SDK code and PDK binary, also can be used to benchmark
65+
- better not use assert library to check output, just compare output itself to keep it simple.
66+
- TestSuite versioning, consider SDK/PDK may introduce breaking change
67+
- prevent overcomplicated

0 commit comments

Comments
 (0)