Skip to content

Commit 48910a0

Browse files
mmulhollammulhollMartin Mulholland
authored
Add new tests for v200 schema (#199)
* new API Tests * Code review comments and new directory structure Co-authored-by: Martin Mulholland <mmulholl@us.ibm.com> Co-authored-by: Martin Mulholland <martinmulholland@mmulholl-mac.lan>
1 parent 2734643 commit 48910a0

File tree

162 files changed

+3569
-0
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

162 files changed

+3569
-0
lines changed

‎.gitignore

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,8 @@
11
# Temporary Build Files
22
build/_output
33
build/_test
4+
test/v200/schemaTest/tmp
5+
test/go/pkg
46
# Created by https://www.gitignore.io/api/go,vim,emacs,visualstudiocode
57
### Emacs ###
68
# -*- mode: gitignore; -*-
@@ -78,3 +80,5 @@ tags
7880
# IDE folders
7981
.idea
8082
generator/build/generator
83+
#Mac file
84+
.DS_Store

‎go.mod

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3,10 +3,12 @@ module github.com/devfile/api/v2
33
go 1.13
44

55
require (
6+
github.com/ghodss/yaml v1.0.0
67
github.com/google/go-cmp v0.4.0
78
github.com/google/gofuzz v1.2.0
89
github.com/hashicorp/go-multierror v1.1.0
910
github.com/mitchellh/reflectwalk v1.0.1
11+
github.com/santhosh-tekuri/jsonschema v1.2.4
1012
github.com/stretchr/testify v1.4.0
1113
k8s.io/api v0.18.6
1214
k8s.io/apiextensions-apiserver v0.18.6

‎go.sum

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -68,6 +68,7 @@ github.com/fsnotify/fsnotify v1.4.7/go.mod h1:jwhsz4b93w/PPRr/qN1Yymfu8t87LnFCMo
6868
github.com/fsnotify/fsnotify v1.4.9 h1:hsms1Qyu0jgnwNXIxa+/V/PDsU6CfLf6CNO8H7IWoS4=
6969
github.com/fsnotify/fsnotify v1.4.9/go.mod h1:znqG4EE+3YCdAaPaxE2ZRY/06pZUdp0tY4IgpuI1SZQ=
7070
github.com/ghodss/yaml v0.0.0-20150909031657-73d445a93680/go.mod h1:4dBDuWmgqj2HViK6kFavaiC9ZROes6MMH2rRYeMEF04=
71+
github.com/ghodss/yaml v1.0.0 h1:wQHKEahhL6wmXdzwWG11gIVCkOv05bNOh+Rxn0yngAk=
7172
github.com/ghodss/yaml v1.0.0/go.mod h1:4dBDuWmgqj2HViK6kFavaiC9ZROes6MMH2rRYeMEF04=
7273
github.com/globalsign/mgo v0.0.0-20180905125535-1ca0a4f7cbcb/go.mod h1:xkRDCp4j0OGD1HRkm4kmhM+pmpv3AKq5SU7GMg4oO/Q=
7374
github.com/globalsign/mgo v0.0.0-20181015135952-eeefdecb41b8/go.mod h1:xkRDCp4j0OGD1HRkm4kmhM+pmpv3AKq5SU7GMg4oO/Q=
@@ -267,6 +268,8 @@ github.com/prometheus/procfs v0.0.2/go.mod h1:TjEm7ze935MbeOT/UhFTIMYKhuLP4wbCsT
267268
github.com/prometheus/procfs v0.0.11/go.mod h1:lV6e/gmhEcM9IjHGsFOCxxuZ+z1YqCvr4OA4YeYWdaU=
268269
github.com/rogpeppe/fastuuid v0.0.0-20150106093220-6724a57986af/go.mod h1:XWv6SoW27p1b0cqNHllgS5HIMJraePCO15w5zCzIWYg=
269270
github.com/russross/blackfriday v1.5.2/go.mod h1:JO/DiYxRf+HjHt06OyowR9PTA263kcR/rfWxYHBV53g=
271+
github.com/santhosh-tekuri/jsonschema v1.2.4 h1:hNhW8e7t+H1vgY+1QeEQpveR6D4+OwKPXCfD2aieJis=
272+
github.com/santhosh-tekuri/jsonschema v1.2.4/go.mod h1:TEAUOeZSmIxTTuHatJzrvARHiuO9LYd+cIxzgEHCQI4=
270273
github.com/sergi/go-diff v1.0.0/go.mod h1:0CfEIISq7TuYL3j771MWULgwwjU+GofnZX9QAmXWZgo=
271274
github.com/sirupsen/logrus v1.2.0/go.mod h1:LxeOpSwHxABJmUn/MG1IvRgCAasNZTLOkJPxbbu5VWo=
272275
github.com/sirupsen/logrus v1.4.2/go.mod h1:tLMulIdttU9McNUspp0xgXVQah82FyeX6MwdIuYE2rE=

‎test/README.md

Lines changed: 46 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,46 @@
1+
# API Tests
2+
3+
The API tests are intended to provide a comprehensive verification of the devfile schemas. This includes:
4+
- Ensuring every possible attribute is valid.
5+
- Ensuring all optional attributes are indeed optional.
6+
- Ensuring any possible specification errors are invalidated by the schema. For example:
7+
- Missing mandatory attributes.
8+
- Multiple use of a one-of attribute.
9+
- Attribute values of the wrong type.
10+
11+
## Test structure
12+
13+
- `test/v200/devfiles` : contains yaml snippets which are used to generate yaml files for the tests. The names of the sub-directories and files should reflect their purpose.
14+
- `test/v200/schemaTest/schema-test.go` : the go unit test program.
15+
- `test/v200/json` : contains the json files which define the tests which the test program will run:
16+
- `test-xxxxxxx.json` : these files are the top level json files, they define the schema to verify and the test files to run.
17+
- `xxxxxx-tests.json` : these are the test files which contain individual tests which provide the yaml snippets to combine and the expected result.
18+
19+
## Running tests
20+
21+
from the test/go/src/test directory run
22+
- `go test -v`
23+
24+
The test will read each of the test-xxxxxx.json files and run the tests defined within. The generated .yaml files used for the tests are created in a `tmp/test-xxxxxx/` directory. These files are not deleted when the test finishes so they can be used to assess any errors, however they will be deleted by a subsequent run of the test. Running the test with the -v option ensures you see a full list of passes and failures.
25+
26+
## Adding Tests
27+
28+
### add a test for a new schema file
29+
30+
1. Create a new `test/v200/json/test-<schema name>.json` file for the schema. In the json file specify the location of the schema to test (relative to the root directory of the repository), and the list of the existing tests to use. If the generated yaml files require a schemaVersion attribute include its value in the json file. see - *link to sample schema to be added*
31+
1. Run the test
32+
33+
### add a test for a schema changes
34+
35+
1. Modify an existing yaml snippet or create a new one.
36+
1. If appropriate create a new snippet for any possible error cases, for example to omit a required attribute.
37+
1. If a new yaml snippet was created add a test which uses the snippet to the appropriate `json/xxxxxx-tests.json` file. Be careful to ensure the file name used for the test is unique for all tests - this is the name used for the yaml file which is generated for the test. For failure scenarios you may need to run the test first to set the outcome correctly.
38+
1. If a new `json/xxxxxx-tests.json` file is created, any existing `test-xxxxxxx.json` files must be updated to use the new file.
39+
40+
### add test for a new schema version
41+
42+
1. Copy and rename the `test/v200` directory for the new version, for example `test\v201`
43+
1. Update the copied `test/v201/json/test-<schema name>.json` files to point to the new schema.
44+
1. Modify the copied tests as needed for the new version as decsribed above.
45+
1. Add `test/v201/schemaTest/tmp` to the .gitignore file.
46+
1. Run the test

‎test/v200/devfiles/commands/VsCodeLaunchAll.yaml

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
- id: testvscode1
2+
attributes:
3+
test: Exec Test test
4+
scope: Api
5+
vscodeLaunch:
6+
inlined: inline
7+
group:
8+
isDefault: false
9+
kind: test
10+
- id: testvscode12
11+
vscodeLaunch:
12+
uri: http://testingland
13+
group:
14+
isDefault: false
15+
kind: run
16+
- id: testvscode13
17+
vscodeLaunch:
18+
inlined: inline
19+
group:
20+
isDefault: true
21+
kind: debug
22+
- id: testvscode14
23+
vscodeLaunch:
24+
uri: http://testingland
25+
group:
26+
isDefault: false
27+
kind: build

‎test/v200/devfiles/commands/VsCodeLaunchBadGroupIsDefault.yaml

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
- id: testvscode
2+
vscodeLaunch:
3+
inlined: inline
4+
group:
5+
isDefault: nope
6+
kind: test

‎test/v200/devfiles/commands/VsCodeLaunchBadGroupKind.yaml

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
- id: testvscode
2+
vscodeLaunch:
3+
inlined: inline
4+
group:
5+
isDefault: false
6+
kind: free

‎test/v200/devfiles/commands/VsCodeLaunchInlinedAndUri.yaml

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
- id: testvscode1
2+
vscodeLaunch:
3+
inlined: inline
4+
uri: "http://testingland"

‎test/v200/devfiles/commands/VsCodeLaunchMinimum.yaml

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
- id: testvscode1
2+
vscodeLaunch:
3+
uri: http://testingland
4+
- id: testvscode2
5+
vscodeLaunch:
6+
inlined: inline

‎test/v200/devfiles/commands/VsCodeLaunchMissingGroupKind.yaml

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
- id: testvscode
2+
vscodeLaunch:
3+
inlined: inline
4+
group:
5+
isDefault: false
6+
kind: free

‎test/v200/devfiles/commands/VsCodeLaunchMissingInlinedAndUri.yaml

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
- id: testvscode
2+
vscodeLaunch:
3+
group:
4+
isDefault: false
5+
kind: test

‎test/v200/devfiles/commands/VsCodeLaunchMixedCaseId.yaml

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
- id: testVscode1
2+
vscodeLaunch:
3+
uri: http://testingland
4+
- id: testvscode2
5+
vscodeLaunch:
6+
inlined: inline

‎test/v200/devfiles/commands/VsCodeLaunchNoId.yaml

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
- vscodeLaunch:
2+
uri: http://testingland

‎test/v200/devfiles/commands/VsCodeTaskAll.yaml

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
- id: testvscodetask1
2+
attributes:
3+
test: Exec Test test
4+
scope: Api
5+
vscodeTask:
6+
inlined: inline
7+
group:
8+
isDefault: false
9+
kind: test
10+
- id: testvscodetask2
11+
vscodeTask:
12+
uri: http://testingland
13+
group:
14+
isDefault: false
15+
kind: run
16+
- id: testvscodetask3
17+
vscodeTask:
18+
inlined: inline
19+
group:
20+
isDefault: true
21+
kind: debug
22+
- id: testvscodetask4
23+
vscodeTask:
24+
uri: http://testingland
25+
group:
26+
isDefault: false
27+
kind: build

‎test/v200/devfiles/commands/VsCodeTaskBadGroupIsDefault.yaml

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
- id: testvscodetask
2+
vscodeTask:
3+
inlined: inline
4+
group:
5+
isDefault: nada
6+
kind: test

‎test/v200/devfiles/commands/VsCodeTaskBadGroupKind.yaml

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
- id: testvscodetask
2+
vscodeTask:
3+
inlined: inline
4+
group:
5+
isDefault: false
6+
kind: free

‎test/v200/devfiles/commands/VsCodeTaskInlinedAndUri.yaml

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
- id: testvscodetask
2+
vscodeTask:
3+
inlined: inline
4+
uri: "http://testingland"

‎test/v200/devfiles/commands/VsCodeTaskMinimum.yaml

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
- id: testvscodetask1
2+
vscodeTask:
3+
uri: http://testingland
4+
- id: testvscodetask2
5+
vscodeTask:
6+
inlined: inline

‎test/v200/devfiles/commands/VsCodeTaskMissingGroupKind.yaml

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
- id: testvscodetask
2+
vscodeTask:
3+
inlined: inline
4+
group:
5+
isDefault: false
6+
kind: free

‎test/v200/devfiles/commands/VsCodeTaskMissingInlinedAndUri.yaml

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
- id: testvscodetask
2+
vscodeTask:
3+
group:
4+
isDefault: false
5+
kind: test

‎test/v200/devfiles/commands/VsCodeTaskMixedCaseId.yaml

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
- id: testVscodetask1
2+
vscodeTask:
3+
uri: http://testingland
4+
- id: testvscodetask2
5+
vscodeTask:
6+
inlined: inline

‎test/v200/devfiles/commands/VsCodeTaskNoId.yaml

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
- vscodeTask:
2+
uri: http://testingland

‎test/v200/devfiles/commands/applyAll.yaml

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,31 @@
1+
- id: testapplytest
2+
attributes:
3+
test: Apply Test
4+
scope: Api
5+
apply:
6+
component: api-testing
7+
group:
8+
isDefault: true
9+
kind: test
10+
label: Test Apply test
11+
- id: testapplybuild
12+
apply:
13+
component: api-testing
14+
group:
15+
isDefault: true
16+
kind: build
17+
label: Test Apply build
18+
- id: testapplyrun
19+
apply:
20+
component: api-testing
21+
group:
22+
isDefault: false
23+
kind: run
24+
label: Test Apply run
25+
- id: testapplydebug
26+
apply:
27+
component: api-testing
28+
group:
29+
isDefault: false
30+
kind: debug
31+
label: Test Apply debug

‎test/v200/devfiles/commands/applyBadGroupIsDefault.yaml

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
- id: testapply
2+
apply:
3+
component: api-testing
4+
group:
5+
isDefault: fasle
6+
kind: test
7+
label: Test Apply command

‎test/v200/devfiles/commands/applyBadGroupKind.yaml

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
- id: testapply
2+
apply:
3+
component: api-testing
4+
group:
5+
isDefault: true
6+
kind: green
7+
label: Test Apply command

‎test/v200/devfiles/commands/applyMinimum.yaml

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
- id: testapplytest1
2+
apply:
3+
component: api-testing
4+
- id: testapplytest2
5+
apply:
6+
component: api-testing
7+
group:
8+
kind: test

‎test/v200/devfiles/commands/applyMissingGroupKind.yaml

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
- id: testapply
2+
apply:
3+
component: api-testing
4+
group:
5+
isDefault: true
6+
label: Test Apply command

‎test/v200/devfiles/commands/applyMixedCaseId.yaml

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
- id: testApplytest1
2+
apply:
3+
component: api-testing
4+
- id: testapplytest2
5+
apply:
6+
component: api-testing
7+
group:
8+
kind: test

‎test/v200/devfiles/commands/applyNoComponent.yaml

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
- id: testapply
2+
apply:
3+
group:
4+
isDefault: true
5+
kind: test
6+
label: Test Apply command

‎test/v200/devfiles/commands/applyNoId.yaml

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
- apply:
2+
component: api-testing
3+
group:
4+
isDefault: true
5+
kind: test
6+
label: Test Apply test

‎test/v200/devfiles/commands/commandStart.yaml

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
commands:

‎test/v200/devfiles/commands/compositeAll.yaml

Lines changed: 43 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,43 @@
1+
- id: compositetest1
2+
attributes:
3+
test: Composite Test test
4+
scope: Api
5+
composite:
6+
label: Composite Test
7+
commands:
8+
- runTest1
9+
- runTest2
10+
parallel: false
11+
group:
12+
isDefault: true
13+
kind: test
14+
- id: compositetest2
15+
composite:
16+
label: Composite Test
17+
commands:
18+
- runTest1
19+
- runTest2
20+
parallel: true
21+
group:
22+
isDefault: true
23+
kind: debug
24+
- id: compositetest3
25+
composite:
26+
label: Composite Test
27+
commands:
28+
- runTest1
29+
- runTest2
30+
parallel: false
31+
group:
32+
isDefault: false
33+
kind: test
34+
- id: compositetest4
35+
composite:
36+
label: Composite Test
37+
commands:
38+
- runTest1
39+
- runTest2
40+
parallel: true
41+
group:
42+
isDefault: false
43+
kind: build

‎test/v200/devfiles/commands/compositeBadGroupIsDefault.yaml

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
- id: compositetest1
2+
composite:
3+
label: Composite Test
4+
commands:
5+
- runTest1
6+
- runTest2
7+
parallel: false
8+
group:
9+
isDefault: confirm
10+
kind: test

0 commit comments

Comments
 (0)