Skip to content

Commit

Permalink
Add CI approved test cases tag infrastructure
Browse files Browse the repository at this point in the history 
Use Robot Test Framework's use of Tags to create a subset of tests that
can be run on real and simulated hardware for a basic set of validations
against each commit.

Our yocto qemu session has real basic support and can not pass very many
of the robot tests.  Not because there is a bug but simply because the
simulation does not support many of the openbmc rest interfaces (i.e.
inventory, sensors).  I found some REST and Event Log test suites that
are available in QEMU to start testing now with.  When we move the QEMU
simulation to more of a palmetto based model more test cases will get
the CI tag.  Eventually we will be able to go to a negative model where
we mark the tests that should not be part of a CI run rather then this
small set of includes.

The tag of 'CI' was chosen over a 'QEMUCI' because I fear the use of
tags could get out of hand (BARRELEYECI vs WEDGECI vs BARRELEYE_SIM_CI,
etc).

Invoking....

To test in QEMU first launch the session, find the IP and then run...
OPENBMC_HOST=<ip> tox -e qemu -- --include CI tests

You can also run the CI suite against real hardware...
OPENBMC_HOST=<ip> tox -e barreleye -- --include CI tests

Change-Id: I2c5fbcf2457d958edf7b89c70d7990f5f97e8b75
Signed-off-by: Chris Austen <austenc@us.ibm.com>
  • Loading branch information
Chris Austen committed Jul 8, 2016
1 parent d988e64 commit 859be60
Show file tree
Hide file tree
Showing 3 changed files with 67 additions and 7 deletions.
14 changes: 14 additions & 0 deletions tests/test_eventlog.robot
View file
Original file line number Diff line number Diff line change
Expand Up @@ -19,11 +19,13 @@ ${WAIT_FOR_SERVICES_UP} 3min

valid path to logs
[Documentation] Test list all events
[Tags] CI
${resp} = openbmc get request /org/openbmc/records/events/
should be equal as strings ${resp.status_code} ${HTTP_OK}

clear any logs
[Documentation] Test delete all events
[Tags] CI
${resp} = openbmc post request /org/openbmc/records/events/action/clear data=${NIL}
should be equal as strings ${resp.status_code} ${HTTP_OK}
${resp} = openbmc get request /org/openbmc/records/events/
Expand All @@ -32,29 +34,34 @@ clear any logs

write a log
[Documentation] Test create event
[Tags] CI
create a test log

Message attribute should match
[Documentation] Check message attribute for created event
[Tags] CI
${uri} = create a test log
${content} = Read Attribute ${uri} message
Should Be Equal ${content} A Test event log just happened

Severity attribute should match
[Documentation] Check severity attribute for created event
[Tags] CI
${uri} = create a test log
${content}= Read Attribute ${uri} severity
Should Be Equal ${content} Info

data_bytes attribute should match
[Documentation] Check data_bytes attribute for created event
[Tags] CI
@{data_list} = Create List ${48} ${0} ${19} ${127} ${136} ${255}
${uri} = create a test log
${content} = Read Attribute ${uri} debug_data
Lists Should Be Equal ${content} ${data_list}

delete the log
[Documentation] Test the delete event
[Tags] CI
${uri} = create a test log
${deluri} = catenate SEPARATOR= ${uri} /action/delete
${resp} = openbmc post request ${deluri} data=${NIL}
Expand All @@ -64,6 +71,7 @@ delete the log

2nd delete should fail
[Documentation] Negative scnenario to delete already deleted event
[Tags] CI
${uri} = create a test log
${deluri} = catenate SEPARATOR= ${uri} /action/delete
${resp} = openbmc post request ${deluri} data=${NIL}
Expand All @@ -78,6 +86,7 @@ Intermixed delete
... delete middle log
... middle log should not exist
... time stamp should not match between logs(1st and 3rd)
[Tags] CI
${event1}= create a test log
${event2}= create a test log
${event3}= create a test log
Expand All @@ -91,6 +100,7 @@ Intermixed delete
restarting event process retains logs
[Documentation] This is to test events are in place even after the
... event service is restarted.
[Tags] CI
${resp} = openbmc get request /org/openbmc/records/events/
${json} = to json ${resp.content}
${logs_pre_restart}= set variable ${json['data']}
Expand All @@ -107,6 +117,7 @@ restarting event process retains logs
deleting log after obmc-phosphor-event.service restart
[Documentation] This is to test event can be deleted created prior to
... event service is restarted.
[Tags] CI
${uri}= create a test log

Open Connection And Log In
Expand All @@ -120,6 +131,7 @@ deleting log after obmc-phosphor-event.service restart
makeing new log after obmc-phosphor-event.service restart
[Documentation] This is for testing event creation after the
... event service is restarted.
[Tags] CI
Open Connection And Log In
${uptime}= Execute Command systemctl restart obmc-phosphor-event.service
Sleep ${10}
Expand All @@ -129,6 +141,7 @@ makeing new log after obmc-phosphor-event.service restart
deleting new log after obmc-phosphor-event.service restart
[Documentation] This testcase is for testing deleted newly created event
... after event service is restarted.
[Tags] CI
Open Connection And Log In
${uptime}= Execute Command systemctl restart obmc-phosphor-event.service
Sleep ${10}
Expand Down Expand Up @@ -170,6 +183,7 @@ Test events after openbmc reboot

clearing logs results in no logs
[Documentation] This testcase is for clearning the events when no logs present
[Tags] CI
${resp} = openbmc post request /org/openbmc/records/events/action/clear data=${NIL}
should be equal as strings ${resp.status_code} ${HTTP_OK}
${resp} = openbmc get request /org/openbmc/records/events/
Expand Down
41 changes: 41 additions & 0 deletions tests/test_obmcrest.robot
View file
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ Resource ../lib/rest_client.robot

*** Test Cases ***
Good connection for testing
[Tags] CI
${content}= Read Properties /
${c}= get from List ${content} 0
Should Be Equal ${c} /org
Expand All @@ -30,74 +31,86 @@ Get a null Property
Should Be Equal ${jsondata['data']['description']} The specified property cannot be found: ''is_fru''

get directory listing /
[Tags] CI
${resp} = openbmc get request /
should be equal as strings ${resp.status_code} ${HTTP_OK}
${json} = to json ${resp.content}
list should contain value ${json['data']} /org
should be equal as strings ${json['status']} ok

get directory listing /org/
[Tags] CI
${resp} = openbmc get request /org/
should be equal as strings ${resp.status_code} ${HTTP_OK}
${json} = to json ${resp.content}
list should contain value ${json['data']} /org/openbmc
should be equal as strings ${json['status']} ok

get invalid directory listing /i/dont/exist/
[Tags] CI
${resp} = openbmc get request /i/dont/exist/
should be equal as strings ${resp.status_code} ${HTTP_NOT_FOUND}
${json} = to json ${resp.content}
should be equal as strings ${json['status']} error

put directory listing /
[Tags] CI
${resp} = openbmc put request /
should be equal as strings ${resp.status_code} ${HTTP_METHOD_NOT_ALLOWED}
${json} = to json ${resp.content}
should be equal as strings ${json['status']} error

put directory listing /org/
[Tags] CI
${resp} = openbmc put request /org/
should be equal as strings ${resp.status_code} ${HTTP_METHOD_NOT_ALLOWED}
${json} = to json ${resp.content}
should be equal as strings ${json['status']} error

put invalid directory listing /i/dont/exist/
[Tags] CI
${resp} = openbmc put request /i/dont/exist/
should be equal as strings ${resp.status_code} ${HTTP_NOT_FOUND}
${json} = to json ${resp.content}
should be equal as strings ${json['status']} error

post directory listing /
[Tags] CI
${resp} = openbmc post request /
should be equal as strings ${resp.status_code} ${HTTP_METHOD_NOT_ALLOWED}
${json} = to json ${resp.content}
should be equal as strings ${json['status']} error

post directory listing /org/
[Tags] CI
${resp} = openbmc post request /org/
should be equal as strings ${resp.status_code} ${HTTP_METHOD_NOT_ALLOWED}
${json} = to json ${resp.content}
should be equal as strings ${json['status']} error

post invalid directory listing /i/dont/exist/
[Tags] CI
${resp} = openbmc post request /i/dont/exist/
should be equal as strings ${resp.status_code} ${HTTP_NOT_FOUND}
${json} = to json ${resp.content}
should be equal as strings ${json['status']} error

delete directory listing /
[Tags] CI
${resp} = openbmc delete request /
should be equal as strings ${resp.status_code} ${HTTP_METHOD_NOT_ALLOWED}
${json} = to json ${resp.content}
should be equal as strings ${json['status']} error

delete directory listing /org/
[Tags] CI
${resp} = openbmc delete request /
should be equal as strings ${resp.status_code} ${HTTP_METHOD_NOT_ALLOWED}
${json} = to json ${resp.content}
should be equal as strings ${json['status']} error

delete invalid directory listing /org/nothere/
[Tags] CI
${resp} = openbmc delete request /org/nothere/
should be equal as strings ${resp.status_code} ${HTTP_NOT_FOUND}
${json} = to json ${resp.content}
Expand All @@ -118,60 +131,70 @@ get list names /org/
should be equal as strings ${json['status']} ok

get invalid list names /i/dont/exist/
[Tags] CI
${resp} = openbmc get request /i/dont/exist/list
should be equal as strings ${resp.status_code} ${HTTP_NOT_FOUND}
${json} = to json ${resp.content}
should be equal as strings ${json['status']} error

put list names /
[Tags] CI
${resp} = openbmc put request /list
should be equal as strings ${resp.status_code} ${HTTP_METHOD_NOT_ALLOWED}
${json} = to json ${resp.content}
should be equal as strings ${json['status']} error

put list names /org/
[Tags] CI
${resp} = openbmc put request /org/list
should be equal as strings ${resp.status_code} ${HTTP_METHOD_NOT_ALLOWED}
${json} = to json ${resp.content}
should be equal as strings ${json['status']} error

put invalid list names /i/dont/exist/
[Tags] CI
${resp} = openbmc put request /i/dont/exist/list
should be equal as strings ${resp.status_code} ${HTTP_NOT_FOUND}
${json} = to json ${resp.content}
should be equal as strings ${json['status']} error

post list names /
[Tags] CI
${resp} = openbmc post request /list
should be equal as strings ${resp.status_code} ${HTTP_METHOD_NOT_ALLOWED}
${json} = to json ${resp.content}
should be equal as strings ${json['status']} error

post list names /org/
[Tags] CI
${resp} = openbmc post request /org/list
should be equal as strings ${resp.status_code} ${HTTP_METHOD_NOT_ALLOWED}
${json} = to json ${resp.content}
should be equal as strings ${json['status']} error

post invalid list names /i/dont/exist/
[Tags] CI
${resp} = openbmc post request /i/dont/exist/list
should be equal as strings ${resp.status_code} ${HTTP_NOT_FOUND}
${json} = to json ${resp.content}
should be equal as strings ${json['status']} error

delete list names /
[Tags] CI
${resp} = openbmc delete request /list
should be equal as strings ${resp.status_code} ${HTTP_METHOD_NOT_ALLOWED}
${json} = to json ${resp.content}
should be equal as strings ${json['status']} error

delete list names /org/
[Tags] CI
${resp} = openbmc delete request /list
should be equal as strings ${resp.status_code} ${HTTP_METHOD_NOT_ALLOWED}
${json} = to json ${resp.content}
should be equal as strings ${json['status']} error

delete invalid list names /org/nothere/
[Tags] CI
${resp} = openbmc delete request /org/nothere/list
should be equal as strings ${resp.status_code} ${HTTP_NOT_FOUND}
${json} = to json ${resp.content}
Expand All @@ -192,84 +215,98 @@ get names /org/
should be equal as strings ${json['status']} ok

get invalid names /i/dont/exist/
[Tags] CI
${resp} = openbmc get request /i/dont/exist/enumerate
should be equal as strings ${resp.status_code} ${HTTP_NOT_FOUND}
${json} = to json ${resp.content}
should be equal as strings ${json['status']} error

put names /
[Tags] CI
${resp} = openbmc put request /enumerate
should be equal as strings ${resp.status_code} ${HTTP_METHOD_NOT_ALLOWED}
${json} = to json ${resp.content}
should be equal as strings ${json['status']} error

put names /org/
[Tags] CI
${resp} = openbmc put request /org/enumerate
should be equal as strings ${resp.status_code} ${HTTP_METHOD_NOT_ALLOWED}
${json} = to json ${resp.content}
should be equal as strings ${json['status']} error

put invalid names /i/dont/exist/
[Tags] CI
${resp} = openbmc put request /i/dont/exist/enumerate
should be equal as strings ${resp.status_code} ${HTTP_NOT_FOUND}
${json} = to json ${resp.content}
should be equal as strings ${json['status']} error

post names /
[Tags] CI
${resp} = openbmc post request /enumerate
should be equal as strings ${resp.status_code} ${HTTP_METHOD_NOT_ALLOWED}
${json} = to json ${resp.content}
should be equal as strings ${json['status']} error

post names /org/
[Tags] CI
${resp} = openbmc post request /org/enumerate
should be equal as strings ${resp.status_code} ${HTTP_METHOD_NOT_ALLOWED}
${json} = to json ${resp.content}
should be equal as strings ${json['status']} error

post invalid names /i/dont/exist/
[Tags] CI
${resp} = openbmc post request /i/dont/exist/enumerate
should be equal as strings ${resp.status_code} ${HTTP_NOT_FOUND}
${json} = to json ${resp.content}
should be equal as strings ${json['status']} error

delete names /
[Tags] CI
${resp} = openbmc delete request /enumerate
should be equal as strings ${resp.status_code} ${HTTP_METHOD_NOT_ALLOWED}
${json} = to json ${resp.content}
should be equal as strings ${json['status']} error

delete names /org/
[Tags] CI
${resp} = openbmc delete request /enumerate
should be equal as strings ${resp.status_code} ${HTTP_METHOD_NOT_ALLOWED}
${json} = to json ${resp.content}
should be equal as strings ${json['status']} error

delete invalid names /org/nothere/
[Tags] CI
${resp} = openbmc delete request /org/nothere/enumerate
should be equal as strings ${resp.status_code} ${HTTP_NOT_FOUND}
${json} = to json ${resp.content}
should be equal as strings ${json['status']} error

get method org/openbmc/records/events/action/acceptTestMessage
[Tags] CI
${resp} = openbmc get request org/openbmc/records/events/action/acceptTestMessage
should be equal as strings ${resp.status_code} ${HTTP_METHOD_NOT_ALLOWED}
${json} = to json ${resp.content}
should be equal as strings ${json['status']} error

get invalid method /i/dont/exist/
[Tags] CI
${resp} = openbmc get request /i/dont/exist/action/foo
should be equal as strings ${resp.status_code} ${HTTP_NOT_FOUND}
${json} = to json ${resp.content}
should be equal as strings ${json['status']} error

put method org/openbmc/records/events/action/acceptTestMessage
[Tags] CI
${resp} = openbmc put request org/openbmc/records/events/action/acceptTestMessage
should be equal as strings ${resp.status_code} ${HTTP_METHOD_NOT_ALLOWED}
${json} = to json ${resp.content}
should be equal as strings ${json['status']} error

put invalid method /i/dont/exist/
[Tags] CI
${resp} = openbmc put request /i/dont/exist/action/foo
should be equal as strings ${resp.status_code} ${HTTP_NOT_FOUND}
${json} = to json ${resp.content}
Expand All @@ -284,6 +321,7 @@ post method power/getPowerState no args
should be equal as strings ${json['status']} ok

post method org/openbmc/records/events/action/acceptTestMessage invalid args
[Tags] CI
${data} = create dictionary foo=bar
${resp} = openbmc post request org/openbmc/records/events/action/acceptTestMessage data=${data}
should be equal as strings ${resp.status_code} ${HTTP_BAD_REQUEST}
Expand All @@ -303,18 +341,21 @@ post method org/openbmc/sensors/host/BootCount with args
Should Be Equal ${content} ${COUNT}

delete method org/openbmc/records/events/action/acceptTestMessage
[Tags] CI
${resp} = openbmc delete request org/openbmc/records/events/action/acceptTestMessage
should be equal as strings ${resp.status_code} ${HTTP_METHOD_NOT_ALLOWED}
${json} = to json ${resp.content}
should be equal as strings ${json['status']} error

delete invalid method /org/nothere/
[Tags] CI
${resp} = openbmc delete request /org/nothere/action/foomethod
should be equal as strings ${resp.status_code} ${HTTP_NOT_FOUND}
${json} = to json ${resp.content}
should be equal as strings ${json['status']} error

post method org/openbmc/records/events/action/acceptTestMessage no args
[Tags] CI
${data} = create dictionary data=@{EMPTY}
${resp} = openbmc post request org/openbmc/records/events/action/acceptTestMessage data=${data}
should be equal as strings ${resp.status_code} ${HTTP_OK}
Expand Down
Loading

0 comments on commit 859be60

Please sign in to comment.