Implement functional Wifi tests #5161

SeppoTakalo · 2017-09-21T14:12:08Z

Description

Implement 100% function coverage for WifiInterface as specified
in "Wifi test plan"

Idea is to have full coverage of WiFiInterface API callbacks. Does not test any of the socket functionality, those will be provided later, perhaps within #4795

Uses idea of separating the code which detects the interface into one singe file get_interface.c and its callback get_interface() which always gives a new initialised interface and destroys the previous one. Assumption is that destructor should clean up the driver state and constructor should bring it in sane condition. That allows us to write all test cases separately, everyone within their own file for easy maintain.

Status

IN DEVELOPMENT

Verified in ESP8266, does not pass without fixing the driver.

Missing the example mbed_app.json and some description about the test environment.

sarahmarshy · 2017-09-21T14:43:25Z

TESTS/network/wifi/get_interface.cpp

+#include "mbed.h"
+#include "ESP8266Interface.h"
+
+WiFiInterface *get_interface()


Do you think you could use the JSON files introduced in #4795 for this functionality?

The idea is that you have a JSON file describing how to instantiate the interface. That file is mapped to from some keywords. And each target has keywords it supports.

Here is an example of how the test configuration files are used in a test.

If the tests are not target specific or the interface is outside of mbed OS, then you can create a new config file outside of mbed OS and use it in the test command like. mbed test -m [mcu] -t [toolchain] --test-config path/outside/mbed

I could, these two approaches are doing the same effect.

I can rebase this PR and take your changes once you are in master.

Actually, now after review I noticed that the --test-config overrides the --app-config so I cannot use it.
We need app-config to deliver parameters like Wifi passwords, etc, for test cases.

0xc0170 · 2017-09-29T14:02:25Z

@SeppoTakalo Who do you think should review these besides @sarahmarshy ?

@kjbracey-arm @KariHaapalehto ?

KariHaapalehto

help text for "wifi-secure-ssid" and "wifi-unsecure-ssid" are same, should they specify the secure/unsecure?

Lot of compile warnings:
06:31:14 Compile [ 5.6%]: get_interface.cpp
06:31:14 [Warning] OdinWiFiInterface.h@54,0: #1300-D: ~OdinWiFiInterface inherits implicit virtual
06:31:14 Compile [ 11.1%]: main.cpp
06:31:14 [Warning] wifi_tests.h@26,0: #1-D: last line of file ends without a newline
06:31:14 Compile [ 16.7%]: wifi-constructor.cpp
06:31:14 [Warning] wifi_tests.h@26,0: #1-D: last line of file ends without a newline
06:31:14 Compile [ 22.2%]: wifi_connect.cpp
06:31:14 [Warning] wifi_tests.h@26,0: #1-D: last line of file ends without a newline
06:31:14 Compile [ 27.8%]: wifi_connect_disconnect_repeat.cpp
06:31:14 [Warning] wifi_tests.h@26,0: #1-D: last line of file ends without a newline
06:31:14 Compile [ 33.3%]: wifi_connect_nocredentials.cpp
06:31:14 [Warning] wifi_tests.h@26,0: #1-D: last line of file ends without a newline
06:31:14 Compile [ 38.9%]: wifi_connect_params_channel.cpp
06:31:14 [Warning] wifi_tests.h@26,0: #1-D: last line of file ends without a newline
06:31:14 Compile [ 44.4%]: wifi_connect_params_channel_fail.cpp
06:31:14 [Warning] wifi_tests.h@26,0: #1-D: last line of file ends without a newline
06:31:14 Compile [ 50.0%]: wifi_connect_params_null.cpp
06:31:14 [Warning] wifi_tests.h@26,0: #1-D: last line of file ends without a newline
06:31:14 Compile [ 55.6%]: wifi_connect_params_valid_secure.cpp
06:31:14 [Warning] wifi_tests.h@26,0: #1-D: last line of file ends without a newline
06:31:14 Compile [ 61.1%]: wifi_connect_params_valid_unsecure.cpp
06:31:14 [Warning] wifi_tests.h@26,0: #1-D: last line of file ends without a newline
06:31:14 Compile [ 66.7%]: wifi_connect_secure.cpp
06:31:14 [Warning] wifi_tests.h@26,0: #1-D: last line of file ends without a newline
06:31:14 Compile [ 72.2%]: wifi_connect_secure_fail.cpp
06:31:14 [Warning] wifi_tests.h@26,0: #1-D: last line of file ends without a newline
06:31:14 Compile [ 77.8%]: wifi_get_rssi.cpp
06:31:14 [Warning] wifi_tests.h@26,0: #1-D: last line of file ends without a newline
06:31:14 Compile [ 83.3%]: wifi_scan.cpp
06:31:14 [Warning] wifi_tests.h@26,0: #1-D: last line of file ends without a newline
06:31:14 Compile [ 88.9%]: wifi_scan_null.cpp
06:31:14 [Warning] wifi_tests.h@26,0: #1-D: last line of file ends without a newline
06:31:14 Compile [ 94.4%]: wifi_set_channel.cpp
06:31:14 [Warning] wifi_tests.h@26,0: #1-D: last line of file ends without a newline
06:31:14 Compile [100.0%]: wifi_set_credential.cpp
06:31:14 [Warning] wifi_tests.h@26,0: #1-D: last line of file ends without a newline

KariHaapalehto

If the proper app_config file isn't defined, the compiling of the wifi test will fail. Build shouldn't fail, it should be skipped.

SeppoTakalo · 2017-10-03T11:39:10Z

@KariHaapalehto Fixed now. Looks like its building OK.

KariHaapalehto

Looks ok to me

SeppoTakalo · 2017-10-03T13:10:40Z

@0xc0170 These are testcases defined in our internal test plan, reviewer should be aware of those.
Therefore KariHaapalehto was the correct person to review.

0xc0170 · 2017-10-04T08:04:45Z

TESTS/network/wifi/wifi_tests.h

+WiFiInterface *get_interface(void);
+
+/* Test cases */
+void wifi_constructor(void);


To be aware, we are enabling doxygen generation for tests (#5238). It would be beneficial to have these test cases documented here

0xc0170 · 2017-10-04T08:05:07Z

TESTS/network/wifi/wifi_tests.h

@@ -0,0 +1,26 @@
+#ifndef WIFI_TESTS_H


I believe every file in the codebase should have license header on the top, even test files.

0xc0170 · 2017-10-04T08:05:56Z

TESTS/network/wifi/wifi_tests.h

+void wifi_scan_null(void);
+void wifi_scan(void);
+
+#endif //WIFI_TESTS_H


does not this produce a warning (no new line at the end of the file) in ARMCC ? It should have or github is not showing it correctly here?

Implement 100% function coverage for WifiInterface as specified in "Wifi test plan"

Wifi connections and scanning takes long time. 2 minutes might not be enough.

Cannot include header file witin a function (without severe side effects)

SeppoTakalo · 2017-10-04T09:33:36Z

Updated based on @0xc0170 feedback.
Rebased on top of master.

0xc0170 · 2017-10-04T14:15:14Z

What about the license headers in these new files?

SeppoTakalo · 2017-10-04T14:40:23Z

Oops.. Good catch.

I updated only Doxygen tags and newline error.

0xc0170 · 2017-10-05T08:40:30Z

restarted travis (it failed due to time limit, not certain why so restarted), who else should review @SeppoTakalo ? Ready for CI?

SeppoTakalo · 2017-10-05T08:42:07Z

I think we are good to go if Kari have reviewed.
These are testcases, implemented according to our plan, so do not need a large review round.

0xc0170 · 2017-10-07T14:30:23Z

/morph test-nightly

mbed-bot · 2017-10-08T01:40:29Z

Result: SUCCESS

Your command has finished executing! Here's what you wrote!

/morph test-nightly

Output

mbed Build Number: 1542

All builds and test passed!

sarahmarshy · 2017-10-09T16:22:19Z

@SeppoTakalo, I was out of town, so I couldn't review. Does it make sense to make a PR to port these tests to use this style - #4795? We now have two ways of accomplishing the same thing. CC @theotherjimmy

SeppoTakalo requested review from bulislaw, adbridge, sarahmarshy and KariHaapalehto September 21, 2017 14:12

sarahmarshy reviewed Sep 21, 2017

View reviewed changes

0xc0170 added the needs: review label Sep 22, 2017

SeppoTakalo force-pushed the wifi_tests branch from 4fd3a56 to 73f2c68 Compare September 29, 2017 09:56

KariHaapalehto reviewed Oct 2, 2017

View reviewed changes

KariHaapalehto suggested changes Oct 2, 2017

View reviewed changes

KariHaapalehto approved these changes Oct 3, 2017

View reviewed changes

SeppoTakalo removed request for bulislaw and adbridge October 3, 2017 13:09

SeppoTakalo removed the needs: review label Oct 3, 2017

0xc0170 reviewed Oct 4, 2017

View reviewed changes

0xc0170 added the needs: CI label Oct 4, 2017

Seppo Takalo added 7 commits October 4, 2017 12:01

Implement functional Wifi tests

3d87507

Implement 100% function coverage for WifiInterface as specified in "Wifi test plan"

Set timeout to 4 minutes.

3ac788f

Wifi connections and scanning takes long time. 2 minutes might not be enough.

Update template mbed_app.json

0498e8d

Fix includes.

fde08a1

Cannot include header file witin a function (without severe side effects)

Update help text

5b40280

Fix builds for targets without wifi or without mbed_app.json

72e7c32

Add Doxygen documentation for testcases.

80198f2

SeppoTakalo force-pushed the wifi_tests branch from 638dc4d to 80198f2 Compare October 4, 2017 09:33

Add license texts

429b068

0xc0170 approved these changes Oct 5, 2017

View reviewed changes

0xc0170 added ready for merge and removed needs: CI labels Oct 9, 2017

theotherjimmy merged commit 42532d7 into ARMmbed:master Oct 9, 2017

theotherjimmy added release-version: 5.6.3 and removed ready for merge labels Oct 9, 2017

Implement functional Wifi tests #5161

Implement functional Wifi tests #5161

Uh oh!

Conversation

SeppoTakalo commented Sep 21, 2017

Description

Status

Uh oh!

sarahmarshy Sep 21, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SeppoTakalo Sep 21, 2017

Choose a reason for hiding this comment

Uh oh!

SeppoTakalo Oct 3, 2017

Choose a reason for hiding this comment

Uh oh!

0xc0170 commented Sep 29, 2017

Uh oh!

KariHaapalehto left a comment

Choose a reason for hiding this comment

Uh oh!

KariHaapalehto left a comment

Choose a reason for hiding this comment

Uh oh!

SeppoTakalo commented Oct 3, 2017

Uh oh!

KariHaapalehto left a comment

Choose a reason for hiding this comment

Uh oh!

SeppoTakalo commented Oct 3, 2017

Uh oh!

0xc0170 Oct 4, 2017

Choose a reason for hiding this comment

Uh oh!

0xc0170 Oct 4, 2017

Choose a reason for hiding this comment

Uh oh!

0xc0170 Oct 4, 2017

Choose a reason for hiding this comment

Uh oh!

SeppoTakalo commented Oct 4, 2017

Uh oh!

0xc0170 commented Oct 4, 2017

Uh oh!

SeppoTakalo commented Oct 4, 2017

Uh oh!

0xc0170 commented Oct 5, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

SeppoTakalo commented Oct 5, 2017

Uh oh!

0xc0170 commented Oct 7, 2017

Uh oh!

mbed-bot commented Oct 8, 2017

Result: SUCCESS

Output

Uh oh!

sarahmarshy commented Oct 9, 2017

Uh oh!

Uh oh!

sarahmarshy Sep 21, 2017 •

edited

Loading

0xc0170 commented Oct 5, 2017 •

edited

Loading