PDMC test failed with ESP8266

[ccli8]

Description

mbed-os
27e1bc3 Merge pull request #9073 from vervaekejonathan/master

simple-mbed-cloud-client
0a3221a (HEAD -> master, origin/master, origin/HEAD) Merge pull request #60 from ARMmbed/override-callbacks

mbed-cloud-client
62de4b8 (HEAD, tag: 2.0.1.1, origin/master, origin/HEAD) Merge pull request #20 from ARMmbed/release-2.0.1.1

NuMaker-mbed-SD-driver
9557d57 (HEAD -> master, origin/master, origin/HEAD) Fix user buffer is not SD DMA-compatible

Related issue or PR

#8927

I am running PDMC tests on NUMAKER_PFM_M487. On M487+Ethernet, tests are passed. On M487+ESP8266, almost simple-mbed-cloud-client-tests-basic* tests are failed. Per my check, these tests are failed in download_test() function (simple-mbed-cloud-client/TESTS/COMMON/download_test.cpp), which tries to download a large text file through HTTP. On M487+ESP8266, there are buffer overrun or packet dropped in these failed tests. As I know, ESP8266 driver can be seen with 3 level buffers:

1. H/W UART FIFO
2. MBED_CONF_DRIVERS_UART_SERIAL_RXBUF_SIZE (enlared from 256 to 1024) in mbed-os/drivers/UARTSerial.h(cpp)
3. MBED_CONF_ESP8266_SOCKET_BUFSIZE (default to 8 KiB) in mbed-os/components/wifi/esp8266-driver

When M487+ESP8266 cannot consume received packets in time, these packets will be dropped when MBED_CONF_ESP8266_SOCKET_BUFSIZE limit is reached, or buffer overrun with MBED_CONF_DRIVERS_UART_SERIAL_RXBUF_SIZE. The best solution to it seems to be support for UART flow control (CTS/RTS). But for targets like M487 which cannot consume packets in time and no UART flow control ready, is there another solution?

M487+ARM+Ethernet

mbedgt: test case report:
+----------------------+------------------+----------------------------------------------------+--------------------------------------------+--------+--------+--------+--------------------+
| target               | platform_name    | test suite                                         | test case                                  | passed | failed | result | elapsed_time (sec) |
+----------------------+------------------+----------------------------------------------------+--------------------------------------------+--------+--------+--------+--------------------+
| NUMAKER_PFM_M487-ARM | NUMAKER_PFM_M487 | simple-mbed-cloud-client-tests-basic-fs-single     | NUSD+FAT 1 file, buff     1                | 1      | 0      | OK     | 5.72               |
| NUMAKER_PFM_M487-ARM | NUMAKER_PFM_M487 | simple-mbed-cloud-client-tests-basic-fs-single     | NUSD+FAT 1 file, buff     4                | 1      | 0      | OK     | 3.89               |
| NUMAKER_PFM_M487-ARM | NUMAKER_PFM_M487 | simple-mbed-cloud-client-tests-basic-fs-single     | NUSD+FAT 1 file, buff    16                | 1      | 0      | OK     | 3.42               |
| NUMAKER_PFM_M487-ARM | NUMAKER_PFM_M487 | simple-mbed-cloud-client-tests-basic-fs-single     | NUSD+FAT 1 file, buff    64                | 1      | 0      | OK     | 3.33               |
| NUMAKER_PFM_M487-ARM | NUMAKER_PFM_M487 | simple-mbed-cloud-client-tests-basic-fs-single     | NUSD+FAT 1 file, buff   256                | 1      | 0      | OK     | 3.26               |
| NUMAKER_PFM_M487-ARM | NUMAKER_PFM_M487 | simple-mbed-cloud-client-tests-basic-fs-single     | NUSD+FAT 1 file, buff  1024                | 1      | 0      | OK     | 2.89               |
| NUMAKER_PFM_M487-ARM | NUMAKER_PFM_M487 | simple-mbed-cloud-client-tests-basic-fs-single     | NUSD+FAT 1 file, buff  4096                | 1      | 0      | OK     | 1.94               |
| NUMAKER_PFM_M487-ARM | NUMAKER_PFM_M487 | simple-mbed-cloud-client-tests-basic-fs-single     | NUSD+FAT 1 file, buff 16384                | 1      | 0      | OK     | 1.47               |
| NUMAKER_PFM_M487-ARM | NUMAKER_PFM_M487 | simple-mbed-cloud-client-tests-basic-fs-single     | NUSD+FAT format                            | 1      | 0      | OK     | 0.03               |
| NUMAKER_PFM_M487-ARM | NUMAKER_PFM_M487 | simple-mbed-cloud-client-tests-basic-fs-threaded   | NUSD+FAT 2 files, buff 4b/256b             | 1      | 0      | OK     | 7.18               |
| NUMAKER_PFM_M487-ARM | NUMAKER_PFM_M487 | simple-mbed-cloud-client-tests-basic-fs-threaded   | NUSD+FAT 3 files, buff 256b/1kb/4kb        | 1      | 0      | OK     | 8.33               |
| NUMAKER_PFM_M487-ARM | NUMAKER_PFM_M487 | simple-mbed-cloud-client-tests-basic-fs-threaded   | NUSD+FAT format                            | 1      | 0      | OK     | 0.02               |
| NUMAKER_PFM_M487-ARM | NUMAKER_PFM_M487 | simple-mbed-cloud-client-tests-basic-net-single    | Ethernet   128 buffer                      | 1      | 0      | OK     | 9.59               |
| NUMAKER_PFM_M487-ARM | NUMAKER_PFM_M487 | simple-mbed-cloud-client-tests-basic-net-single    | Ethernet   256 buffer                      | 1      | 0      | OK     | 6.23               |
| NUMAKER_PFM_M487-ARM | NUMAKER_PFM_M487 | simple-mbed-cloud-client-tests-basic-net-single    | Ethernet  1024 buffer                      | 1      | 0      | OK     | 6.19               |
| NUMAKER_PFM_M487-ARM | NUMAKER_PFM_M487 | simple-mbed-cloud-client-tests-basic-net-single    | Ethernet  4096 buffer                      | 1      | 0      | OK     | 6.42               |
| NUMAKER_PFM_M487-ARM | NUMAKER_PFM_M487 | simple-mbed-cloud-client-tests-basic-net-single    | Ethernet network setup                     | 1      | 0      | OK     | 5.52               |
| NUMAKER_PFM_M487-ARM | NUMAKER_PFM_M487 | simple-mbed-cloud-client-tests-basic-net-threaded  | Ethernet 1 thread                          | 1      | 0      | OK     | 7.05               |
| NUMAKER_PFM_M487-ARM | NUMAKER_PFM_M487 | simple-mbed-cloud-client-tests-basic-net-threaded  | Ethernet 2 threads                         | 1      | 0      | OK     | 10.95              |
| NUMAKER_PFM_M487-ARM | NUMAKER_PFM_M487 | simple-mbed-cloud-client-tests-basic-net-threaded  | Ethernet network setup                     | 1      | 0      | OK     | 4.52               |
| NUMAKER_PFM_M487-ARM | NUMAKER_PFM_M487 | simple-mbed-cloud-client-tests-basic-stress-net-fs | Ethernet network setup                     | 1      | 0      | OK     | 4.5                |
| NUMAKER_PFM_M487-ARM | NUMAKER_PFM_M487 | simple-mbed-cloud-client-tests-basic-stress-net-fs | NUSD+FAT format                            | 1      | 0      | OK     | 0.05               |
| NUMAKER_PFM_M487-ARM | NUMAKER_PFM_M487 | simple-mbed-cloud-client-tests-basic-stress-net-fs | NUSD+FAT+Ethernet 1 thread, dl, file seq.  | 1      | 0      | OK     | 6.15               |
| NUMAKER_PFM_M487-ARM | NUMAKER_PFM_M487 | simple-mbed-cloud-client-tests-basic-stress-net-fs | NUSD+FAT+Ethernet 2 threads, dl, 1kb       | 1      | 0      | OK     | 8.09               |
| NUMAKER_PFM_M487-ARM | NUMAKER_PFM_M487 | simple-mbed-cloud-client-tests-basic-stress-net-fs | NUSD+FAT+Ethernet 3 threads, dl, 256b, 1kb | 1      | 0      | OK     | 10.19              |
| NUMAKER_PFM_M487-ARM | NUMAKER_PFM_M487 | simple-mbed-cloud-client-tests-dev_mgmt-connect    | Connect to Ethernet                        | 2      | 0      | OK     | 4.55               |
| NUMAKER_PFM_M487-ARM | NUMAKER_PFM_M487 | simple-mbed-cloud-client-tests-dev_mgmt-connect    | Format FAT                                 | 1      | 0      | OK     | 0.08               |
| NUMAKER_PFM_M487-ARM | NUMAKER_PFM_M487 | simple-mbed-cloud-client-tests-dev_mgmt-connect    | Initialize NUSD+FAT                        | 2      | 0      | OK     | 0.03               |
| NUMAKER_PFM_M487-ARM | NUMAKER_PFM_M487 | simple-mbed-cloud-client-tests-dev_mgmt-connect    | Initialize Simple PDMC                     | 2      | 0      | OK     | 1.48               |
| NUMAKER_PFM_M487-ARM | NUMAKER_PFM_M487 | simple-mbed-cloud-client-tests-dev_mgmt-connect    | Pelion Bootstrap & Reg.                    | 1      | 0      | OK     | 9.18               |
| NUMAKER_PFM_M487-ARM | NUMAKER_PFM_M487 | simple-mbed-cloud-client-tests-dev_mgmt-connect    | Pelion Directory                           | 1      | 0      | OK     | 1.83               |
| NUMAKER_PFM_M487-ARM | NUMAKER_PFM_M487 | simple-mbed-cloud-client-tests-dev_mgmt-connect    | Pelion Re-register                         | 1      | 0      | OK     | 3.16               |
| NUMAKER_PFM_M487-ARM | NUMAKER_PFM_M487 | simple-mbed-cloud-client-tests-dev_mgmt-connect    | Post-reset Identity                        | 1      | 0      | OK     | 0.11               |
| NUMAKER_PFM_M487-ARM | NUMAKER_PFM_M487 | simple-mbed-cloud-client-tests-dev_mgmt-connect    | Resource LwM2M GET                         | 1      | 0      | OK     | 1.65               |
| NUMAKER_PFM_M487-ARM | NUMAKER_PFM_M487 | simple-mbed-cloud-client-tests-dev_mgmt-connect    | Resource LwM2M POST                        | 1      | 0      | OK     | 0.97               |
| NUMAKER_PFM_M487-ARM | NUMAKER_PFM_M487 | simple-mbed-cloud-client-tests-dev_mgmt-connect    | Resource LwM2M PUT                         | 1      | 0      | OK     | 1.34               |
| NUMAKER_PFM_M487-ARM | NUMAKER_PFM_M487 | simple-mbed-cloud-client-tests-dev_mgmt-connect    | Resource LwM2M SET                         | 1      | 0      | OK     | 0.73               |
| NUMAKER_PFM_M487-ARM | NUMAKER_PFM_M487 | simple-mbed-cloud-client-tests-dev_mgmt-update     | Connect to Ethernet                        | 2      | 0      | OK     | 4.54               |
| NUMAKER_PFM_M487-ARM | NUMAKER_PFM_M487 | simple-mbed-cloud-client-tests-dev_mgmt-update     | Format FAT                                 | 1      | 0      | OK     | 0.08               |
| NUMAKER_PFM_M487-ARM | NUMAKER_PFM_M487 | simple-mbed-cloud-client-tests-dev_mgmt-update     | Initialize NUSD+FAT                        | 2      | 0      | OK     | 0.03               |
| NUMAKER_PFM_M487-ARM | NUMAKER_PFM_M487 | simple-mbed-cloud-client-tests-dev_mgmt-update     | Initialize Simple PDMC                     | 2      | 0      | OK     | 1.47               |
| NUMAKER_PFM_M487-ARM | NUMAKER_PFM_M487 | simple-mbed-cloud-client-tests-dev_mgmt-update     | Pelion Bootstrap & Reg.                    | 1      | 0      | OK     | 8.84               |
| NUMAKER_PFM_M487-ARM | NUMAKER_PFM_M487 | simple-mbed-cloud-client-tests-dev_mgmt-update     | Pelion Directory                           | 1      | 0      | OK     | 1.36               |
| NUMAKER_PFM_M487-ARM | NUMAKER_PFM_M487 | simple-mbed-cloud-client-tests-dev_mgmt-update     | Pelion Firmware Download                   | 1      | 0      | OK     | 140.81             |
| NUMAKER_PFM_M487-ARM | NUMAKER_PFM_M487 | simple-mbed-cloud-client-tests-dev_mgmt-update     | Pelion Firmware Prepare                    | 1      | 0      | OK     | 0.58               |
| NUMAKER_PFM_M487-ARM | NUMAKER_PFM_M487 | simple-mbed-cloud-client-tests-dev_mgmt-update     | Pelion Firmware Update                     | 1      | 0      | OK     | 20.3               |
| NUMAKER_PFM_M487-ARM | NUMAKER_PFM_M487 | simple-mbed-cloud-client-tests-dev_mgmt-update     | Pelion Re-register                         | 1      | 0      | OK     | 2.99               |
| NUMAKER_PFM_M487-ARM | NUMAKER_PFM_M487 | simple-mbed-cloud-client-tests-dev_mgmt-update     | Post-update Erase                          | 1      | 0      | OK     | 0.01               |
| NUMAKER_PFM_M487-ARM | NUMAKER_PFM_M487 | simple-mbed-cloud-client-tests-dev_mgmt-update     | Post-update Identity                       | 1      | 0      | OK     | 0.11               |
+----------------------+------------------+----------------------------------------------------+--------------------------------------------+--------+--------+--------+--------------------+
mbedgt: test case results: 49 OK

M487+ARM+ESP8266

mbedgt: test case report:
+----------------------+------------------+----------------------------------------------------+--------------------------------------------+--------+--------+---------+--------------------+
| target               | platform_name    | test suite                                         | test case                                  | passed | failed | result  | elapsed_time (sec) |
+----------------------+------------------+----------------------------------------------------+--------------------------------------------+--------+--------+---------+--------------------+
| NUMAKER_PFM_M487-ARM | NUMAKER_PFM_M487 | simple-mbed-cloud-client-tests-basic-fs-single     | NUSD+FAT 1 file, buff     1                | 1      | 0      | OK      | 5.72               |
| NUMAKER_PFM_M487-ARM | NUMAKER_PFM_M487 | simple-mbed-cloud-client-tests-basic-fs-single     | NUSD+FAT 1 file, buff     4                | 1      | 0      | OK      | 3.84               |
| NUMAKER_PFM_M487-ARM | NUMAKER_PFM_M487 | simple-mbed-cloud-client-tests-basic-fs-single     | NUSD+FAT 1 file, buff    16                | 1      | 0      | OK      | 3.48               |
| NUMAKER_PFM_M487-ARM | NUMAKER_PFM_M487 | simple-mbed-cloud-client-tests-basic-fs-single     | NUSD+FAT 1 file, buff    64                | 1      | 0      | OK      | 3.27               |
| NUMAKER_PFM_M487-ARM | NUMAKER_PFM_M487 | simple-mbed-cloud-client-tests-basic-fs-single     | NUSD+FAT 1 file, buff   256                | 1      | 0      | OK      | 3.25               |
| NUMAKER_PFM_M487-ARM | NUMAKER_PFM_M487 | simple-mbed-cloud-client-tests-basic-fs-single     | NUSD+FAT 1 file, buff  1024                | 1      | 0      | OK      | 2.9                |
| NUMAKER_PFM_M487-ARM | NUMAKER_PFM_M487 | simple-mbed-cloud-client-tests-basic-fs-single     | NUSD+FAT 1 file, buff  4096                | 1      | 0      | OK      | 1.94               |
| NUMAKER_PFM_M487-ARM | NUMAKER_PFM_M487 | simple-mbed-cloud-client-tests-basic-fs-single     | NUSD+FAT 1 file, buff 16384                | 1      | 0      | OK      | 1.48               |
| NUMAKER_PFM_M487-ARM | NUMAKER_PFM_M487 | simple-mbed-cloud-client-tests-basic-fs-single     | NUSD+FAT format                            | 1      | 0      | OK      | 0.06               |
| NUMAKER_PFM_M487-ARM | NUMAKER_PFM_M487 | simple-mbed-cloud-client-tests-basic-fs-threaded   | NUSD+FAT 2 files, buff 4b/256b             | 1      | 0      | OK      | 7.2                |
| NUMAKER_PFM_M487-ARM | NUMAKER_PFM_M487 | simple-mbed-cloud-client-tests-basic-fs-threaded   | NUSD+FAT 3 files, buff 256b/1kb/4kb        | 1      | 0      | OK      | 8.27               |
| NUMAKER_PFM_M487-ARM | NUMAKER_PFM_M487 | simple-mbed-cloud-client-tests-basic-fs-threaded   | NUSD+FAT format                            | 1      | 0      | OK      | 0.01               |
| NUMAKER_PFM_M487-ARM | NUMAKER_PFM_M487 | simple-mbed-cloud-client-tests-basic-net-single    | Ethernet   128 buffer                      | 0      | 1      | FAIL    | 3.17               |
| NUMAKER_PFM_M487-ARM | NUMAKER_PFM_M487 | simple-mbed-cloud-client-tests-basic-net-single    | Ethernet   256 buffer                      | 0      | 0      | SKIPPED | 0.0                |
| NUMAKER_PFM_M487-ARM | NUMAKER_PFM_M487 | simple-mbed-cloud-client-tests-basic-net-single    | Ethernet  1024 buffer                      | 0      | 0      | SKIPPED | 0.0                |
| NUMAKER_PFM_M487-ARM | NUMAKER_PFM_M487 | simple-mbed-cloud-client-tests-basic-net-single    | Ethernet  4096 buffer                      | 0      | 0      | SKIPPED | 0.0                |
| NUMAKER_PFM_M487-ARM | NUMAKER_PFM_M487 | simple-mbed-cloud-client-tests-basic-net-single    | Ethernet network setup                     | 1      | 0      | OK      | 7.33               |
| NUMAKER_PFM_M487-ARM | NUMAKER_PFM_M487 | simple-mbed-cloud-client-tests-basic-net-threaded  | Ethernet 1 thread                          | 0      | 1      | FAIL    | 2.05               |
| NUMAKER_PFM_M487-ARM | NUMAKER_PFM_M487 | simple-mbed-cloud-client-tests-basic-net-threaded  | Ethernet 2 threads                         | 0      | 0      | SKIPPED | 0.0                |
| NUMAKER_PFM_M487-ARM | NUMAKER_PFM_M487 | simple-mbed-cloud-client-tests-basic-net-threaded  | Ethernet network setup                     | 1      | 0      | OK      | 7.32               |
| NUMAKER_PFM_M487-ARM | NUMAKER_PFM_M487 | simple-mbed-cloud-client-tests-basic-stress-net-fs | Ethernet network setup                     | 1      | 0      | OK      | 7.33               |
| NUMAKER_PFM_M487-ARM | NUMAKER_PFM_M487 | simple-mbed-cloud-client-tests-basic-stress-net-fs | NUSD+FAT format                            | 1      | 0      | OK      | 0.03               |
| NUMAKER_PFM_M487-ARM | NUMAKER_PFM_M487 | simple-mbed-cloud-client-tests-basic-stress-net-fs | NUSD+FAT+Ethernet 1 thread, dl, file seq.  | 0      | 1      | FAIL    | 2.03               |
| NUMAKER_PFM_M487-ARM | NUMAKER_PFM_M487 | simple-mbed-cloud-client-tests-basic-stress-net-fs | NUSD+FAT+Ethernet 2 threads, dl, 1kb       | 0      | 0      | SKIPPED | 0.0                |
| NUMAKER_PFM_M487-ARM | NUMAKER_PFM_M487 | simple-mbed-cloud-client-tests-basic-stress-net-fs | NUSD+FAT+Ethernet 3 threads, dl, 256b, 1kb | 0      | 0      | SKIPPED | 0.0                |
+----------------------+------------------+----------------------------------------------------+--------------------------------------------+--------+--------+---------+--------------------+
mbedgt: test case results: 3 FAIL / 6 SKIPPED / 16 OK

Issue request type

[ ] Question
[ ] Enhancement
[X] Bug

[ciarmcom]

Internal Jira reference: https://jira.arm.com/browse/MBOCUSTRIA-313

[screamerbg]

FYI @samchuarm

[screamerbg]

@VeijoPesonen

[VeijoPesonen]

@ccli8 my recommendation would be that you should update ESP8266's firmware to version 1.7.0. With that firmware the driver doesn't allocate memory for TCP packets but uses application provided buffers directly instead. That would be the solution in case of TCP. For further details please read ARMmbed/mbed-os/master/components/wifi/esp8266-driver/README.md

[ccli8]

@VeijoPesonen Thanks for your information.
1.7.0 AT requires 16Mb or above flash, but M487 IoT on-board ESP8266 module just has 8Mb flash. Currently, we cannot upgrade the on-board ESP8266 module to 1.7.0 AT. Besides, we fly-wire another 32Mb ESP8266 module which upgrades to 1.7.0 AT for PDMC tests. It is passed. Currently, we are struggling for the upgrade...
@samchuarm

[0xc0170]

1.7.0 AT requires 16Mb or above flash, but M487 IoT on-board ESP8266 module just has 8Mb flash.

cc @SeppoTakalo

[SeppoTakalo]

Please run full set on Mbed OS socket tests for this platform as instructed in Network Socket test plan.

If it passes the Mbed OS tests, then you should continue testing with Client.

[SeppoTakalo]

From: https://www.espressif.com/en/products/hardware/esp8266ex/resources

ESP8266 AT Bin V1.7.0
Based on ESP8266_NonOS_SDK V3.0.0.

Limited by the size of the AT bin file,  only `1024+1024 flash map` is supported by default. 

We have test cases that proof the 1.6.2 and older to be not working properly, and therefore we cannot support those.
For any commercial platform, the minimum requirement would be that lates ESP AT firmware is installed and CTS, RTS and Reset for the ESP8266 as well.

And as we cannot support this, we need to close this as "Won't fix"