sanitycheck errors when device-testing frdm_k64f

[andrewboie]

$ sanitycheck -p frdm_k64f --device-testing --device-serial=/dev/ttyACM0 
JOBS: 80
Building testcase defconfigs...
228 tests selected, 90246 tests discarded due to filters
total complete:    0/ 225   0%  failed:    0

frdm_k64f                 samples/basic/blinky/test                          FAILED: timeout
	see: sanity-out/frdm_k64f/samples/basic/blinky/test/handler.log

total complete:    1/ 225   0%  failed:    1Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/lib64/python3.7/threading.py", line 917, in _bootstrap_inner
    self.run()
  File "/usr/lib64/python3.7/threading.py", line 865, in run
    self._target(*self._args, **self._kwargs)
  File "/projects/zephyr3/zephyr/scripts/sanitycheck", line 624, in monitor_serial
    serial_line = ser.readline()
  File "/home/apboie/.local/lib/python3.7/site-packages/serial/serialposix.py", line 478, in read
    raise portNotOpenError
serial.serialutil.SerialException: Attempting to use a port that is not open

total complete:    3/ 225   1%  failed:    1

frdm_k64f                 samples/basic/disco/test                           FAILED: timeout
	see: sanity-out/frdm_k64f/samples/basic/disco/test/handler.log

total complete:    4/ 225   1%  failed:    2Exception in thread Thread-4:
Traceback (most recent call last):
  File "/usr/lib64/python3.7/threading.py", line 917, in _bootstrap_inner
    self.run()
  File "/usr/lib64/python3.7/threading.py", line 865, in run
    self._target(*self._args, **self._kwargs)
  File "/projects/zephyr3/zephyr/scripts/sanitycheck", line 624, in monitor_serial
    serial_line = ser.readline()
  File "/home/apboie/.local/lib/python3.7/site-packages/serial/serialposix.py", line 478, in read
    raise portNotOpenError
  File "/usr/lib64/python3.7/threading.py", line 917, in _bootstrap_inner
    self.run()
  File "/usr/lib64/python3.7/threading.py", line 865, in run
    self._target(*self._args, **self._kwargs)
  File "/projects/zephyr3/zephyr/scripts/sanitycheck", line 624, in monitor_serial
    serial_line = ser.readline()
  File "/home/apboie/.local/lib/python3.7/site-packages/serial/serialposix.py", line 478, in read
    raise portNotOpenError
serial.serialutil.SerialException: Attempting to use a port that is not open

total complete:    6/ 225   2%  failed:    2

It appears that if a test times out, sanitycheck immediately has a SerialException

[galak]

@doanac is this something you might be able to look at

[doanac]

@galak - i'm not going to have time this week, so you'd probably be better off getting someone else to look.

[nashif]

It appears that if a test times out, sanitycheck immediately has a SerialException

they timeout because they have no output and there is nothing to be evaluated. I however did not get the exception above.

Need to tag those tests differently, they should not run with device-testing if we can't evaluate if test has passed or not.

[nashif]

ok, got the exception now. Looking into it.