two tests fail in qemu_cortex_m3 with new SDK

[nashif]

qemu_cortex_m3            tests/kernel/threads/scheduling/schedule_api/test  FAILED: qemu_crash
	see: sanity-out/qemu_cortex_m3/tests/kernel/threads/scheduling/schedule_api/test/run.log

total complete:    1/   2  50%  failed:    1

qemu_cortex_m3            tests/kernel/timer/timer_monotonic/test            FAILED: failed
	see: sanity-out/qemu_cortex_m3/tests/kernel/timer/timer_monotonic/test/qemu.log

[nashif]

@youvedeep can you please take a look?

[nashif]

This is with SDK 0.9.2-rc4

[youvedeep]

i looked at this issue today here are findings

I just concentrated only on "tests/kernel/threads/scheduling/schedule_api", here are outcomes of debug

1. On Cortex m3 QEMU:-
   A. Test Getting Failed at run time on Cortex_m3 QEMU from SDK 0.9.2-rc1 onwards. Failure signature is same between rc1/rc2/rc4.
   B. Test is getting passed on SDK 0.9.1.
   So looks like there are some patches between SDK 0.9.1 and SDK 0.9.2-rc1 which are causing this issue.

2, Test is not getting failed with SDK 0.9.2-rc1/2/4 on Arduino_Due (using Cortex m3 SOC).

here is Failure Signature

 BOOTING ZEPHYR OS v1.9.0 - BUILD: Sep 21 2017 09:53:00 
Running test suite test_threads_scheduling
===================================================================
starting test - test_priority_cooperative
PASS - test_priority_cooperative.
===================================================================
starting test - test_priority_preemptible
PASS - test_priority_preemptible.
===================================================================
starting test - test_yield_cooperative
PASS - test_yield_cooperative.
===================================================================
starting test - test_sleep_cooperative
PASS - test_sleep_cooperative.
===================================================================
starting test - test_sleep_wakeup_preemptible
PASS - test_sleep_wakeup_preemptible.
===================================================================
starting test - test_time_slicing_preemptible

    Assertion failed at /home/ysingh/IOT/Zephyr/Source_Code/github/Zephyr_July17/zephyr-1/tests/kernel/threads/scheduling/schedule_api/src/test_sched_timeslice_and_lock.c:155: test_time_slicing_preemptible:  tdata[1].executed == 1 is false
FAIL - test_time_slicing_preemptible.
===================================================================
starting test - test_time_slicing_disable_preemptible
PASS - test_time_slicing_disable_preemptible.
===================================================================
starting test - test_lock_preemptible
***** USAGE FAULT *****
  Executing thread ID (thread): 0x00000001
  Faulting instruction address:  0x0
  Illegal use of the EPSR
Fatal fault in thread 0x00000001! Aborting.
qemu-system-arm: Trying to execute code outside RAM or ROM at 0x3500001e

This usually means one of the following happened:

(1) You told QEMU to execute a kernel for the wrong machine type, and it crashed on startup (eg trying to run a raspberry pi kernel on a versatilepb QEMU machine)
(2) You didn't give QEMU a kernel or BIOS filename at all, and QEMU executed a ROM full of no-op instructions until it fell off the end
(3) Your guest kernel has a bug and crashed by jumping off into nowhere

This is almost always one of the first two, so check your command line and that you are using the right type of kernel for this machine.
If you think option (3) is likely then you can try debugging your guest with the -d debug options; in particular -d guest_errors will cause the log to include a dump of the guest register state at this point.

Execution cannot continue; stopping here.

/home/ysingh/IOT/Zephyr/Source_Code/github/Zephyr_July17/zephyr-1/scripts/Makefile.qemu:25: recipe for target 'run' failed
make[2]: *** [run] Error 1
make[2]: Leaving directory '/home/ysingh/IOT/Zephyr/Source_Code/github/Zephyr_July17/zephyr-1/tests/kernel/threads/scheduling/schedule_api/outdir/qemu_cortex_m3'

[nashif]

SDK-0.9.2-rc4 has a newer Qemu

[youvedeep]

Further debug on this issue

Summery : -

• Issue reproducible even on QEMU 2.9 too.
• Looks like patch (8d04fb55dec381bc5105cb47f29d918e579e8cbd) is creating issue .

Details :-
SDK 0.9.1 uses QEMU 2.6
SDK 0.9.2 uses QEMU 2.10

Did following Experiments :-

1. SDK 0.9.1 + QEMU 2.6 ==> Test run successful.
2. SDK 0.9.2 + QEMU 2.10 ==> Test run Fail.
3. SDK 0.9.1 + QEMU 2.10 ==> Test run Fail.
4. SDK 0.9.2 + QEMU 2.6 ==> Test run Successful.
   So above Experiments indicates that, something crazy is going on with QEMU 2.10.

So to root cause the patch , there are 3 QEMU releases in between (2.7 /8/9), to point out the the QEMU from which the issue started, we did few experiments with different QEMUs.
QEMU 2.7 ==> Issue not reproducible
QEMU 2.8 ==> Issue not reproducible
QEMU 2.9 ==> Issue reproducible
So looks like issue is occurring in between 2.8 and 2.9.

While doing the bisect we reached at patch with commit ID - 8d04fb55dec381bc5105cb47f29d918e579e8cbd .

However I am not able to revert this patch cleanly on 2.10 due to some dependency issue. I will continue debugging his further tomorrow to root cause issue.

[youvedeep]

As issue is present on QEMU 2.9 so we thought of fixing the issue on QEMU 2.9, So i focused on QEMU2.9 to fix this issue.

Summery :-

1. As patch 8d04fb55dec381bc5105cb47f29d918e579e8cbd touches multiple files but it is only changes with respect to files - cpu-exec.c, cpus.c, cputlb.c seems to be creating this issue.
2. On QEMU 2.9 even if i checkout these changes (in files cpu-exec.c, cpus.c, cputlb.c wrt 8d04fb55dec381bc5105cb47f29d918e579e8cbd ), I still get hang. So looks like there some other changes too which got merged later are also creating the issue.

Details :-
So here are experiment results :
Experiment 1: Checkout @ 8d04fb55dec381bc5105cb47f29d918e579e8cbd
QEMU cortex m3 run fails
Experiment 2: revert 8d04fb55dec381bc5105cb47f29d918e579e8cbd
QEMU cortex m3 run Pass
Experiment 3: Checkout @ 8d04fb55dec381bc5105cb47f29d918e579e8cbd + revert changes into CPU specific file cpu-exec.c, cpus.c, cputlb.c.
QEMU cortex m3 run Pass
Experiment 4: Checkout is QEMU 2.9 tip and revert checkes as found in Experiment 3 ("revert changes into CPU specific file cpu-exec.c, cpus.c, cputlb.c. wrt to patch
QEMU cortex m3 run fails

So looks like some other patches too merged after 8d04fb55dec381bc5105cb47f29d918e579e8cbd which too are creating the issue.
Also I am not expert on QEMU, so it is taking more time to debug if some expert will look into the issue this issue can be fixed faster.
Else i will continue debug on this issue.

[nashif]

not relevant with current SDK