sanitycheck --coverage on qemu_x86: stack overflows on qemu_x86 and mps2_an385

[cinlyooi-intel]

Describe the bug
We have too many handler_crash failure when running code coverage using qemu_x86

Impact
Currently blocking code coverage improvement work.

To Reproduce and console output

$ ./scripts/sanitycheck -v --coverage --coverage-platform qemu_x86 -p qemu_x86  --gcov-tool /opt/zephyr-sdk-0.9.5/sysroots/x86_64-pokysdk-linux/usr/bin/i586-zephyr-elf/i586-zephyr-elf-gcov -T tests/kernel/ -T tests/subsys/
JOBS: 8
Cleaning output directory /path/to/zephyr/sanity-out
Reading test case configuration files under /path/to/zephyr/tests/kernel...
Reading test case configuration files under /path/to/zephyr/tests/subsys...
Reading platform configuration files under /path/to/zephyr.git/boards...
Reading platform configuration files under /path/to/zephyr.git/scripts/sanity_chk/boards...
Building testcase defconfigs...
86 tests selected, 17302 tests discarded due to filters
 1/85 qemu_x86                  common/kernel.common.misra                         FAILED: handler_crash
	see: sanity-out/qemu_x86/common/kernel.common.misra/build.log
 2/85 qemu_x86                  common/kernel.common                               FAILED: handler_crash
	see: sanity-out/qemu_x86/common/kernel.common/build.log
 3/85 qemu_x86                  fatal/kernel.common.stack_protection               FAILED: handler_crash
	see: sanity-out/qemu_x86/fatal/kernel.common.stack_protection/build.log
 4/85 qemu_x86                  context/kernel.common                              FAILED: handler_crash
	see: sanity-out/qemu_x86/context/kernel.common/run.log
 5/85 qemu_x86                  fatal/kernel.common.stack_sentinel                 FAILED: handler_crash
	see: sanity-out/qemu_x86/fatal/kernel.common.stack_sentinel/build.log
 6/85 qemu_x86                  device/kernel.device.pm                            PASSED
 7/85 qemu_x86                  fifo/fifo_api/kernel.fifo                          FAILED: handler_crash
	see: sanity-out/qemu_x86/fifo/fifo_api/kernel.fifo/run.log
 8/85 qemu_x86                  device/kernel.device                               PASSED
 9/85 qemu_x86                  boot_page_table/kernel.memory_protection           PASSED
10/85 qemu_x86                  can/frame/can.frame                                PASSED
11/85 qemu_x86                  fifo/fifo_api/kernel.fifo.poll                     FAILED: handler_crash
	see: sanity-out/qemu_x86/fifo/fifo_api/kernel.fifo.poll/run.log
12/85 qemu_x86                  fifo/fifo_timeout/kernel.fifo.timeout              FAILED: handler_crash
	see: sanity-out/qemu_x86/fifo/fifo_timeout/kernel.fifo.timeout/build.log
13/85 qemu_x86                  fifo/fifo_timeout/kernel.fifo.timeout.poll         FAILED: handler_crash
	see: sanity-out/qemu_x86/fifo/fifo_timeout/kernel.fifo.timeout.poll/build.log
14/85 qemu_x86                  early_sleep/kernel.common                          FAILED: handler_crash
	see: sanity-out/qemu_x86/early_sleep/kernel.common/run.log
15/85 qemu_x86                  critical/kernel.common                             PASSED
16/85 qemu_x86                  fs/multi-fs/filesystem.fs_shell                    FAILED: unexpected eof
	see: sanity-out/qemu_x86/fs/multi-fs/filesystem.fs_shell/handler.log
17/85 qemu_x86                  fifo/fifo_usage/kernel.fifo.usage                  PASSED
18/85 qemu_x86                  fs/fat_fs_dual_drive/filesystem.fat                PASSED
19/85 qemu_x86                  fifo/fifo_usage/kernel.fifo.usage.poll             PASSED
20/85 qemu_x86                  interrupt/arch.interrupt                           FAILED: handler_crash
	see: sanity-out/qemu_x86/interrupt/arch.interrupt/build.log
21/85 qemu_x86                  fs/multi-fs/filesystem.multifs                     PASSED
22/85 qemu_x86                  fs/nffs_fs_api/basic/filesystem.nffs.basic         PASSED
23/85 qemu_x86                  lifo/lifo_api/kernel.lifo                          FAILED: handler_crash
	see: sanity-out/qemu_x86/lifo/lifo_api/kernel.lifo/run.log
24/85 qemu_x86                  lifo/lifo_usage/kernel.lifo.usage                  FAILED: handler_crash
	see: sanity-out/qemu_x86/lifo/lifo_usage/kernel.lifo.usage/run.log
25/85 qemu_x86                  fs/nffs_fs_api/cache/filesystem.nffs.cache         PASSED
26/85 qemu_x86                  fs/nffs_fs_api/large/filesystem.nffs.large         PASSED
27/85 qemu_x86                  jwt/libraries.encoding                             PASSED
28/85 qemu_x86                  logging/log_core/logging.log_core                  PASSED
29/85 qemu_x86                  fs/nffs_fs_api/performance/filesystem.nffs.performance PASSED
30/85 qemu_x86                  mbox/mbox_api/kernel.mailbox                       FAILED: handler_crash
	see: sanity-out/qemu_x86/mbox/mbox_api/kernel.mailbox/run.log
31/85 qemu_x86                  logging/log_list/logging.log_list                  PASSED
32/85 qemu_x86                  logging/log_msg/logging.log_msg                    PASSED
33/85 qemu_x86                  logging/log_output/logging.log_output              PASSED
34/85 qemu_x86                  mem_pool/mem_pool/kernel.memory_pool               FAILED: handler_crash
	see: sanity-out/qemu_x86/mem_pool/mem_pool/kernel.memory_pool/run.log
35/85 qemu_x86                  mem_pool/mem_pool_api/kernel.memory_pool           FAILED: handler_crash
	see: sanity-out/qemu_x86/mem_pool/mem_pool_api/kernel.memory_pool/run.log
36/85 qemu_x86                  mem_pool/mem_pool_concept/kernel.memory_pool       FAILED: handler_crash
	see: sanity-out/qemu_x86/mem_pool/mem_pool_concept/kernel.memory_pool/run.log
37/85 qemu_x86                  mbox/mbox_usage/kernel.mailbox                     PASSED
38/85 qemu_x86                  mem_protect/obj_validation/kernel.memory_protection.obj_validation FAILED: handler_crash
	see: sanity-out/qemu_x86/mem_protect/obj_validation/kernel.memory_protection.obj_validation/build.log
39/85 qemu_x86                  mem_protect/mem_protect/kernel.memory_protection   FAILED: handler_crash
	see: sanity-out/qemu_x86/mem_protect/mem_protect/kernel.memory_protection/build.log
40/85 qemu_x86                  mem_protect/syscalls/kernel.memory_protection.syscalls FAILED: handler_crash
	see: sanity-out/qemu_x86/mem_protect/syscalls/kernel.memory_protection.syscalls/build.log
41/85 qemu_x86                  mem_heap/mheap_api_concept/kernel.memory_heap      PASSED
42/85 qemu_x86                  mem_pool/mem_pool_threadsafe/kernel.memory_pool    FAILED: handler_crash
	see: sanity-out/qemu_x86/mem_pool/mem_pool_threadsafe/kernel.memory_pool/run.log
43/85 qemu_x86                  mem_protect/stackprot/kernel.memory_protection     FAILED: handler_crash
	see: sanity-out/qemu_x86/mem_protect/stackprot/kernel.memory_protection/run.log
44/85 qemu_x86                  mem_protect/userspace/kernel.memory_protection.userspace FAILED: handler_crash
	see: sanity-out/qemu_x86/mem_protect/userspace/kernel.memory_protection.userspace/build.log
45/85 qemu_x86                  mem_protect/x86_mmu_api/kernel.memory_protection.mmu FAILED: handler_crash
	see: sanity-out/qemu_x86/mem_protect/x86_mmu_api/kernel.memory_protection.mmu/build.log
46/85 qemu_x86                  mem_protect/stack_random/kernel.memory_protection.stack_random PASSED
47/85 qemu_x86                  mem_slab/mslab/kernel.memory_slabs                 FAILED: handler_crash
	see: sanity-out/qemu_x86/mem_slab/mslab/kernel.memory_slabs/run.log
48/85 qemu_x86                  mem_pool/sys_mem_pool/kernel.memory_pool           PASSED
49/85 qemu_x86                  mem_slab/mslab_concept/kernel.memory_slabs         FAILED: handler_crash
	see: sanity-out/qemu_x86/mem_slab/mslab_concept/kernel.memory_slabs/run.log
50/85 qemu_x86                  mem_slab/mslab_api/kernel.memory_slabs             FAILED: handler_crash
	see: sanity-out/qemu_x86/mem_slab/mslab_api/kernel.memory_slabs/run.log
51/85 qemu_x86                  msgq/msgq_api/kernel.message_queue                 FAILED: handler_crash
	see: sanity-out/qemu_x86/msgq/msgq_api/kernel.message_queue/run.log
52/85 qemu_x86                  mem_slab/mslab_threadsafe/kernel.memory_slabs      PASSED
53/85 qemu_x86                  mutex/mutex_api/kernel.mutex                       FAILED: handler_crash
	see: sanity-out/qemu_x86/mutex/mutex_api/kernel.mutex/run.log
54/85 qemu_x86                  obj_tracing/kernel.object_tracing                  FAILED: handler_crash
	see: sanity-out/qemu_x86/obj_tracing/kernel.object_tracing/run.log
55/85 qemu_x86                  mutex/mutex/kernel.mutex                           FAILED: handler_crash
	see: sanity-out/qemu_x86/mutex/mutex/kernel.mutex/run.log
56/85 qemu_x86                  pending/kernel.common.timing                       FAILED: handler_crash
	see: sanity-out/qemu_x86/pending/kernel.common.timing/run.log
57/85 qemu_x86                  mem_protect/protection/kernel.memory_protection.protection FAILED: timeout
	see: sanity-out/qemu_x86/mem_protect/protection/kernel.memory_protection.protection/handler.log
58/85 qemu_x86                  poll/kernel.poll                                   FAILED: handler_crash
	see: sanity-out/qemu_x86/poll/kernel.poll/run.log
59/85 qemu_x86                  pipe/pipe_api/kernel.pipe                          FAILED: handler_crash
	see: sanity-out/qemu_x86/pipe/pipe_api/kernel.pipe/run.log
60/85 qemu_x86                  queue/kernel.queue.poll                            FAILED: handler_crash
	see: sanity-out/qemu_x86/queue/kernel.queue.poll/run.log
61/85 qemu_x86                  sched/schedule_api/kernel.sched                    FAILED: handler_crash
	see: sanity-out/qemu_x86/sched/schedule_api/kernel.sched/build.log
62/85 qemu_x86                  profiling/profiling_api/kernel.profiling           PASSED
63/85 qemu_x86                  sched/deadline/kernel.sched.deadline               FAILED: handler_crash
	see: sanity-out/qemu_x86/sched/deadline/kernel.sched.deadline/run.log
64/85 qemu_x86                  queue/kernel.queue                                 FAILED: handler_crash
	see: sanity-out/qemu_x86/queue/kernel.queue/run.log
65/85 qemu_x86                  sleep/kernel.common.timing                         FAILED: handler_crash
	see: sanity-out/qemu_x86/sleep/kernel.common.timing/build.log
66/85 qemu_x86                  semaphore/sema_api/kernel.semaphore                FAILED: handler_crash
	see: sanity-out/qemu_x86/semaphore/sema_api/kernel.semaphore/run.log
67/85 qemu_x86                  semaphore/semaphore/kernel.semaphore               FAILED: handler_crash
	see: sanity-out/qemu_x86/semaphore/semaphore/kernel.semaphore/run.log
68/85 qemu_x86                  sched/preempt/kernel.sched.preempt                 PASSED
69/85 qemu_x86                  threads/thread_apis/kernel.threads                 FAILED: handler_crash
	see: sanity-out/qemu_x86/threads/thread_apis/kernel.threads/build.log
70/85 qemu_x86                  threads/dynamic_thread/kernel.threads.dynamic      FAILED: handler_crash
	see: sanity-out/qemu_x86/threads/dynamic_thread/kernel.threads.dynamic/run.log
71/85 qemu_x86                  stack/stack_usage/kernel.stack.usage               FAILED: handler_crash
	see: sanity-out/qemu_x86/stack/stack_usage/kernel.stack.usage/run.log
72/85 qemu_x86                  pipe/pipe/kernel.pipe                              FAILED: timeout
	see: sanity-out/qemu_x86/pipe/pipe/kernel.pipe/handler.log
73/85 qemu_x86                  threads/no-multithreading/kernel.threads.no-multithreading PASSED
74/85 qemu_x86                  stack/stack_api/kernel.stack                       FAILED: handler_crash
	see: sanity-out/qemu_x86/stack/stack_api/kernel.stack/run.log
75/85 qemu_x86                  static_idt/arch.interrupt                          PASSED
76/85 qemu_x86                  tickless/tickless/kernel.tickless                  FAILED: handler_crash
	see: sanity-out/qemu_x86/tickless/tickless/kernel.tickless/build.log
77/85 qemu_x86                  workq/work_queue/kernel.workqueue                  FAILED: handler_crash
	see: sanity-out/qemu_x86/workq/work_queue/kernel.workqueue/build.log
78/85 qemu_x86                  threads/thread_init/kernel.threads                 FAILED: handler_crash
	see: sanity-out/qemu_x86/threads/thread_init/kernel.threads/run.log
79/85 qemu_x86                  timer/timer_api/kernel.timer.tickless              PASSED
80/85 qemu_x86                  timer/timer_api/kernel.timer                       FAILED: handler_crash
	see: sanity-out/qemu_x86/timer/timer_api/kernel.timer/run.log
81/85 qemu_x86                  timer/timer_monotonic/kernel.timer                 FAILED: handler_crash
	see: sanity-out/qemu_x86/timer/timer_monotonic/kernel.timer/run.log
82/85 qemu_x86                  workq/work_queue/kernel.workqueue.poll             FAILED: handler_crash
	see: sanity-out/qemu_x86/workq/work_queue/kernel.workqueue.poll/build.log
83/85 qemu_x86                  tickless/tickless_concept/kernel.tickless          FAILED: handler_crash
	see: sanity-out/qemu_x86/tickless/tickless_concept/kernel.tickless/run.log
84/85 qemu_x86                  workq/work_queue_api/kernel.workqueue              FAILED: handler_crash
	see: sanity-out/qemu_x86/workq/work_queue_api/kernel.workqueue/run.log
85/85 qemu_x86                  xip/arch.common.xip                                PASSED
Generating coverage files...
Working on /path/to/zephyr/sanity-out/qemu_x86/jwt/libraries.encoding/handler.log
Generating gcda files
Working on /path/to/zephyr/sanity-out/qemu_x86/logging/log_msg/logging.log_msg/handler.log
Generating gcda files
Working on /path/to/zephyr/sanity-out/qemu_x86/logging/log_list/logging.log_list/handler.log
Generating gcda files
Working on /path/to/zephyr/sanity-out/qemu_x86/logging/log_output/logging.log_output/handler.log
Generating gcda files
Working on /path/to/zephyr/sanity-out/qemu_x86/logging/log_core/logging.log_core/handler.log
Generating gcda files
Working on /path/to/zephyr/sanity-out/qemu_x86/pending/kernel.common.timing/handler.log
Generating gcda files
Working on /path/to/zephyr/sanity-out/qemu_x86/fs/nffs_fs_api/basic/filesystem.nffs.basic/handler.log
Generating gcda files
Working on /path/to/zephyr/sanity-out/qemu_x86/fs/nffs_fs_api/large/filesystem.nffs.large/handler.log
Generating gcda files
Working on /path/to/zephyr/sanity-out/qemu_x86/fs/nffs_fs_api/performance/filesystem.nffs.performance/handler.log
Generating gcda files
Working on /path/to/zephyr/sanity-out/qemu_x86/fs/nffs_fs_api/cache/filesystem.nffs.cache/handler.log
Generating gcda files
Working on /path/to/zephyr/sanity-out/qemu_x86/fs/multi-fs/filesystem.fs_shell/handler.log
Generating gcda files
Working on /path/to/zephyr/sanity-out/qemu_x86/fs/multi-fs/filesystem.multifs/handler.log
Generating gcda files
Working on /path/to/zephyr/sanity-out/qemu_x86/fs/fat_fs_dual_drive/filesystem.fat/handler.log
Generating gcda files
Working on /path/to/zephyr/sanity-out/qemu_x86/mem_heap/mheap_api_concept/kernel.memory_heap/handler.log
Generating gcda files
Working on /path/to/zephyr/sanity-out/qemu_x86/stack/stack_api/kernel.stack/handler.log
Generating gcda files
Working on /path/to/zephyr/sanity-out/qemu_x86/stack/stack_usage/kernel.stack.usage/handler.log
Generating gcda files
Working on /path/to/zephyr/sanity-out/qemu_x86/threads/no-multithreading/kernel.threads.no-multithreading/handler.log
Generating gcda files
Working on /path/to/zephyr/sanity-out/qemu_x86/threads/dynamic_thread/kernel.threads.dynamic/handler.log
Generating gcda files
Working on /path/to/zephyr/sanity-out/qemu_x86/threads/thread_init/kernel.threads/handler.log
Generating gcda files
Working on /path/to/zephyr/sanity-out/qemu_x86/queue/kernel.queue.poll/handler.log
Generating gcda files
Working on /path/to/zephyr/sanity-out/qemu_x86/queue/kernel.queue/handler.log
Generating gcda files
Working on /path/to/zephyr/sanity-out/qemu_x86/obj_tracing/kernel.object_tracing/handler.log
Generating gcda files
Working on /path/to/zephyr/sanity-out/qemu_x86/early_sleep/kernel.common/handler.log
Generating gcda files
Working on /path/to/zephyr/sanity-out/qemu_x86/semaphore/semaphore/kernel.semaphore/handler.log
Generating gcda files
Working on /path/to/zephyr/sanity-out/qemu_x86/semaphore/sema_api/kernel.semaphore/handler.log
Generating gcda files
Working on /path/to/zephyr/sanity-out/qemu_x86/mbox/mbox_api/kernel.mailbox/handler.log
Generating gcda files
Working on /path/to/zephyr/sanity-out/qemu_x86/mbox/mbox_usage/kernel.mailbox/handler.log
Generating gcda files
Working on /path/to/zephyr/sanity-out/qemu_x86/pipe/pipe/kernel.pipe/handler.log
Generating gcda files
Working on /path/to/zephyr/sanity-out/qemu_x86/pipe/pipe_api/kernel.pipe/handler.log
Generating gcda files
Working on /path/to/zephyr/sanity-out/qemu_x86/poll/kernel.poll/handler.log
Generating gcda files
Working on /path/to/zephyr/sanity-out/qemu_x86/mem_slab/mslab_concept/kernel.memory_slabs/handler.log
Generating gcda files
Working on /path/to/zephyr/sanity-out/qemu_x86/mem_slab/mslab_threadsafe/kernel.memory_slabs/handler.log
Generating gcda files
Working on /path/to/zephyr/sanity-out/qemu_x86/mem_slab/mslab_api/kernel.memory_slabs/handler.log
Generating gcda files
Working on /path/to/zephyr/sanity-out/qemu_x86/mem_slab/mslab/kernel.memory_slabs/handler.log
Generating gcda files
Working on /path/to/zephyr/sanity-out/qemu_x86/lifo/lifo_usage/kernel.lifo.usage/handler.log
Generating gcda files
Working on /path/to/zephyr/sanity-out/qemu_x86/lifo/lifo_api/kernel.lifo/handler.log
Generating gcda files
Working on /path/to/zephyr/sanity-out/qemu_x86/msgq/msgq_api/kernel.message_queue/handler.log
Generating gcda files
Working on /path/to/zephyr/sanity-out/qemu_x86/device/kernel.device.pm/handler.log
Generating gcda files
Working on /path/to/zephyr/sanity-out/qemu_x86/device/kernel.device/handler.log
Generating gcda files
Working on /path/to/zephyr/sanity-out/qemu_x86/fifo/fifo_usage/kernel.fifo.usage/handler.log
Generating gcda files
Working on /path/to/zephyr/sanity-out/qemu_x86/fifo/fifo_usage/kernel.fifo.usage.poll/handler.log
Generating gcda files
Working on /path/to/zephyr/sanity-out/qemu_x86/fifo/fifo_api/kernel.fifo/handler.log
Generating gcda files
Working on /path/to/zephyr/sanity-out/qemu_x86/fifo/fifo_api/kernel.fifo.poll/handler.log
Generating gcda files
Working on /path/to/zephyr/sanity-out/qemu_x86/timer/timer_monotonic/kernel.timer/handler.log
Generating gcda files
Working on /path/to/zephyr/sanity-out/qemu_x86/timer/timer_api/kernel.timer/handler.log
Generating gcda files
Working on /path/to/zephyr/sanity-out/qemu_x86/tickless/tickless_concept/kernel.tickless/handler.log
Generating gcda files
Working on /path/to/zephyr/sanity-out/qemu_x86/critical/kernel.common/handler.log
Generating gcda files
Working on /path/to/zephyr/sanity-out/qemu_x86/sched/preempt/kernel.sched.preempt/handler.log
Generating gcda files
Working on /path/to/zephyr/sanity-out/qemu_x86/sched/deadline/kernel.sched.deadline/handler.log
Generating gcda files
Working on /path/to/zephyr/sanity-out/qemu_x86/mem_pool/mem_pool_threadsafe/kernel.memory_pool/handler.log
Generating gcda files
Working on /path/to/zephyr/sanity-out/qemu_x86/mem_pool/sys_mem_pool/kernel.memory_pool/handler.log
Generating gcda files
Working on /path/to/zephyr/sanity-out/qemu_x86/mem_pool/mem_pool_api/kernel.memory_pool/handler.log
Generating gcda files
Working on /path/to/zephyr/sanity-out/qemu_x86/mem_pool/mem_pool/kernel.memory_pool/handler.log
Generating gcda files
Working on /path/to/zephyr/sanity-out/qemu_x86/mem_pool/mem_pool_concept/kernel.memory_pool/handler.log
Generating gcda files
Working on /path/to/zephyr/sanity-out/qemu_x86/workq/work_queue_api/kernel.workqueue/handler.log
Generating gcda files
Working on /path/to/zephyr/sanity-out/qemu_x86/xip/arch.common.xip/handler.log
Generating gcda files
Working on /path/to/zephyr/sanity-out/qemu_x86/mem_protect/stackprot/kernel.memory_protection/handler.log
Generating gcda files
Working on /path/to/zephyr/sanity-out/qemu_x86/mem_protect/stack_random/kernel.memory_protection.stack_random/handler.log
Generating gcda files
Working on /path/to/zephyr/sanity-out/qemu_x86/mem_protect/protection/kernel.memory_protection.protection/handler.log
Generating gcda files
Working on /path/to/zephyr/sanity-out/qemu_x86/can/frame/can.frame/handler.log
Generating gcda files
Working on /path/to/zephyr/sanity-out/qemu_x86/static_idt/arch.interrupt/handler.log
Generating gcda files
Working on /path/to/zephyr/sanity-out/qemu_x86/context/kernel.common/handler.log
Generating gcda files
Working on /path/to/zephyr/sanity-out/qemu_x86/boot_page_table/kernel.memory_protection/handler.log
Generating gcda files
Working on /path/to/zephyr/sanity-out/qemu_x86/profiling/profiling_api/kernel.profiling/handler.log
Generating gcda files
Working on /path/to/zephyr/sanity-out/qemu_x86/mutex/mutex_api/kernel.mutex/handler.log
Generating gcda files
Working on /path/to/zephyr/sanity-out/qemu_x86/mutex/mutex/kernel.mutex/handler.log
Generating gcda files
HTML report generated: /path/to/zephyr/sanity-out/coverage/index.html
29 of 85 tests passed with 0 warnings in 327 seconds

Environment (please complete the following information):

• OS: fedora29
• Toolchain (e.g Zephyr SDK, ...): zephyr
• Commit SHA or Version used: a907e69

[cinlyooi-intel]

On @inakypg suggestion I started to examine the handler.log files

The few handler.log files that I check all have qemu commands. I took away the --serial parameter and run them. All produces "Double Fault" -- "Stack Check Fail!" like the example below

$/opt/zephyr-sdk/sysroots/x86_64-pokysdk-linux/usr/bin/qemu-system-i386 -m 8 -cpu qemu32,+nx,+pae -device isa-debug-exit,iobase=0xf4,iosize=0x04 -no-reboot -nographic -no-acpi -net none -pidfile qemu.pid -kernel /path/to/sanity-out/qemu_x86/tests/kernel/mem_pool/mem_pool/kernel.memory_pool/zephyr/zephyr.elf

SeaBIOS (version rel-1.11.2-0-gf9626ccb91-prebuilt.qemu-project.org)
Booting from ROM..***** Booting Zephyr OS v1.14.0-rc1-1138-g4278bf567348 *****
Running test suite mempool
===================================================================
starting test - test_pool_block_get
PASS - test_pool_block_get
===================================================================
starting test - test_pool_block_get_timeout
***** Double Fault *****
***** Stack Check Fail! *****
Current thread ID = 0x0040d560
eax: 0x0040c000, ebx: 0x80000000, ecx: 0x0040c007, edx: 0x00000000
esi: 0x0040c007, edi: 0x0040d5c0, ebp: 0x0042908c, esp: 0x00429000
eflags: 0x00000012 cs: 0x0008
call trace:
eip: 0x0000d1cb
     0x0000d7a9 (0x440000)
     0x0000d885 (0x40d3e0)
     0x0000d95d (0x40d3e0)
     0x0000c7a9 (0x40d380)
     0x000014bf (0x40d380)
     0x00011d50 (0x0)
     0x00011dbe (0x0)
     0x00006870 (0x0)
Fatal fault in essential thread! Spinning...
Terminate emulator due to fatal kernel error

[wentongwu]

run the case(e.g. kernel.workqueue) manually, it works well. It should be someone's commit cause this issue. I'm checking this issue.

[wentongwu]

.noinit."/home/wentongw/zephyr/zephyrproject/zephyr/kernel/init.c".2
0x0000000000424000 0x3000 zephyr/kernel/libkernel.a(init.c.obj)
0x0000000000424000 _idle_stack

***** Stack Check Fail! *****
Current thread ID = 0x0040c3a0
eax: 0x0040b000, ebx: 0x80000000, ecx: 0x0040b067, edx: 0x00000000
esi: 0x0040b067, edi: 0x0040c440, ebp: 0x0042608c, esp: 0x00426000
eflags: 0x00000012 cs: 0x0008

@andrewboie for above log, if increasing the idle thread stack, the case works well. it seems x86 also has stack overflow issue. But I'm not familiar with x86 arch, why either ebp or esp not equal to _idle_stack? and what's x86's mpu(hope I named it correctly) mechanism? Thanks a lot!

also I get the trace when exception happens, idle -> sys_power_save_idle -> k_cpu_idle(x86's hlt instruction) and then the trace shows it's in __swap -> _x86_swap_update_page_tables -> z_arch_mem_domain_configure -> _x86_mem_domain_pages_update -> activate_partition -> z_x86_mmu_set_flags -> tlb_flush_page . So the hlt instruction will make cpu in privilege mode and run __swap? what's the trigger instruction when want to do thread swap like arm's SVC (is it int 0x80 or something else ?) And for this case, it seems cpu is running idle thread and will go to low power mode(hope named it correctly), so there will be page switching for idle thread? and why zephyr using paging for x86? I'm not familiar x86 arch, so maybe what I said is wrong and Thanks for your time! @andrewboie

Anyway I'm increasing all kinds of stack size to fix this issue.

[cinlyooi-intel]

@wentongwu : Do you get this error linker_pass_final.cmd:65 cannot move location counter backwards (from 000000000002e34e to 000000000002dfaa) at all?

[wentongwu]

@wentongwu : Do you get this error linker_pass_final.cmd:65 cannot move location counter backwards (from 000000000002e34e to 000000000002dfaa) at all?

I didn't get this. which case are you running? Did you apply my PR?

[cinlyooi-intel]

@wentongwu :
Two scenarios

(1) Pure PR15002 no longer have handler_crash, but does not generate gcov data

$ git fetch origin pull/15002/head:pull/15002
$ git checkout pull/15002
$ ./scripts/sanitycheck --coverage --coverage-platform qemu_x86 -p qemu_x86 --gcov-tool /opt/zephyr-sdk-0.10.0/i586-zephyr-elf/bin/i586-zephyr-elf-gcov -T tests/kernel/common  | tee output.log
JOBS: 8
Cleaning output directory /home/cinlyooi/tests/zephyr.git/sanity-out
Building testcase defconfigs...
2 tests selected, 322 tests discarded due to filters
1/2 qemu_x86                  tests/kernel/common/kernel.common.misra            PASSED
2/2 qemu_x86                  tests/kernel/common/kernel.common                  PASSED
Generating coverage files...
geninfo: WARNING: no .gcda files found in /home/cinlyooi/tests/zephyr.git/sanity-out - skipping!
lcov: ERROR: no valid records found in tracefile /home/cinlyooi/tests/zephyr.git/sanity-out/coverage.info
lcov: ERROR: no valid records found in tracefile /home/cinlyooi/tests/zephyr.git/sanity-out/coverage.info
lcov: ERROR: no valid records found in tracefile /home/cinlyooi/tests/zephyr.git/sanity-out/coverage.info
lcov: ERROR: no valid records found in tracefile /home/cinlyooi/tests/zephyr.git/sanity-out/coverage.info
genhtml: ERROR: no valid records found in tracefile /home/cinlyooi/tests/zephyr.git/sanity-out/coverage.info
2 of 2 tests passed with 0 warnings in 19 seconds

Examining the output shows aht the GCDA data is not created:

$ tail sanity-out/qemu_x86/tests/kernel/pending/kernel.common.timing/handler.log 
Testing lifos delivered data correctly ...
Testing preemptible thread waiting on timer ...
PASS - test_pending
===================================================================
Test suite pend succeeded
===================================================================
PROJECT EXECUTION SUCCESSFUL

GCOV_COVERAGE_DUMP_START
GCOV_COVERAGE_DUMP_END

Note that I can get GCOV_COVERAGE_DUMP if I run on mpns2_an385. As such, the command line itself is not the problem.

(2) If I merge PR15002 into master branch

$ git checkout master #9e81cbea7df8efad52cc23e0bbdbdde5db997e56
$ git merge pull/15002
$ ./scripts/sanitycheck --coverage --coverage-platform qemu_x86 -p qemu_x86 --gcov-tool /opt/zephyr-sdk-0.10.0/i586-zephyr-elf/bin/i586-zephyr-elf-gcov -T tests/kernel/common  | tee output.log
JOBS: 8
Cleaning output directory /home/cinlyooi/tests/zephyr.git/sanity-out
Building testcase defconfigs...
2 tests selected, 322 tests discarded due to filters
1/2 qemu_x86                  tests/kernel/common/kernel.common                  FAILED: handler_crash
	see: sanity-out/qemu_x86/tests/kernel/common/kernel.common/build.log
2/2 qemu_x86                  tests/kernel/common/kernel.common.misra            FAILED: handler_crash
	see: sanity-out/qemu_x86/tests/kernel/common/kernel.common.misra/build.log
Generating coverage files...
geninfo: WARNING: no .gcda files found in /home/cinlyooi/tests/zephyr.git/sanity-out - skipping!
lcov: ERROR: no valid records found in tracefile /home/cinlyooi/tests/zephyr.git/sanity-out/coverage.info
lcov: ERROR: no valid records found in tracefile /home/cinlyooi/tests/zephyr.git/sanity-out/coverage.info
lcov: ERROR: no valid records found in tracefile /home/cinlyooi/tests/zephyr.git/sanity-out/coverage.info
lcov: ERROR: no valid records found in tracefile /home/cinlyooi/tests/zephyr.git/sanity-out/coverage.info
genhtml: ERROR: no valid records found in tracefile /home/cinlyooi/tests/zephyr.git/sanity-out/coverage.info
0 of 2 tests passed with 0 warnings in 8 seconds

Handler crash again. if we look at build.log we have the error shown in previous comment

$ cat sanity-out/qemu_x86/tests/kernel/common/kernel.common/build.log
# <snip>
[ 98%] Building C object zephyr/CMakeFiles/kernel_elf.dir/misc/empty_file.c.obj
[100%] Linking C executable zephyr.elf
/opt/zephyr-sdk-0.10.0/i586-zephyr-elf/bin/../lib/gcc/i586-zephyr-elf/8.3.0/../../../../i586-zephyr-elf/bin/ld:linker_pass_final.cm
d:65 cannot move location counter backwards (from 0000000000047b1c to 0000000000047774)
collect2: error: ld returned 1 exit status
make[3]: *** [zephyr/CMakeFiles/kernel_elf.dir/build.make:102: zephyr/zephyr.elf] Error 1

#<snip>

[wentongwu]

@cinlyooi-intel without my PR, master branch also can't run sanitycheck. It's not my PR cause master branch broken, please open another github issue to track the issue. Thanks

[cinlyooi-intel]

@wentongwu: Sent (2) to #15009. Not going to deal with issue (1) for now as it would had been blocked by issue (2) if I filed a GitHub ticket for it. It is also unclear whether issue (1) will persist once issue (2) is resolved.