sanitycheck --coverage: stack overflows on qemu_x86 and mps2_an385

[cinlyooi-intel]

Describe the bug
We have too many timeouts when running sanitycheck --coverageon mps2_an385. The handler.logs generated by sanitycheck are empty.

Impact
Blocking code coverage improvement work.

To Reproduce and screenshots

$./scripts/sanitycheck -v  --coverage --coverage-platform mps2_an385 -p mps2_an385 --gcov-tool /opt/zephyr-sdk-0.9.5/sysroots/x86_64-pokysdk-linux/usr/bin/i586-zephyr-elf/i586-zephyr-elf-gcov -T tests/kernel/ -T tests/subsys/
JOBS: 8
Cleaning output directory /path/to/zephyr.git/sanity-out
Reading test case configuration files under /path/to/zephyr.git/tests/kernel...
Reading test case configuration files under /path/to/zephyr.git/tests/subsys...
Reading platform configuration files under /path/to/zephyr.git/boards...
Reading platform configuration files under /path/to/zephyr.git/scripts/sanity_chk/boards...
Building testcase defconfigs...
73 tests selected, 17315 tests discarded due to filters
 1/73 mps2_an385                tests/kernel/fatal/kernel.common.stack_sentinel    PASSED
 2/73 mps2_an385                tests/kernel/common/kernel.common                  PASSED
 3/73 mps2_an385                tests/kernel/fatal/kernel.common.stack_protection  FAILED: timeout
	see: sanity-out/mps2_an385/tests/kernel/fatal/kernel.common.stack_protection/handler.log
 4/73 mps2_an385                tests/kernel/arm_irq_vector_table/arch.interrupt   FAILED: timeout
	see: sanity-out/mps2_an385/tests/kernel/arm_irq_vector_table/arch.interrupt/handler.log
 5/73 mps2_an385                tests/kernel/critical/kernel.common                FAILED: timeout
	see: sanity-out/mps2_an385/tests/kernel/critical/kernel.common/handler.log
 6/73 mps2_an385                tests/kernel/early_sleep/kernel.common             FAILED: timeout
	see: sanity-out/mps2_an385/tests/kernel/early_sleep/kernel.common/handler.log
 7/73 mps2_an385                tests/kernel/context/kernel.common                 FAILED: timeout
	see: sanity-out/mps2_an385/tests/kernel/context/kernel.common/handler.log
 8/73 mps2_an385                tests/kernel/arm_runtime_nmi/arch.interrupt        FAILED: timeout
	see: sanity-out/mps2_an385/tests/kernel/arm_runtime_nmi/arch.interrupt/handler.log
 9/73 mps2_an385                tests/kernel/fifo/fifo_api/kernel.fifo             FAILED: timeout
	see: sanity-out/mps2_an385/tests/kernel/fifo/fifo_api/kernel.fifo/handler.log
10/73 mps2_an385                tests/kernel/fifo/fifo_api/kernel.fifo.poll        FAILED: timeout
	see: sanity-out/mps2_an385/tests/kernel/fifo/fifo_api/kernel.fifo.poll/handler.log
11/73 mps2_an385                tests/kernel/fifo/fifo_usage/kernel.fifo.usage     FAILED: timeout
	see: sanity-out/mps2_an385/tests/kernel/fifo/fifo_usage/kernel.fifo.usage/handler.log
12/73 mps2_an385                tests/kernel/fifo/fifo_timeout/kernel.fifo.timeout FAILED: timeout
	see: sanity-out/mps2_an385/tests/kernel/fifo/fifo_timeout/kernel.fifo.timeout/handler.log
13/73 mps2_an385                tests/kernel/fifo/fifo_timeout/kernel.fifo.timeout.poll FAILED: timeout
	see: sanity-out/mps2_an385/tests/kernel/fifo/fifo_timeout/kernel.fifo.timeout.poll/handler.log
14/73 mps2_an385                tests/kernel/gen_isr_table/arch.interrupt          FAILED: timeout
	see: sanity-out/mps2_an385/tests/kernel/gen_isr_table/arch.interrupt/handler.log
15/73 mps2_an385                tests/kernel/fifo/fifo_usage/kernel.fifo.usage.poll FAILED: timeout
	see: sanity-out/mps2_an385/tests/kernel/fifo/fifo_usage/kernel.fifo.usage.poll/handler.log
16/73 mps2_an385                tests/kernel/interrupt/arch.interrupt              FAILED: timeout
	see: sanity-out/mps2_an385/tests/kernel/interrupt/arch.interrupt/handler.log
17/73 mps2_an385                tests/kernel/lifo/lifo_api/kernel.lifo             FAILED: timeout
	see: sanity-out/mps2_an385/tests/kernel/lifo/lifo_api/kernel.lifo/handler.log
18/73 mps2_an385                tests/kernel/lifo/lifo_usage/kernel.lifo.usage     FAILED: timeout
	see: sanity-out/mps2_an385/tests/kernel/lifo/lifo_usage/kernel.lifo.usage/handler.log
19/73 mps2_an385                tests/kernel/mbox/mbox_api/kernel.mailbox          FAILED: timeout
	see: sanity-out/mps2_an385/tests/kernel/mbox/mbox_api/kernel.mailbox/handler.log
20/73 mps2_an385                tests/kernel/mbox/mbox_usage/kernel.mailbox        FAILED: timeout
	see: sanity-out/mps2_an385/tests/kernel/mbox/mbox_usage/kernel.mailbox/handler.log
21/73 mps2_an385                tests/kernel/mem_heap/mheap_api_concept/kernel.memory_heap FAILED: timeout
	see: sanity-out/mps2_an385/tests/kernel/mem_heap/mheap_api_concept/kernel.memory_heap/handler.log
22/73 mps2_an385                tests/kernel/mem_pool/mem_pool/kernel.memory_pool  FAILED: timeout
	see: sanity-out/mps2_an385/tests/kernel/mem_pool/mem_pool/kernel.memory_pool/handler.log
23/73 mps2_an385                tests/kernel/mem_pool/mem_pool_api/kernel.memory_pool FAILED: timeout
	see: sanity-out/mps2_an385/tests/kernel/mem_pool/mem_pool_api/kernel.memory_pool/handler.log
24/73 mps2_an385                tests/kernel/mem_pool/mem_pool_concept/kernel.memory_pool FAILED: timeout
	see: sanity-out/mps2_an385/tests/kernel/mem_pool/mem_pool_concept/kernel.memory_pool/handler.log
25/73 mps2_an385                tests/kernel/mem_pool/mem_pool_threadsafe/kernel.memory_pool FAILED: timeout
	see: sanity-out/mps2_an385/tests/kernel/mem_pool/mem_pool_threadsafe/kernel.memory_pool/handler.log
26/73 mps2_an385                tests/kernel/mem_pool/sys_mem_pool/kernel.memory_pool FAILED: timeout
	see: sanity-out/mps2_an385/tests/kernel/mem_pool/sys_mem_pool/kernel.memory_pool/handler.log
27/73 mps2_an385                tests/kernel/mem_protect/obj_validation/kernel.memory_protection.obj_validation FAILED: timeout
	see: sanity-out/mps2_an385/tests/kernel/mem_protect/obj_validation/kernel.memory_protection.obj_validation/handler.log
28/73 mps2_an385                tests/kernel/mem_protect/protection/kernel.memory_protection.protection FAILED: timeout
	see: sanity-out/mps2_an385/tests/kernel/mem_protect/protection/kernel.memory_protection.protection/handler.log
29/73 mps2_an385                tests/kernel/mem_protect/stackprot/kernel.memory_protection FAILED: timeout
	see: sanity-out/mps2_an385/tests/kernel/mem_protect/stackprot/kernel.memory_protection/handler.log
30/73 mps2_an385                tests/kernel/mem_protect/stack_random/kernel.memory_protection.stack_random FAILED: timeout
	see: sanity-out/mps2_an385/tests/kernel/mem_protect/stack_random/kernel.memory_protection.stack_random/handler.log
31/73 mps2_an385                tests/kernel/mem_protect/mem_protect/kernel.memory_protection FAILED: timeout
	see: sanity-out/mps2_an385/tests/kernel/mem_protect/mem_protect/kernel.memory_protection/handler.log
32/73 mps2_an385                tests/kernel/mem_protect/syscalls/kernel.memory_protection.syscalls FAILED: timeout
	see: sanity-out/mps2_an385/tests/kernel/mem_protect/syscalls/kernel.memory_protection.syscalls/handler.log
33/73 mps2_an385                tests/kernel/mem_protect/userspace/kernel.memory_protection.userspace FAILED: timeout
	see: sanity-out/mps2_an385/tests/kernel/mem_protect/userspace/kernel.memory_protection.userspace/handler.log
34/73 mps2_an385                tests/kernel/mem_slab/mslab/kernel.memory_slabs    FAILED: timeout
	see: sanity-out/mps2_an385/tests/kernel/mem_slab/mslab/kernel.memory_slabs/handler.log
35/73 mps2_an385                tests/kernel/mem_slab/mslab_threadsafe/kernel.memory_slabs FAILED: timeout
	see: sanity-out/mps2_an385/tests/kernel/mem_slab/mslab_threadsafe/kernel.memory_slabs/handler.log
36/73 mps2_an385                tests/kernel/mem_slab/mslab_concept/kernel.memory_slabs FAILED: timeout
	see: sanity-out/mps2_an385/tests/kernel/mem_slab/mslab_concept/kernel.memory_slabs/handler.log
37/73 mps2_an385                tests/kernel/mem_slab/mslab_api/kernel.memory_slabs FAILED: timeout
	see: sanity-out/mps2_an385/tests/kernel/mem_slab/mslab_api/kernel.memory_slabs/handler.log
38/73 mps2_an385                tests/kernel/msgq/msgq_api/kernel.message_queue    FAILED: timeout
	see: sanity-out/mps2_an385/tests/kernel/msgq/msgq_api/kernel.message_queue/handler.log
39/73 mps2_an385                tests/kernel/mutex/mutex_api/kernel.mutex          FAILED: timeout
	see: sanity-out/mps2_an385/tests/kernel/mutex/mutex_api/kernel.mutex/handler.log
40/73 mps2_an385                tests/kernel/mutex/mutex/kernel.mutex              FAILED: timeout
	see: sanity-out/mps2_an385/tests/kernel/mutex/mutex/kernel.mutex/handler.log
41/73 mps2_an385                tests/kernel/obj_tracing/kernel.object_tracing     PASSED
42/73 mps2_an385                tests/kernel/pending/kernel.common.timing          FAILED: timeout
	see: sanity-out/mps2_an385/tests/kernel/pending/kernel.common.timing/handler.log
43/73 mps2_an385                tests/kernel/pipe/pipe/kernel.pipe                 FAILED: timeout
	see: sanity-out/mps2_an385/tests/kernel/pipe/pipe/kernel.pipe/handler.log
44/73 mps2_an385                tests/kernel/poll/kernel.poll                      FAILED: timeout
	see: sanity-out/mps2_an385/tests/kernel/poll/kernel.poll/handler.log
45/73 mps2_an385                tests/kernel/pipe/pipe_api/kernel.pipe             FAILED: timeout
	see: sanity-out/mps2_an385/tests/kernel/pipe/pipe_api/kernel.pipe/handler.log
46/73 mps2_an385                tests/kernel/profiling/profiling_api/kernel.profiling FAILED: timeout
	see: sanity-out/mps2_an385/tests/kernel/profiling/profiling_api/kernel.profiling/handler.log
47/73 mps2_an385                tests/kernel/queue/kernel.queue                    FAILED: timeout
	see: sanity-out/mps2_an385/tests/kernel/queue/kernel.queue/handler.log
48/73 mps2_an385                tests/kernel/queue/kernel.queue.poll               FAILED: timeout
	see: sanity-out/mps2_an385/tests/kernel/queue/kernel.queue.poll/handler.log
49/73 mps2_an385                tests/kernel/sched/deadline/kernel.sched.deadline  FAILED: timeout
	see: sanity-out/mps2_an385/tests/kernel/sched/deadline/kernel.sched.deadline/handler.log
50/73 mps2_an385                tests/kernel/sched/preempt/kernel.sched.preempt    FAILED: timeout
	see: sanity-out/mps2_an385/tests/kernel/sched/preempt/kernel.sched.preempt/handler.log
51/73 mps2_an385                tests/kernel/semaphore/semaphore/kernel.semaphore  FAILED: timeout
	see: sanity-out/mps2_an385/tests/kernel/semaphore/semaphore/kernel.semaphore/handler.log
52/73 mps2_an385                tests/kernel/sched/schedule_api/kernel.sched       FAILED: timeout
	see: sanity-out/mps2_an385/tests/kernel/sched/schedule_api/kernel.sched/handler.log
53/73 mps2_an385                tests/kernel/sleep/kernel.common.timing            FAILED: timeout
	see: sanity-out/mps2_an385/tests/kernel/sleep/kernel.common.timing/handler.log
54/73 mps2_an385                tests/kernel/semaphore/sema_api/kernel.semaphore   FAILED: timeout
	see: sanity-out/mps2_an385/tests/kernel/semaphore/sema_api/kernel.semaphore/handler.log
55/73 mps2_an385                tests/kernel/stack/stack_usage/kernel.stack.usage  FAILED: timeout
	see: sanity-out/mps2_an385/tests/kernel/stack/stack_usage/kernel.stack.usage/handler.log
56/73 mps2_an385                tests/kernel/threads/dynamic_thread/kernel.threads.dynamic FAILED: timeout
	see: sanity-out/mps2_an385/tests/kernel/threads/dynamic_thread/kernel.threads.dynamic/handler.log
57/73 mps2_an385                tests/kernel/timer/timer_api/kernel.timer.tickless PASSED
58/73 mps2_an385                tests/kernel/stack/stack_api/kernel.stack          FAILED: timeout
	see: sanity-out/mps2_an385/tests/kernel/stack/stack_api/kernel.stack/handler.log
59/73 mps2_an385                tests/kernel/tickless/tickless_concept/kernel.tickless FAILED: timeout
	see: sanity-out/mps2_an385/tests/kernel/tickless/tickless_concept/kernel.tickless/handler.log
60/73 mps2_an385                tests/kernel/threads/no-multithreading/kernel.threads.no-multithreading PASSED
61/73 mps2_an385                tests/kernel/threads/thread_init/kernel.threads    FAILED: timeout
	see: sanity-out/mps2_an385/tests/kernel/threads/thread_init/kernel.threads/handler.log
62/73 mps2_an385                tests/kernel/timer/timer_monotonic/kernel.timer    FAILED: timeout
	see: sanity-out/mps2_an385/tests/kernel/timer/timer_monotonic/kernel.timer/handler.log
63/73 mps2_an385                tests/kernel/workq/work_queue/kernel.workqueue     FAILED: timeout
	see: sanity-out/mps2_an385/tests/kernel/workq/work_queue/kernel.workqueue/handler.log
64/73 mps2_an385                tests/kernel/timer/timer_api/kernel.timer          FAILED: timeout
	see: sanity-out/mps2_an385/tests/kernel/timer/timer_api/kernel.timer/handler.log
65/73 mps2_an385                tests/kernel/workq/work_queue/kernel.workqueue.poll FAILED: timeout
	see: sanity-out/mps2_an385/tests/kernel/workq/work_queue/kernel.workqueue.poll/handler.log
66/73 mps2_an385                tests/kernel/xip/arch.common.xip                   FAILED: timeout
	see: sanity-out/mps2_an385/tests/kernel/xip/arch.common.xip/handler.log
67/73 mps2_an385                tests/kernel/workq/work_queue_api/kernel.workqueue FAILED: timeout
	see: sanity-out/mps2_an385/tests/kernel/workq/work_queue_api/kernel.workqueue/handler.log
68/73 mps2_an385                tests/subsys/jwt/libraries.encoding                PASSED
69/73 mps2_an385                tests/kernel/threads/thread_apis/kernel.threads    FAILED: timeout
	see: sanity-out/mps2_an385/tests/kernel/threads/thread_apis/kernel.threads/handler.log
70/73 mps2_an385                tests/subsys/logging/log_list/logging.log_list     PASSED
71/73 mps2_an385                tests/subsys/logging/log_output/logging.log_output PASSED
72/73 mps2_an385                tests/subsys/logging/log_core/logging.log_core     FAILED: timeout
	see: sanity-out/mps2_an385/tests/subsys/logging/log_core/logging.log_core/handler.log
73/73 mps2_an385                tests/subsys/logging/log_msg/logging.log_msg       FAILED: timeout
	see: sanity-out/mps2_an385/tests/subsys/logging/log_msg/logging.log_msg/handler.log
Generating coverage files...
Working on /path/to/zephyr.git/sanity-out/mps2_an385/tests/kernel/pending/kernel.common.timing/handler.log
Generating gcda files
Working on /path/to/zephyr.git/sanity-out/mps2_an385/tests/kernel/mem_heap/mheap_api_concept/kernel.memory_heap/handler.log
Generating gcda files
Working on /path/to/zephyr.git/sanity-out/mps2_an385/tests/kernel/gen_isr_table/arch.interrupt/handler.log
Generating gcda files
Working on /path/to/zephyr.git/sanity-out/mps2_an385/tests/kernel/stack/stack_api/kernel.stack/handler.log
Generating gcda files
Working on /path/to/zephyr.git/sanity-out/mps2_an385/tests/kernel/stack/stack_usage/kernel.stack.usage/handler.log
Generating gcda files
Working on /path/to/zephyr.git/sanity-out/mps2_an385/tests/kernel/threads/no-multithreading/kernel.threads.no-multithreading/handler.log
Generating gcda files
Working on /path/to/zephyr.git/sanity-out/mps2_an385/tests/kernel/threads/dynamic_thread/kernel.threads.dynamic/handler.log
Generating gcda files
Working on /path/to/zephyr.git/sanity-out/mps2_an385/tests/kernel/threads/thread_apis/kernel.threads/handler.log
Generating gcda files
Working on /path/to/zephyr.git/sanity-out/mps2_an385/tests/kernel/threads/thread_init/kernel.threads/handler.log
Generating gcda files
Working on /path/to/zephyr.git/sanity-out/mps2_an385/tests/kernel/queue/kernel.queue.poll/handler.log
Generating gcda files
Working on /path/to/zephyr.git/sanity-out/mps2_an385/tests/kernel/queue/kernel.queue/handler.log
Generating gcda files
Working on /path/to/zephyr.git/sanity-out/mps2_an385/tests/kernel/obj_tracing/kernel.object_tracing/handler.log
Generating gcda files
Working on /path/to/zephyr.git/sanity-out/mps2_an385/tests/kernel/early_sleep/kernel.common/handler.log
Generating gcda files
Working on /path/to/zephyr.git/sanity-out/mps2_an385/tests/kernel/interrupt/arch.interrupt/handler.log
Generating gcda files
Working on /path/to/zephyr.git/sanity-out/mps2_an385/tests/kernel/semaphore/semaphore/kernel.semaphore/handler.log
Generating gcda files
Working on /path/to/zephyr.git/sanity-out/mps2_an385/tests/kernel/semaphore/sema_api/kernel.semaphore/handler.log
Generating gcda files
Working on /path/to/zephyr.git/sanity-out/mps2_an385/tests/kernel/mbox/mbox_api/kernel.mailbox/handler.log
Generating gcda files
Working on /path/to/zephyr.git/sanity-out/mps2_an385/tests/kernel/mbox/mbox_usage/kernel.mailbox/handler.log
Generating gcda files
Working on /path/to/zephyr.git/sanity-out/mps2_an385/tests/kernel/pipe/pipe/kernel.pipe/handler.log
Generating gcda files
Working on /path/to/zephyr.git/sanity-out/mps2_an385/tests/kernel/pipe/pipe_api/kernel.pipe/handler.log
Generating gcda files
Working on /path/to/zephyr.git/sanity-out/mps2_an385/tests/kernel/sleep/kernel.common.timing/handler.log
Generating gcda files
Working on /path/to/zephyr.git/sanity-out/mps2_an385/tests/kernel/poll/kernel.poll/handler.log
Generating gcda files
Working on /path/to/zephyr.git/sanity-out/mps2_an385/tests/kernel/mem_slab/mslab_concept/kernel.memory_slabs/handler.log
Generating gcda files
Working on /path/to/zephyr.git/sanity-out/mps2_an385/tests/kernel/mem_slab/mslab_threadsafe/kernel.memory_slabs/handler.log
Generating gcda files
Working on /path/to/zephyr.git/sanity-out/mps2_an385/tests/kernel/mem_slab/mslab_api/kernel.memory_slabs/handler.log
Generating gcda files
Working on /path/to/zephyr.git/sanity-out/mps2_an385/tests/kernel/mem_slab/mslab/kernel.memory_slabs/handler.log
Generating gcda files
Working on /path/to/zephyr.git/sanity-out/mps2_an385/tests/kernel/lifo/lifo_usage/kernel.lifo.usage/handler.log
Generating gcda files
Working on /path/to/zephyr.git/sanity-out/mps2_an385/tests/kernel/lifo/lifo_api/kernel.lifo/handler.log
Generating gcda files
Working on /path/to/zephyr.git/sanity-out/mps2_an385/tests/kernel/msgq/msgq_api/kernel.message_queue/handler.log
Generating gcda files
Working on /path/to/zephyr.git/sanity-out/mps2_an385/tests/kernel/fifo/fifo_usage/kernel.fifo.usage/handler.log
Generating gcda files
Working on /path/to/zephyr.git/sanity-out/mps2_an385/tests/kernel/fifo/fifo_usage/kernel.fifo.usage.poll/handler.log
Generating gcda files
Working on /path/to/zephyr.git/sanity-out/mps2_an385/tests/kernel/fifo/fifo_timeout/kernel.fifo.timeout/handler.log
Generating gcda files
Working on /path/to/zephyr.git/sanity-out/mps2_an385/tests/kernel/fifo/fifo_timeout/kernel.fifo.timeout.poll/handler.log
Generating gcda files
Working on /path/to/zephyr.git/sanity-out/mps2_an385/tests/kernel/fifo/fifo_api/kernel.fifo/handler.log
Generating gcda files
Working on /path/to/zephyr.git/sanity-out/mps2_an385/tests/kernel/fifo/fifo_api/kernel.fifo.poll/handler.log
Generating gcda files
Working on /path/to/zephyr.git/sanity-out/mps2_an385/tests/kernel/timer/timer_monotonic/kernel.timer/handler.log
Generating gcda files
Working on /path/to/zephyr.git/sanity-out/mps2_an385/tests/kernel/timer/timer_api/kernel.timer/handler.log
Generating gcda files
Working on /path/to/zephyr.git/sanity-out/mps2_an385/tests/kernel/tickless/tickless_concept/kernel.tickless/handler.log
Generating gcda files
Working on /path/to/zephyr.git/sanity-out/mps2_an385/tests/kernel/critical/kernel.common/handler.log
Generating gcda files
Working on /path/to/zephyr.git/sanity-out/mps2_an385/tests/kernel/sched/preempt/kernel.sched.preempt/handler.log
Generating gcda files
Working on /path/to/zephyr.git/sanity-out/mps2_an385/tests/kernel/sched/schedule_api/kernel.sched/handler.log
Generating gcda files
Working on /path/to/zephyr.git/sanity-out/mps2_an385/tests/kernel/sched/deadline/kernel.sched.deadline/handler.log
Generating gcda files
Working on /path/to/zephyr.git/sanity-out/mps2_an385/tests/kernel/arm_runtime_nmi/arch.interrupt/handler.log
Generating gcda files
Working on /path/to/zephyr.git/sanity-out/mps2_an385/tests/kernel/mem_pool/mem_pool_threadsafe/kernel.memory_pool/handler.log
Generating gcda files
Working on /path/to/zephyr.git/sanity-out/mps2_an385/tests/kernel/mem_pool/sys_mem_pool/kernel.memory_pool/handler.log
Generating gcda files
Working on /path/to/zephyr.git/sanity-out/mps2_an385/tests/kernel/mem_pool/mem_pool_api/kernel.memory_pool/handler.log
Generating gcda files
Working on /path/to/zephyr.git/sanity-out/mps2_an385/tests/kernel/mem_pool/mem_pool/kernel.memory_pool/handler.log
Generating gcda files
Working on /path/to/zephyr.git/sanity-out/mps2_an385/tests/kernel/mem_pool/mem_pool_concept/kernel.memory_pool/handler.log
Generating gcda files
Working on /path/to/zephyr.git/sanity-out/mps2_an385/tests/kernel/workq/work_queue_api/kernel.workqueue/handler.log
Generating gcda files
Working on /path/to/zephyr.git/sanity-out/mps2_an385/tests/kernel/workq/work_queue/kernel.workqueue.poll/handler.log
Generating gcda files
Working on /path/to/zephyr.git/sanity-out/mps2_an385/tests/kernel/workq/work_queue/kernel.workqueue/handler.log
Generating gcda files
Working on /path/to/zephyr.git/sanity-out/mps2_an385/tests/kernel/xip/arch.common.xip/handler.log
Generating gcda files
Working on /path/to/zephyr.git/sanity-out/mps2_an385/tests/kernel/fatal/kernel.common.stack_protection/handler.log
Generating gcda files
Working on /path/to/zephyr.git/sanity-out/mps2_an385/tests/kernel/fatal/kernel.common.stack_sentinel/handler.log
Generating gcda files
Working on /path/to/zephyr.git/sanity-out/mps2_an385/tests/kernel/mem_protect/stackprot/kernel.memory_protection/handler.log
Generating gcda files
Working on /path/to/zephyr.git/sanity-out/mps2_an385/tests/kernel/mem_protect/stack_random/kernel.memory_protection.stack_random/handler.log
Generating gcda files
Working on /path/to/zephyr.git/sanity-out/mps2_an385/tests/kernel/mem_protect/syscalls/kernel.memory_protection.syscalls/handler.log
Generating gcda files
Working on /path/to/zephyr.git/sanity-out/mps2_an385/tests/kernel/mem_protect/protection/kernel.memory_protection.protection/handler.log
Generating gcda files
Working on /path/to/zephyr.git/sanity-out/mps2_an385/tests/kernel/mem_protect/mem_protect/kernel.memory_protection/handler.log
Generating gcda files
Working on /path/to/zephyr.git/sanity-out/mps2_an385/tests/kernel/mem_protect/obj_validation/kernel.memory_protection.obj_validation/handler.log
Generating gcda files
Working on /path/to/zephyr.git/sanity-out/mps2_an385/tests/kernel/mem_protect/userspace/kernel.memory_protection.userspace/handler.log
Generating gcda files
Working on /path/to/zephyr.git/sanity-out/mps2_an385/tests/kernel/context/kernel.common/handler.log
Generating gcda files
Working on /path/to/zephyr.git/sanity-out/mps2_an385/tests/kernel/arm_irq_vector_table/arch.interrupt/handler.log
Generating gcda files
Working on /path/to/zephyr.git/sanity-out/mps2_an385/tests/kernel/profiling/profiling_api/kernel.profiling/handler.log
Generating gcda files
Working on /path/to/zephyr.git/sanity-out/mps2_an385/tests/kernel/common/kernel.common/handler.log
Generating gcda files
Working on /path/to/zephyr.git/sanity-out/mps2_an385/tests/kernel/mutex/mutex_api/kernel.mutex/handler.log
Generating gcda files
Working on /path/to/zephyr.git/sanity-out/mps2_an385/tests/kernel/mutex/mutex/kernel.mutex/handler.log
Generating gcda files
Working on /path/to/zephyr.git/sanity-out/mps2_an385/tests/subsys/jwt/libraries.encoding/handler.log
Generating gcda files
Working on /path/to/zephyr.git/sanity-out/mps2_an385/tests/subsys/logging/log_msg/logging.log_msg/handler.log
Generating gcda files
Working on /path/to/zephyr.git/sanity-out/mps2_an385/tests/subsys/logging/log_list/logging.log_list/handler.log
Generating gcda files
Working on /path/to/zephyr.git/sanity-out/mps2_an385/tests/subsys/logging/log_output/logging.log_output/handler.log
Generating gcda files
Working on /path/to/zephyr.git/sanity-out/mps2_an385/tests/subsys/logging/log_core/logging.log_core/handler.log
Generating gcda files
HTML report generated: /path/to/zephyr.git/sanity-out/coverage/index.html
8 of 73 tests passed with 0 warnings in 969 seconds

Environment (please complete the following information):

• OS: fedora29
• Toolchain (e.g Zephyr SDK, ...): zephyr
• Commit SHA or Version used: a907e69

[wentongwu]

run the case(e.g. kernel.common) manually, it works well. It should be someone's commit cause this issue. I'm checking this issue.

[wentongwu]

can reproduce the issue(tests/kernel/workq/work_queue_api case) when CONFIG_COVERAGE=y. And if CONFIG_COVERAGE=n, the issue won't happen. and the output is that:
qemu-system-arm: warning: nic lan9118.0 has no peer
***** MPU FAULT *****
Data Access Violation
MMFAR Address: 0x20018014
***** Hardware exception *****
Current thread ID = 0x20014c58
Faulting instruction address = 0x20fe0
Fatal fault in thread 0x20014c58! Aborting.

check the zephyr.lst file, and address 0x20018014 is in sys_work_q_stack and 0x20014c58 is the thread ID of k_sys_work_q.thread.
And also check arm mpu code, the GCOV section is configured as a mpu region with K_MEM_PARTITION_P_RW_U_RW(I think it's read and write) attribute. It seems there will be MPU fault if add the coverage code(-ftest-coverage -fprofile-arcs).
@andrewboie could you please give some ideas ? Thanks

[andrewboie]

@wentongwu I don't understand well what the compiler is doing when profiling is enabled. I saw your email earlier this morning but I haven't had a chance to analyze it yet.

What code is at address 0x20fe0 which is accessing this data?
Nominally, the thread is supposed to be able to access its own stack buffer, so it's not clear to me why we would be faulting in this case.

[wentongwu]

from below, it seems 0x20fe4 - 0x20ff0 is some code added by compiler for gcov. And it seems 0x20fe0 is doing stack pushing operation. I'm not sure if there is stack overflow. @andrewboie
00020fd8 <z_remove_thread_from_ready_q>:
{
20fd8: e92d 4ff0 stmdb sp!, {r4, r5, r6, r7, r8, r9, sl, fp, lr}
20fdc: b0ad sub sp, #180 ; 0xb4
20fde: af00 add r7, sp, #0
20fe0: f8c7 0084 str.w r0, [r7, #132] ; 0x84
20fe4: 4b4c ldr r3, [pc, #304] ; (21118 <z_remove_thread_from_ready_q+0x140>)
20fe6: cb18 ldmia r3, {r3, r4}
20fe8: 1c59 adds r1, r3, #1
20fea: f144 0200 adc.w r2, r4, #0
20fee: 4b4a ldr r3, [pc, #296] ; (21118 <z_remove_thread_from_ready_q+0x140>)
20ff0: e9c3 1200 strd r1, r2, [r3]
LOCKED(&sched_spinlock) {
20ff4: 2300 movs r3, #0
20ff6: f8c7 3098 str.w r3, [r7, #152] ; 0x98
20ffa: 4b48 ldr r3, [pc, #288] ; (2111c <z_remove_thread_from_ready_q+0x144>)
20ffc: f8c7 30ac str.w r3, [r7, #172] ; 0xac
asm volatile(

[andrewboie]

Can you backtrace to where in the source code this is coming from?
Try qemu_x86, it prints a backtrace and also has memory protection enabled.

[wentongwu]

@andrewboie I will do that. And I increased system workq's stack size, system workq thread will not be mpu fault, another thread trigger's mpu fault, I'm trying to increase this thread's stack size. And could you please give me some ideas about the strategies for zephyr's memory protection? Thanks.

[wentongwu]

do the same test on qemu_x86, but it's Stack Check Fail as #14499 instead of MPU fault. And for mps2_an385, enlarge the stack of thread, some not passed tests can pass, and the others still MPU fault.

[wentongwu]

go back to old version 9072d34, with 0.9.5 SDK the test can pass, but with 0.10.0 SDK, still MPU fault. both tests are set -DCONFIG_COVERAGE=y .

[wentongwu]

with latest code version and SDK 0.9.5, the test can also pass with DCONFIG_COVERAGE=y.

before the test, I apply below patch:
diff --git a/cmake/toolchain/zephyr/host-tools.cmake b/cmake/toolchain/zephyr/host-tools.cmake
index 21417fb858..e9d8a71ecb 100644
--- a/cmake/toolchain/zephyr/host-tools.cmake
+++ b/cmake/toolchain/zephyr/host-tools.cmake
@@ -15,7 +15,7 @@ if(NOT ZEPHYR_SDK_INSTALL_DIR)
return()
endif()

-set(REQUIRED_SDK_VER 0.10.0)
+set(REQUIRED_SDK_VER 0.9.5)
set(TOOLCHAIN_VENDOR zephyr)
set(TOOLCHAIN_ARCH x86_64)

[cinlyooi-intel]

@wentongwu Confirmed if switch to 0.9.5, sanitycheck --coverage works. If not, I have a lot of handler crashs and timeouts.

[andrewboie]

Even if the errors are different between the two targets, I have a suspicion that this is a stack overflow in both cases. The new GCC may be using a lot more stack space when profiling is enabled in the 0.10 SDK.

Some things to investigate:

• What is the offset from the beginning of sys_work_q_stack where the MPU fault occurred? If it is within the first 32 bytes, on ARM the first 32 bytes of the stack buffer are reserved for a small MPU region which detects stack overflows in supervisor mode.
• What are the characteristics of the MPU region that is causing this access to fault?

Stack overflows are detected via MPU faults on ARM. The code is supposed to do some checks to see if an MPU fault is due to a stack overflow instead of some other reason, and report Stack Check Fail, but it might not be working correctly.

[wentongwu]

thanks @andrewboie.
MMFAR Address: 0x20018014 and sys_work_q_stack: 0x0000000020018000, so the address of the location that generated MPU fault located in the first 32bytes of this thread’ stack. I'm reviewing zephyr's mpu code for arm, try to finding the attribute of this MPU region.

by the way, as below the new compiler will consume more stack space when doing function call.
new compiler:
00020fd8 <z_remove_thread_from_ready_q>:
{
20fd8: e92d 4ff0 stmdb sp!, {r4, r5, r6, r7, r8, r9, sl, fp, lr}
20fdc: b0ad sub sp, #180 ; 0xb4
20fde: af00 add r7, sp, #0
20fe0: f8c7 0084 str.w r0, [r7, #132] ; 0x84

old compiler:
0001c350 <z_remove_thread_from_ready_q>:
{
1c350: b590 push {r4, r7, lr}
1c352: b08d sub sp, #52 ; 0x34
1c354: af00 add r7, sp, #0
1c356: 6078 str r0, [r7, #4]