Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] multiple-pipeline test fails with DSP EXCEPTION with multicore nocodec topology #8492

Closed
fredoh9 opened this issue Nov 17, 2023 · 3 comments
Closed

[BUG] multiple-pipeline test fails with DSP EXCEPTION with multicore nocodec topology #8492

fredoh9 opened this issue Nov 17, 2023 · 3 comments
Labels
ADL Applies to Alder Lake platform bug Something isn't working as expected I2S Applies to I2S bus for codec connection multicore Issues observed when not only core#0 is used. multicore-3cores Issues observed when 3 cores are used. P1 Blocker bugs or important features TGL Applies to Tiger Lake
Milestone
v2.9

Comments

@fredoh9
Copy link
Contributor

fredoh9 commented Nov 17, 2023 •
edited
Loading

Describe the bug
multiple-pipeline test failed with DSP Exception. dmesg has ipc error with MOD_SET_DX.
ADLP_RVP_NOCODEC and TGLU_RVP_NOCODEC has same/similar issue.

[  616.335232] kernel: snd_sof:sof_ipc4_log_header: sof-audio-pci-intel-tgl 0000:00:1f.3: ipc tx      : 0x47000000|0x0: MOD_SET_DX [data size: 8]
[  616.335728] kernel: snd_sof:sof_ipc4_log_header: sof-audio-pci-intel-tgl 0000:00:1f.3: ipc tx reply: 0x67000006|0x0: MOD_SET_DX
[  616.335730] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: FW reported error: 6 - Unknown error while processing the request
[  616.335869] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: ipc error for msg 0x47000000|0x0
[  616.335875] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: error: failed to disable target core: 2 for widget pipeline.6
[  616.335878] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: Failed to free connected widgets
[  616.335882] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: ASoC: error at snd_soc_pcm_component_hw_free on 0000:00:1f.3: -22

mtrace shows FATAL EXCEPTION

[   51.966658] <inf> ipc: ipc_cmd: rx	: 0x47000000|0x0
[   51.966756] <err> ipc: ipc4_module_process_dx: failed to disable core 2
[   51.966766] <err> ipc: ipc_cmd: ipc4: MODULE_MSG failed with err 6
ASSERTION FAIL [k_sched_current_thread_query()->base.sched_locked != 0U] @ /srv/home/jenkins/workspace/sof_config_build@2/zephyr/kernel/sched.c:1117
	
[   51.967073] <err> os: xtensa_excint1_c:  ** FATAL EXCEPTION
ASSERTION FAIL [z_spin_lock_valid(l)] @ /srv/home/jenkins/workspace/sof_config_build@2/zephyr/include/zephyr/spinlock.h:136
[   51.967153] <err> os: xtensa_excint1_c:  **    (INTLEVEL:0 EXCM: 0 UM:1 RING:0 WOE:1 OWB:4 CALLINC:2)
[   51.967373] <err> os: xtensa_excint1_c:  **    (INTLEVEL:0 EXCM: 0 UM:1 RING:0 WOE:1 OWB:10 CALLINC:2)
[   51.967585] <err> os: xtensa_excint1_c:  **    (INTLEVEL:0 EXCM: 0 UM:1 RING:0 WOE:1 OWB:0 CALLINC:2)
[   51.967828] <err> os: xtensa_excint1_c:  **    (INTLEVEL:0 EXCM: 0 UM:1 RING:0 WOE:1 OWB:6 CALLINC:2)
[   51.967985] <err> os: xtensa_excint1_c:  **    (INTLEVEL:0 EXCM: 0 UM:1 RING:0 WOE:1 OWB:12 CALLINC:2)
[   51.968188] <err> os: xtensa_excint1_c:  **    (INTLEVEL:0 EXCM: 0 UM:1 RING:0 WOE:1 OWB:2 CALLINC:2)
[   51.968425] <err> os: xtensa_excint1_c:  **    (INTLEVEL:0 EXCM: 0 UM:1 RING:0 WOE:1 OWB:8 CALLINC:2)
	Invalid spinlock 0x9e0b8518
	Invalid spinlock 0x9e0b8518
	Invalid spinlock 0x9e0b8518
	Invalid spinlock 0x9e0b8518
ASSERTION FAIL [z_spin_lock_valid(l)] @ /srv/home/jenkins/workspace/sof_config_build@2/zephyr/include/zephyr/spinlock.h:136
ASSERTION FAIL [z_spin_lock_valid(l)] @ /srv/home/jenkins/workspace/sof_config_build@2/zephyr/include/zephyr/spinlock.h:136
ASSERTION FAIL [z_spin_lock_valid(l)] @ /srv/home/jenkins/workspace/sof_config_build@2/zephyr/include/zephyr/spinlock.h:136

To Reproduce
TPLG=/lib/firmware/intel/development/sof-adl-nocodec.tplg MODEL=ADLP_RVP_NOCODEC-ipc4 SOF_TEST_INTERVAL=5 ~/sof-test/test-case/multiple-pipeline.sh -f a -c 20 -l 50

Reproduction Rate
Need some iteration but 100%

Environment
First broken daily build,

  • SOF Build Result: 1379
  • SOF Branch: main
  • SOF Commit: 9eae757
  • Zephyr Commit: 26002b060708

The TPLG has multicore support.
sof-adl-nocodec-multicore

Last good daily build,

  • SOF Build Result: 1366
  • SOF Branch: main
  • SOF Commit: b7d7fb0
  • Zephyr Commit: 26002b060708

The TPLG doesn't have multicore support.
sof-adl-nocodec

NOTE: If I just replace the tplg from good build, this is not reproducible.
This PR #8240 makes multicore part of default nocodec topology and causes the problem.

Screenshots or console output

dmesg-adl-fw-error6.txt
mtrace-adl-dsp-exception.txt
@fredoh9 fredoh9 added bug Something isn't working as expected P1 Blocker bugs or important features I2S Applies to I2S bus for codec connection TGL Applies to Tiger Lake multicore Issues observed when not only core#0 is used. ADL Applies to Alder Lake platform multicore-3cores Issues observed when 3 cores are used. labels Nov 17, 2023
@marc-hb
Copy link
Collaborator

marc-hb commented Nov 18, 2023

Another multiple-pipeline failure, dunno whether it's related but it's a very recent regression too: https://sof-ci.01.org/sofpr/PR8494/build14998/devicetest/index.html?model=ADLP_RVP_NOCODEC_IPC4ZPH&testcase=multiple-pipeline-all (#8494)

@fredoh9
Copy link
Contributor Author

fredoh9 commented Nov 18, 2023

multiple-pipeline-playback-50 test has similar but a bit different logs.

dmesg

[ 4227.456412] kernel: snd_sof:sof_ipc4_log_header: sof-audio-pci-intel-tgl 0000:00:1f.3: ipc tx      : 0x47000000|0x0: MOD_SET_DX [data size: 8]
[ 4227.456934] kernel: snd_sof:sof_ipc4_log_header: sof-audio-pci-intel-tgl 0000:00:1f.3: ipc tx reply: 0x67000006|0x0: MOD_SET_DX
[ 4227.456938] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: FW reported error: 6 - Unknown error while processing the request
[ 4227.457151] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: ipc error for msg 0x47000000|0x0
[ 4227.457156] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: error: failed to disable target core: 2 for widget pipeline.6
[ 4227.457158] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: Failed to free connected widgets
[ 4227.457163] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: ASoC: error at snd_soc_pcm_component_hw_free on 0000:00:1f.3: -22
[ 4227.457187] kernel: snd_sof:sof_pcm_hw_free: sof-audio-pci-intel-tgl 0000:00:1f.3: pcm: free stream 1 dir 0
[ 4227.457195] kernel: snd_sof_intel_hda_common:hda_dsp_stream_trigger: sof-audio-pci-intel-tgl 0000:00:1f.3: FW Poll Status: reg[0x1c0]=0x20440000 successful
[ 4227.457204] kernel: snd_sof:sof_ipc4_trigger_pipelines: sof-audio-pci-intel-tgl 0000:00:1f.3: trigger cmd: 0 state: 2
[ 4227.457208] kernel: snd_sof:sof_ipc4_log_header: sof-audio-pci-intel-tgl 0000:00:1f.3: ipc tx      : 0x13000002|0x1: GLB_SET_PIPELINE_STATE [data size: 12]
[ 4227.958422] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: ipc timed out for 0x13000002|0x1

mtrace has ASSERTION FAIL

[   62.122463] <wrn> copier: copier_comp_trigger: comp:2 0x20004 dai is not ready
[   62.122585] <inf> src: src_reset: comp:2 0x8 src_reset()
[   62.123711] <inf> ipc: ipc_cmd: rx	: 0x46010002|0x10003
[   62.124348] <inf> ipc: ipc_cmd: rx	: 0x12020000|0x0
[   62.124416] <inf> dma: dma_put: dma_put(), dma = 0x9e09f250, sref = 1
[   62.124636] <inf> src: src_free: comp:2 0x8 src_free()
[   62.125286] <inf> ipc: ipc_cmd: rx	: 0x12030000|0x0
[   62.125596] <inf> dma: dma_put: dma_put(), dma = 0x9e09f1b0, sref = 0
[   62.125613] <inf> dai_intel_ssp: dai_ssp_pm_runtime_dis_ssp_power: dai_ssp_pm_runtime_dis_ssp_power index 2
[   62.125755] <err> dai_intel_ssp: dai_ssp_po����������������Çÿÿÿ����	:�¾8—�¾”Î�¾Ø—�¾Ø—�¾¸—�¾����ay poll Ø—�¾¸—�¾��������������������”Î�¾��������è–�¾%U�¾x—�¾”Î�¾Ø—�¾”Î�¾Ø—�¾¸—�¾����Ø—�¾¸—�¾����p_pm¸—�¾����ÇÿÿÿP%
žX�¾¨—�¾”Î�¾����pm_runtime_dis_ssp_power warning: timeout
[   62.125778] <inf> dai_intel_ssp: dai_ssp_pm_runtime_dis_ssp_power: dai_ssp_pm_runtime_dis_ssp_power I2SLCTL
[   62.126203] <inf> ipc: ipc_cmd: rx	: 0x47000000|0x0
[   62.126355] <err> ipc: ipc4_module_process_dx: failed to disable core 2
[   62.126370] <err> ipc: ipc_cmd: ipc4: MODULE_MSG failed with err 6
ASSERTION FAIL [k_sched_current_thread_query()->base.sched_locked != 0U] @ /srv/home/jenkins/workspace/sof_config_build@2/zephyr/kernel/sched.c:1117
[   62.126713] <err> os: z_xtensa_dump_stack:  **  A0 0xbe0154f8  SP 0xbe0b20f0  A2 0x4  A3 0xbe0b2100
[   62.126963] <err> os: z_xtensa_dump_stack:  **  A0 0xbe0154f8  SP 0xbe0b3ba0  A2 0x4  A3 0xbe0b3bb0
[   62.127201] <err> os: z_xtensa_dump_stack:  **  A0 0xbe0154f8  SP 0xbe0b3938  A2 0x4  A3 0xbe0b3948
[   62.127411] <err> os: z_xtensa_dump_stack:  **  A0 0xbe0154f8  SP 0xbe0b36d0  A2 0x4  A3 0xbe0b36e0
[   62.127623] <err> os: z_xtensa_dump_stack:  **  A0 0xbe0154f8  SP 0xbe0b3468  A2 0x4  A3 0xbe0b3478
[   62.127833] <err> os: z_xtensa_dump_stack:  **  A0 0xbe0154f8  SP 0xbe0b3200  A2 0x4  A3 0xbe0b3210
[   62.128086] <err> os: z_xtensa_dump_stack:  **  A0 0xbe0154f8  SP 0xbe0b2f98  A2 0x4  A3 0xbe0b2fa8
	Invalid spinlock 0x9e0b8518
	Invalid spinlock 0x9e0b8518
	Invalid spinlock 0x9e0b8518
	Invalid spinlock 0x9e0b8518

Intel Internal Test link:
planresultdetail/34750?model=ADLP_RVP_NOCODEC-ipc4&testcase=multiple-pipeline-playback-50

@RanderWang RanderWang mentioned this issue Nov 20, 2023
Merged
RanderWang added a commit to RanderWang/sof that referenced this issue Nov 23, 2023
@RanderWang
west.yml: update Zephyr to efc32081893d
1e793e8 
Update Zephyr to bring in following Zephyr commit:
efc32081893d soc: intel_adsp: cavs: mask idc interrupt
before halting cpu

Link: thesofproject#8492

Signed-off-by: Rander Wang <rander.wang@intel.com>
@RanderWang RanderWang mentioned this issue Nov 23, 2023
Merged
@lgirdwood lgirdwood added this to the v2.9 milestone Nov 23, 2023
lgirdwood pushed a commit that referenced this issue Nov 24, 2023
@RanderWang @lgirdwood
west.yml: update Zephyr to efc32081893d
f1ee910 
Update Zephyr to bring in following Zephyr commit:
efc32081893d soc: intel_adsp: cavs: mask idc interrupt
before halting cpu

Link: #8492

Signed-off-by: Rander Wang <rander.wang@intel.com>
@fredoh9
Copy link
Contributor Author

fredoh9 commented Nov 27, 2023

working fine with the Zephyr fix and west update. Closing now

@fredoh9 fredoh9 closed this as completed Nov 27, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
ADL Applies to Alder Lake platform bug Something isn't working as expected I2S Applies to I2S bus for codec connection multicore Issues observed when not only core#0 is used. multicore-3cores Issues observed when 3 cores are used. P1 Blocker bugs or important features TGL Applies to Tiger Lake
Projects
None yet
Milestone
v2.9
Development

No branches or pull requests
3 participants
@lgirdwood @fredoh9 @marc-hb