[BUG] firmware crashed when mulitple-pipline on MTLP platforms

[miRoox]

Describe the bug
Ci observed firmware crashed when mulitple-pipline on MTLP_RVP_NOCODEC and MTLP_RVP_HDA.

To Reproduce

TPLG=/lib/firmware/intel/sof-ace-tplg/sof-mtl-nocodec.tplg MODEL=MTLP_RVP_NOCODEC ~/sof-test/test-case/multiple-pipeline.sh -f c -l 50

Reproduction Rate
Twice in CI. Hard to reproduce manually. Need further investigation

Environment

1. Branch name and commit hash of the 2 repositories: sof (firmware/topology) and linux (kernel driver).
   • Kernel: thesofproject/linux@cff45eeb2ecb
   • SOF: 6b91a553284b
     Daily test run 21264
2. Name of the topology file
   • Topology: sof-ace-tplg/sof-mtl-nocodec.tplg, sof-ace-tplg/sof-hda-generic-4ch.tplg
3. Name of the platform(s) on which the bug is observed.
   • Platform: MTLP_RVP_NOCODEC, MTLP_RVP_HDA

Screenshots or console output

mtrace

[00:01:30.938,913] <wrn> dai_comp: comp:3 0x40006 dai_copy(): nothing to copy
[00:01:30.939,825] <inf> host_comp: host_get_copy_bytes_normal: comp:4 0x40007 no bytes to copy, 0 avail in buffer, 768 free in DMA
[00:01:30.939,838] <err> os:  ** FATAL EXCEPTION
[00:01:30.939,845] <err> os:  ** CPU 0 EXCCAUSE 13 (load/store PIF data error)
[00:01:30.939,850] <err> os:  **  PC 0xa004397f VADDR (nil)
[00:01:30.939,853] <err> os:  **  PS 0x60820
[00:01:30.939,863] <err> os:  **    (INTLEVEL:0 EXCM: 0 UM:1 RING:0 WOE:1 OWB:8 CALLINC:2)
[Terminated

dmesg

[ 6182.194370] kernel: snd_sof:sof_ipc4_log_header: sof-audio-pci-intel-mtl 0000:00:1f.3: ipc tx      : 0x11050002|0x0: GLB_CREATE_PIPELINE
[ 6182.619285] kernel: snd_sof:sof_pcm_trigger: sof-audio-pci-intel-mtl 0000:00:1f.3: pcm: trigger stream 0 dir 1 cmd 0
[ 6182.619321] kernel: snd_sof:sof_ipc4_trigger_pipelines: sof-audio-pci-intel-mtl 0000:00:1f.3: trigger cmd: 0 state: 3
[ 6182.619346] kernel: snd_sof:sof_ipc4_set_pipeline_state: sof-audio-pci-intel-mtl 0000:00:1f.3: ipc4 set pipeline 1 state 3
[ 6182.635333] kernel: snd_sof:sof_pcm_trigger: sof-audio-pci-intel-mtl 0000:00:1f.3: pcm: trigger stream 12 dir 1 cmd 0
[ 6182.635369] kernel: snd_sof:sof_ipc4_trigger_pipelines: sof-audio-pci-intel-mtl 0000:00:1f.3: trigger cmd: 0 state: 3
[ 6182.674585] kernel: snd_sof:sof_pcm_trigger: sof-audio-pci-intel-mtl 0000:00:1f.3: pcm: trigger stream 1 dir 1 cmd 0
[ 6182.674622] kernel: snd_sof:sof_ipc4_trigger_pipelines: sof-audio-pci-intel-mtl 0000:00:1f.3: trigger cmd: 0 state: 3
[ 6182.698427] kernel: sof-audio-pci-intel-mtl 0000:00:1f.3: ipc timed out for 0x11050002|0x0
[ 6182.698484] kernel: sof-audio-pci-intel-mtl 0000:00:1f.3: preventing DSP entering D3 state to preserve context
[ 6182.698496] kernel: sof-audio-pci-intel-mtl 0000:00:1f.3: ------------[ IPC dump start ]------------
[ 6182.698520] kernel: sof-audio-pci-intel-mtl 0000:00:1f.3: Host IPC initiator: 0x91050002|0x0|0x0, target: 0x33000000|0x1|0x0, ctl: 0x3
[ 6182.698540] kernel: sof-audio-pci-intel-mtl 0000:00:1f.3: ------------[ IPC dump end ]------------
[ 6182.698554] kernel: sof-audio-pci-intel-mtl 0000:00:1f.3: ------------[ DSP dump start ]------------
[ 6182.698566] kernel: sof-audio-pci-intel-mtl 0000:00:1f.3: IPC timeout
[ 6182.698579] kernel: sof-audio-pci-intel-mtl 0000:00:1f.3: fw_state: SOF_FW_BOOT_COMPLETE (7)
[ 6182.698599] kernel: sof-audio-pci-intel-mtl 0000:00:1f.3: ROM status: 0x5, ROM error: 0x0
[ 6182.698613] kernel: sof-audio-pci-intel-mtl 0000:00:1f.3: ROM debug status: 0x50000005, ROM debug error: 0x0
[ 6182.698628] kernel: sof-audio-pci-intel-mtl 0000:00:1f.3: ROM feature bit enabled
[ 6182.698640] kernel: sof-audio-pci-intel-mtl 0000:00:1f.3: ------------[ DSP dump end ]------------
[ 6182.698697] kernel: sof-audio-pci-intel-mtl 0000:00:1f.3: failed to create module pipeline.12
[ 6182.698720] kernel: sof-audio-pci-intel-mtl 0000:00:1f.3: Failed to set up connected widgets
[ 6182.698749] kernel: sof-audio-pci-intel-mtl 0000:00:1f.3: error: failed widget list set up for pcm 2 dir 1
[ 6182.698767] kernel: sof-audio-pci-intel-mtl 0000:00:1f.3: ASoC: error at snd_soc_pcm_component_hw_params on 0000:00:1f.3: -110
[ 6182.698861] kernel:  Port2: ASoC: error at __soc_pcm_hw_params on Port2: -110
[ 6182.698917] kernel:  Port2: ASoC: error at dpcm_fe_dai_hw_params on Port2: -110

full mtrace.txt
full dmesg.txt

[kv2019i]

V2.5 branching this week and nobody assigned, so moving to v2.6.

[keqiaozhang]

Observed this issue on TGL-IPC4-NOCODEC platform also.
https://sof-ci.ostc.intel.com/#/result/planresultdetail/22994?model=TGLU_RVP_NOCODEC_IPC4ZPH&testcase=multiple-pause-resume-50

[keqiaozhang]

This issue no longer exists in CI, closing this bug.