Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

PCI Passthrough : Starting the guest with CX5 adapter leads to Call Trace EEH #15

Closed
MalleshKoti opened this issue Sep 20, 2017 · 2 comments
Closed

PCI Passthrough : Starting the guest with CX5 adapter leads to Call Trace EEH #15

MalleshKoti opened this issue Sep 20, 2017 · 2 comments

Comments

@MalleshKoti
Copy link

MalleshKoti commented Sep 20, 2017 •
edited by cdeadmin
Loading

Guest failed to start with CX5 passthrough, same time host hit with EEH :

[ 1315.195786] mlx5_0:wait_for_async_commands:732:(pid 3719): done with all pending requests
[ 1317.267625] mlx5_1:wait_for_async_commands:732:(pid 3719): done with all pending requests
[ 1317.762382] EEH: PHB#0 failure detected, location: N/A
[ 1317.762435] CPU: 0 PID: 91 Comm: kworker/0:1 Not tainted 4.13.0-4.dev.git49564cb.el7.centos.ppc64le #1
[ 1317.762590] Workqueue: events work_for_cpu_fn
[ 1317.762710] Call Trace:
[ 1317.762764] [c000003fe7f87930] [c000000000ac906c] dump_stack+0xb0/0xf4 (unreliable)
[ 1317.762889] [c000003fe7f87970] [c00000000003b3b0] eeh_dev_check_failure+0x1f0/0x590
[ 1317.763016] [c000003fe7f87a10] [c0000000000a1204] pnv_pci_read_config+0xc4/0x140
[ 1317.763136] [c000003fe7f87a50] [c0000000005ccff4] pci_bus_read_config_word+0xb4/0x100
[ 1317.763265] [c000003fe7f87ab0] [c0000000005d653c] pci_raw_set_power_state+0xfc/0x280
[ 1317.763388] [c000003fe7f87b40] [c0000000005da310] pci_set_power_state+0xd0/0x1d0
[ 1317.763518] [c000003fe7f87b80] [c000000000785e2c] vfio_pci_probe+0x13c/0x250
[ 1317.763634] [c000003fe7f87bd0] [c0000000005dfe3c] local_pci_probe+0x6c/0x130
[ 1317.763751] [c000003fe7f87c60] [c0000000001169e8] work_for_cpu_fn+0x38/0x60
[ 1317.763862] [c000003fe7f87c90] [c00000000011bc00] process_one_work+0x1a0/0x490
[ 1317.763985] [c000003fe7f87d30] [c00000000011c168] worker_thread+0x278/0x520
[ 1317.764089] [c000003fe7f87dc0] [c0000000001244a8] kthread+0x168/0x1b0
[ 1317.764201] [c000003fe7f87e30] [c00000000000bc60] ret_from_kernel_thread+0x5c/0x7c
[ 1317.764320] EEH: Detected error on PHB#0
[ 1317.764363] EEH: This PCI device has failed 1 times in the last hour
[ 1317.764516] EEH: Notify device drivers to shutdown
[ 1317.764615] EEH: Collect temporary log
[ 1317.764681] PHB4 PHB#0 Diag-data (Version: 1)
[ 1317.764762] brdgCtl: 0000ffff
[ 1317.764821] RootSts: ffffffff ffffffff ffffffff ffffffff 0000ffff
[ 1317.764924] RootErrSts: ffffffff ffffffff ffffffff
[ 1317.765009] RootErrLog: ffffffff ffffffff ffffffff ffffffff
[ 1317.765112] sourceId: ffffffff
[ 1317.765167] nFir: 0000800000000000 0030001c00000000 0000800000000000
[ 1317.765271] PhbSts: 0000001800000000 0000001800000000
[ 1317.765358] Lem: 0000000100300100 ffffffffffffffff 0000000000000000
[ 1317.765467] PhbErr: 00000c8000000000 0000008000000000 2148000098000240 a008400000000000
[ 1317.765598] RxeArbErr: 4000000020000000 4000000000000000 0050000400000000 0000000000000000
[ 1317.765727] RxeMrgErr: 0000000000000001 0000000000000001 0000000000000000 0000000000000000
[ 1317.765859] PblErr: 0000000001000000 0000000001000000 0000000000000000 0000000000000000
[ 1317.765991] PcieDlp: 0000000000000000 0000000000000000 0fd5000000000000
[ 1317.766099] RegbErr: 0040004e54200800 0040000000000000 62000a1018000000 1800000000000000
[ 1317.766233] EEH: Reset without hotplug activity
[ 1317.792802] vfio-pci 0000:01:00.0: Refused to change power state, currently in D3
[ 1317.792918] iommu: Removing device 0000:01:00.0 from group 0
[ 1317.822800] mlx5_core 0000:01:00.0: Refused to change power state, currently in D3
[ 1317.822903] mlx5_core 0000:01:00.0: Using 64-bit DMA iommu bypass
[ 1317.823786] mlx5_core 0000:01:00.0: firmware version: 65535.65535.65535
[ 1327.853266] mlx5_core 0000:01:00.0: Firmware over 10000 MS in pre-initializing state, aborting
[ 1327.853363] mlx5_core 0000:01:00.0: mlx5_load_one failed with error code -16
[ 1327.853588] mlx5_core: probe of 0000:01:00.0 failed with error -16
[ 1327.887193] Unable to handle kernel paging request for data at address 0x00000008
[ 1327.887349] Faulting instruction address: 0xc000000000ad6c98
[ 1327.887462] Oops: Kernel access of bad area, sig: 11 [#1]
[ 1327.887553] SMP NR_CPUS=1024
[ 1327.887554] NUMA
[ 1327.887610] PowerNV
[ 1327.887688] Modules linked in: xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 nf_conntrack_ipv6 nf_defrag_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack libcrc32c iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx5_ib ib_core at24 opal_prd ofpart ipmi_powernv ipmi_devintf powernv_flash ipmi_msghandler i2c_opal mtd kvm_hv nfsd auth_rpcgss oid_registry nfs_acl lockd kvm grace
[ 1327.888930] sunrpc ast i2c_algo_bit drm_kms_helper syscopyarea sysfillrect tg3 sysimgblt fb_sys_fops ttm drm mlx5_core i2c_core ptp pps_core
[ 1327.889168] CPU: 14 PID: 492 Comm: systemd-journal Not tainted 4.13.0-4.dev.git49564cb.el7.centos.ppc64le #1
[ 1327.889334] task: c000003fe7a14a00 task.stack: c000003fdf660000
[ 1327.889432] NIP: c000000000ad6c98 LR: c0000000003c63c8 CTR: c0000000003c4430
[ 1327.889551] REGS: c000003fdf663ac0 TRAP: 0300 Not tainted (4.13.0-4.dev.git49564cb.el7.centos.ppc64le)
[ 1327.889704] MSR: 9000000000009033 <SF,HV,EE,ME,IR,DR,RI,LE>
[ 1327.889718] CR: 28002828 XER: 00000000
[ 1327.889874] CFAR: c0000000003c63c4 DAR: 0000000000000008 DSISR: 40000000 SOFTE: 1
GPR00: c0000000003c6318 c000003fdf663d40 c000000001397a00 c00020397b4e9e00
GPR04: c000000002330368 c00020397552ded0 c00020397552de00 c00020397d455f00
GPR08: 0000000000000000 0000000000000000 c00020397d455f00 c000000000af0f48
GPR12: c0000000003c4430 c00000000fd88c00 000000007552de00 0000000000000000
GPR16: 0000000000000000 0000000000000019 0000000000000000 c00020397b4e9e58
GPR20: c00020397b4e9e18 0000000000000000 0000000000000000 c000000002330308
GPR24: c000000002330300 00000000de045700 c00020397552de30 c000003fde045700
GPR28: c00020397b4e9e00 c00020397552de00 c000000002330368 c00020397552ded0
[ 1327.891071] NIP [c000000000ad6c98] rb_insert_color+0x18/0x1a0
[ 1327.891183] LR [c0000000003c63c8] SyS_epoll_ctl+0x828/0xc10
[ 1327.891273] Call Trace:
[ 1327.891315] [c000003fdf663d40] [c0000000003c6318] SyS_epoll_ctl+0x778/0xc10 (unreliable)
[ 1327.891454] [c000003fdf663e30] [c00000000000b8e0] system_call+0x58/0x6c
[ 1327.891565] Instruction dump:
[ 1327.891623] e8410018 7fc9f378 7fbeeb78 4bffff9c 60000000 60420000 e9430000 2faa0000
[ 1327.891756] 419e0154 e92a0000 792807e1 4c820020 61270001 7d254b78 7faa4040
[ 1327.891897] ---[ end trace 6efc2b1ce3f4f85b ]---

Aafter this adapter is not recovered.. and we do not see the adapter in host 'lspci' until we reboot the host.

Steps to Reproduce:

  1. Edit the guest in shutdown mode and add CX5 as hostdev device :
  1. Start the guest, it fails to boot and Call Trace is seen on host where we hit EEH

Here, we are using in-box driver as MOFED is not present yet in HostOS.

Attaching lspci -vvv before starting guest:
lspci_vvv.txt

cde:info Mirrored with LTC bug #158978 </cde:info>
@MalleshKoti
Copy link
Author

XML used in the guest for hostdev device:

      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x01' function='0x0' multifunction='on'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x01' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x01' function='0x1'/>
    </hostdev>

@cdeadmin
Copy link

------- Comment (attachment only) From magadagi@in.ibm.com 2017-09-20 07:23:55 EDT-------

aik pushed a commit that referenced this issue Sep 22, 2017
@JeffyCN @atseanpaul
drm/rockchip: Fix suspend crash when drm is not bound
0fa375e 
Currently we are allocating drm_device in rockchip_drm_bind, so if the
suspend/resume code access it when drm is not bound, we would hit this
crash:

[  253.402836] Unable to handle kernel NULL pointer dereference at virtual address 00000028
[  253.402837] pgd = ffffffc06c9b0000
[  253.402841] [00000028] *pgd=0000000000000000, *pud=0000000000000000
[  253.402844] Internal error: Oops: 96000005 [#1] PREEMPT SMP
[  253.402859] Modules linked in: btusb btrtl btbcm btintel bluetooth ath10k_pci ath10k_core ar10k_ath ar10k_mac80211 cfg80211 ip6table_filter asix usbnet mii
[  253.402864] CPU: 4 PID: 1331 Comm: cat Not tainted 4.4.70 #15
[  253.402865] Hardware name: Google Scarlet (DT)
[  253.402867] task: ffffffc076c0ce00 ti: ffffffc06c2c8000 task.ti: ffffffc06c2c8000
[  253.402871] PC is at rockchip_drm_sys_suspend+0x20/0x5c

Add sanity checks to prevent that.

Reported-by: Brian Norris <briannorris@chromium.com>
Signed-off-by: Jeffy Chen <jeffy.chen@rock-chips.com>
Signed-off-by: Sean Paul <seanpaul@chromium.org>
Link: https://patchwork.kernel.org/patch/9890297/
malcolmcrossley pushed a commit to malcolmcrossley/linux that referenced this issue Jan 24, 2018
@svaidy @gregkh
cpuidle: powernv: Pass correct drv->cpumask for registration
9e17713 
[ Upstream commit 293d264 ]

drv->cpumask defaults to cpu_possible_mask in __cpuidle_driver_init().
On PowerNV platform cpu_present could be less than cpu_possible in cases
where firmware detects the cpu, but it is not available to the OS.  When
CONFIG_HOTPLUG_CPU=n, such cpus are not hotplugable at runtime and hence
we skip creating cpu_device.

This breaks cpuidle on powernv where register_cpu() is not called for
cpus in cpu_possible_mask that cannot be hot-added at runtime.

Trying cpuidle_register_device() on cpu without cpu_device will cause
crash like this:

cpu 0xf: Vector: 380 (Data SLB Access) at [c000000ff1503490]
    pc: c00000000022c8bc: string+0x34/0x60
    lr: c00000000022ed78: vsnprintf+0x284/0x42c
    sp: c000000ff1503710
   msr: 9000000000009033
   dar: 6000000060000000
  current = 0xc000000ff1480000
  paca    = 0xc00000000fe82d00   softe: 0        irq_happened: 0x01
    pid   = 1, comm = swapper/8
Linux version 4.11.0-rc2 (sv@sagarika) (gcc version 4.9.4
(Buildroot 2017.02-00004-gc28573e) ) open-power-host-os#15 SMP Fri Mar 17 19:32:02 IST 2017
enter ? for help
[link register   ] c00000000022ed78 vsnprintf+0x284/0x42c
[c000000ff1503710] c00000000022ebb8 vsnprintf+0xc4/0x42c (unreliable)
[c000000ff1503800] c00000000022ef40 vscnprintf+0x20/0x44
[c000000ff1503830] c0000000000ab61c vprintk_emit+0x94/0x2cc
[c000000ff15038a0] c0000000000acc9c vprintk_func+0x60/0x74
[c000000ff15038c0] c000000000619694 printk+0x38/0x4c
[c000000ff15038e0] c000000000224950 kobject_get+0x40/0x60
[c000000ff1503950] c00000000022507c kobject_add_internal+0x60/0x2c4
[c000000ff15039e0] c000000000225350 kobject_init_and_add+0x70/0x78
[c000000ff1503a60] c00000000053c288 cpuidle_add_sysfs+0x9c/0xe0
[c000000ff1503ae0] c00000000053aeac cpuidle_register_device+0xd4/0x12c
[c000000ff1503b30] c00000000053b108 cpuidle_register+0x98/0xcc
[c000000ff1503bc0] c00000000085eaf0 powernv_processor_idle_init+0x140/0x1e0
[c000000ff1503c60] c00000000000cd60 do_one_initcall+0xc0/0x15c
[c000000ff1503d20] c000000000833e84 kernel_init_freeable+0x1a0/0x25c
[c000000ff1503dc0] c00000000000d478 kernel_init+0x24/0x12c
[c000000ff1503e30] c00000000000b564 ret_from_kernel_thread+0x5c/0x78

This patch fixes the bug by passing correct cpumask from
powernv-cpuidle driver.

Signed-off-by: Vaidyanathan Srinivasan <svaidy@linux.vnet.ibm.com>
Reviewed-by: Gautham R. Shenoy <ego@linux.vnet.ibm.com>
Acked-by: Michael Ellerman <mpe@ellerman.id.au>
[ rjw: Comment massage ]
Signed-off-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>

Signed-off-by: Sasha Levin <alexander.levin@verizon.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
@cdeadmin cdeadmin closed this as completed Apr 3, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@MalleshKoti @cdeadmin