On , Jul 22, 2020 at 02:44:14PM +0200, Alan wrote:
> On Wed, Jul 22, 2020 at 02:44:14PM +0200, Greg KH wrote:
> > On Wed, Jul 22, 2020 at 07:57:48PM +0800, WeitaoWangoc wrote:
> > > This bug is found in Zhaoxin platform, but it's a commom code bug.
> > > Fail sequence:
> > > step1: Unbind UHCI controller from native driver;
> > > step2: Bind UHCI controller to vfio-pci, which will put UHCI controller in one
> vfio
> > > group's device list and set UHCI's dev->driver_data to struct
> vfio-pci(for UHCI)
> >
> > Hah, that works? How do you do that properly? What code does that?
>
> Yeah, that can't possibly work. The USB core expects that any host
> controller device (or at least, any PCI host controller device) has its
> driver_data set to point to a struct usb_hcd. It doesn't expect a host
> controller to be bound to anything other than a host controller driver.
>
> Things could easily go very wrong here. For example, suppose at this
> point the ehci-hcd driver just happens to bind to the EHCI controller.
> When this happens, the EHCI controller hardware takes over all the USB
> connections that were routed to the UHCI controller. How will vfio-pci
> deal with that? Pretty ungracefully, I imagine.
>
> The only way to make this work at all is to unbind both uhci-hcd and
> ehci-hcd first. Then after both are finished you can safely bind
> vfio-pci to the EHCI controller and the UHCI controllers (in that
> order).
>
I'm agree with you, unbind both uhci-hcd and ehci-hcd first then bind to
vfio-pci is a more reasonable sequence. Our experiments prove that this
sequence is indeed good as expected.
However, I did not find a formal document to prescribe this order.
Unfortunately, some application software such as virt-manager/qemu assign
UHCI/EHCI to guest OS has the same bind/unbind sequence as test “by hand”.
Do we need to consider compatibility with this application scenario?
The following log is captured when starting then shutdown the
virtual machine.
/* starting virtual machine*/
[ 2785.250001] audit: type=1400 audit(1594375837.191:48): apparmor="STATUS" operation="profile_load" profile="unconfined" name="libvirt-fa674e73-67a2-4672-8524-e62dea8a3c6c" pid=2008 comm="apparmor_parser"
[ 2785.467510] uhci_hcd 0000:00:10.0: remove, state 4
[ 2785.472426] usb usb1: USB disconnect, device number 1
/*bind 0000:00:10.0 to vfio-pci*/
[ 2785.478798] uhci_hcd 0000:00:10.0: USB bus 1 deregistered
[ 2785.568741] uhci_hcd 0000:00:10.1: remove, state 1
[ 2785.573562] usb usb2: USB disconnect, device number 1
[ 2785.578793] usb 2-2: USB disconnect, device number 3
[ 2785.758016] uhci_hcd 0000:00:10.1: USB bus 2 deregistered
/*bind 0000:00:10.1 to vfio-pci*/
[ 2786.037448] ehci-pci 0000:00:10.7: remove, state 4
[ 2786.042460] usb usb3: USB disconnect, device number 1
[ 2786.048700] ehci-pci 0000:00:10.7: USB bus 3 deregistered
/*bind 0000:00:10.7 to vfio-pci*/
[ 2787.518041] audit: type=1400 audit(1594375839.459:49): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="libvirt-fa674e73-67a2-4672-8524-e62dea8a3c6c" pid=2034 comm="apparmor_parser"
[ 2788.290706] audit: type=1400 audit(1594375840.231:50): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="libvirt-fa674e73-67a2-4672-8524-e62dea8a3c6c" pid=2037 comm="apparmor_parser"
[ 2788.960070] audit: type=1400 audit(1594375840.899:51): apparmor="STATUS" operation="profile_replace" info="same as current profile, skipping" profile="unconfined" name="libvirt-fa674e73-67a2-4672-8524-e62dea8a3c6c" pid=2040 comm="apparmor_parser"
[ 2788.968821] virbr0: port 2(vnet0) entered blocking state
[ 2788.988159] virbr0: port 2(vnet0) entered disabled state
[ 2788.993711] device vnet0 entered promiscuous mode
[ 2788.999453] virbr0: port 2(vnet0) entered blocking state
[ 2789.005053] virbr0: port 2(vnet0) entered listening state
[ 2789.098717] systemd-journald[286]: Successfully sent stream file descriptor to service manager.
[ 2789.564241] audit: type=1400 audit(1594375841.507:52): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="libvirt-fa674e73-67a2-4672-8524-e62dea8a3c6c" pid=2065 comm="apparmor_parser"
[ 2791.028028] virbr0: port 2(vnet0) entered learning state
[ 2793.047999] virbr0: port 2(vnet0) entered forwarding state
[ 2793.053449] virbr0: topology change detected, propagating
[ 2793.433604] vfio_cap_init: 0000:00:10.7 hiding cap 0xa
/* shutdown virtual machine*/
[ 3838.772058] systemd-journald[286]: Successfully sent stream file descriptor to service manager.
[ 3838.815819] systemd-journald[286]: Successfully sent stream file descriptor to service manager.
[ 3838.871002] systemd-journald[286]: Successfully sent stream file descriptor to service manager.
[ 3838.884606] systemd-journald[286]: Successfully sent stream file descriptor to service manager.
[ 3838.894514] systemd-journald[286]: Successfully sent stream file descriptor to service manager.
[ 3838.896791] rfkill: input handler enabled
[ 3838.903896] systemd-journald[286]: Successfully sent stream file descriptor to service manager.
[ 3838.907998] systemd[1]: Sent message type=signal sender=n/a destination=n/a path=/org/freedesktop/systemd1/unit/packagekit_2eservice interface=org.freedesktop.DBus.Properties member=PropertiesChanged cookie=952 reply_cookie=0 signature=sa{sv}as error-name=n/a error-message=n/a
[ 3838.949500] systemd-journald[286]: Successfully sent stream file descriptor to service manager.
[ 3839.002757] systemd-journald[286]: Successfully sent stream file descriptor to service manager.
[ 3839.182053] systemd-journald[286]: Successfully sent stream file descriptor to service manager.
[ 3839.191313] systemd-journald[286]: Successfully sent stream file descriptor to service manager.
[ 3838.302725] libvirt-guests.sh[2161]: Running guests on default URI: generic
[ 3838.306783] libvirt-guests.sh[2161]: Shutting down guests on default URI...
[ 3838.415103] libvirt-guests.sh[2161]: Starting shutdown on guest: generic
plymouth-poweroff.service
[ OK ] Stopped Firmware update daemon.
[ OK ] Stopped Session 1 of user wang.
[ OK ] Removed slice User Slice of wang.
Stopping Permit User Sessions...
Stopping Login Service...
[ OK ] Stopped Permit User Sessions.
[ OK ] Unmounted Mount unit for core, revision 9289.
[ OK ] Unmounted Mount unit for gnome-system-monitor, revision 148.
[ OK ] Unmounted Mount unit for gnome-3-34-1804, revision 33.
[ OK ] Unmounted Mount unit for gnome-logs, revision 81.
[ OK ] Unmounted Mount unit for core18, revision 1754.
[ OK ] Unmounted Mount unit for gnome-3-28-1804, revision 116.
[ OK ] Unmounted Mount unit for gnome-characters, revision 550.
[ OK ] Unmounted Mount unit for gnome-3-34-1804, revision 36.
[ OK ] Unmounted Mount unit for core, revision 9436.
[ OK ] Unmounted Mount unit for gnome-characters, revision 539.
[ OK ] Unmounted Mount unit for gtk-common-themes, revision 1440.
[ OK ] Unmounted Mount unit for gtk-common-themes, revision 1506.
[ OK ] Unmounted Mount unit for core18, revision 1668.
[ OK ] Unmounted Mount unit for gnome-calculator, revision 544.
[ OK ] Unmounted Mount unit for gnome-calculator, revision 748.
[ OK ] Unmounted Mount unit for gnome-logs, revision 100.
[ OK ] Unmounted Mount unit for gnome-3-28-1804, revision 128.
[ OK ] Stopped Login Service.
[ OK ] Stopped target User and Group Name Lookups.
[ 3839.471635] libvirt-guests.sh[2161]: Waiting for 1 guests to shut down, 120 seconds left
[ 3841.824842] virbr0: port 2(vnet0) entered disabled state
[ 3841.832949] device vnet0 left promiscuous mode
[ 3841.837393] virbr0: port 2(vnet0) entered disabled state
[[ 3843.167495] audit: type=1400 audit(1594376895.107:53): apparmor="STATUS" operation="profile_remove" profile="unconfined" name="libvirt-fa674e73-67a2-4672-8524-e62dea8a3c6c" pid=2301 comm="apparmor_parser"
[ 3843.246397] BUG: unable to handle kernel NULL pointer dereference at 0000000000000000
[ 3843.254157] PGD 0 P4D 0
[ 3843.256671] Oops: 0002 [#1] SMP NOPTI
[ 3843.260301] CPU: 1 PID: 1812 Comm: libvirtd Not tainted 4.19.65 #10
[ 3843.266511] Hardware name: Shanghai Zhaoxin Semiconductor Co., Ltd. HX002EA/HX002EA, BIOS HX002EA0_03_R490_D_200707 07/07/2020
[ 3843.277804] RIP: 0010:vfio_device_put+0xa5/0x140 [vfio]
[ 3843.283129] Code: 1c 4e ce 48 8b 73 28 48 8b 53 20 48 c7 c7 48 54 61 c0 e8 51 1c 4e ce 48 8b 53 20 48 8b 43 28 48 c7 c7 80 54 61 c0 48 89 42 08 <48> 89 10 48 b8 00 01 00 00 00 00 ad de 48 89 43 20 48 05 00 01 00
[ 3843.301726] RSP: 0018:ffff8fde2210bc28 EFLAGS: 00010282
[ 3843.306905] RAX: 0000000000000000 RBX: ffff8fde1a59ed00 RCX: 0000000000000000
[ 3843.313975] RDX: ffff8fdde27d6820 RSI: ffff8fde2fa96438 RDI: ffffffffc0615480
[ 3843.321118] RBP: ffff8fde2210bc48 R08: 0000000000000c90 R09: 3d7478656e2c303d
[ 3843.328190] R10: ffff8fdde2284520 R11: 3032383664373265 R12: ffff8fdde27d6800
[ 3843.335314] R13: ffff8fde1a59ed20 R14: ffff8fdde27d6800 R15: 0000000000000000
[ 3843.342390] FS: 00007f91fe3d6700(0000) GS:ffff8fde2fa80000(0000) knlGS:0000000000000000
[ 3843.350457] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 3843.356221] CR2: 0000000000000000 CR3: 0000000425fec005 CR4: 0000000000160ee0
[ 3843.363312] Call Trace:
[ 3843.365778] vfio_del_group_dev+0x105/0x2e0 [vfio]
[ 3843.370537] ? do_wait_intr_irq+0x90/0x90
[ 3843.374550] ? vprintk_func+0x47/0xc0
[ 3843.378202] vfio_pci_remove+0x20/0xe0 [vfio_pci]
[ 3843.382900] pci_device_remove+0x51/0xd0
[ 3843.386799] device_release_driver_internal+0x18d/0x250
[ 3843.392042] device_release_driver+0x12/0x20
[ 3843.396324] unbind_store+0xbd/0x190
[ 3843.399869] drv_attr_store+0x27/0x40
[ 3843.403541] sysfs_kf_write+0x3c/0x50
[ 3843.407251] kernfs_fop_write+0x125/0x1a0
[ 3843.411431] __vfs_write+0x3a/0x190
[ 3843.414902] ? apparmor_file_permission+0x1a/0x20
[ 3843.419648] ? security_file_permission+0x31/0xc0
[ 3843.424648] ? _cond_resched+0x19/0x40
[ 3843.428407] vfs_write+0xb1/0x1a0
[ 3843.431696] ksys_write+0x5c/0xe0
[ 3843.435048] __x64_sys_write+0x1a/0x20
[ 3843.439056] do_syscall_64+0x5a/0x120
[ 3843.442713] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 3843.447757] RIP: 0033:0x7f924e2902b7
[ 3843.451385] Code: 44 00 00 41 54 55 49 89 d4 53 48 89 f5 89 fb 48 83 ec 10 e8 5b fd ff ff 4c 89 e2 41 89 c0 48 89 ee 89 df b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 35 44 89 c7 48 89 44 24 08 e8 94 fd ff ff 48
[ 3843.470696] RSP: 002b:00007f91fe3d5810 EFLAGS: 00000293 ORIG_RAX: 0000000000000001
[ 3843.478508] RAX: ffffffffffffffda RBX: 0000000000000016 RCX: 00007f924e2902b7
[ 3843.485627] RDX: 000000000000000c RSI: 00007f921001c784 RDI: 0000000000000016
[ 3843.493001] RBP: 00007f921001c784 R08: 0000000000000000 R09: 000000000000000d
[ 3843.500120] R10: 0000000000000000 R11: 0000000000000293 R12: 000000000000000c
[ 3843.507413] R13: 0000000000000000 R14: 0000000000000016 R15: 00007f91e8112e20
[ 3843.514541] Modules linked in: vfio_pci vfio_virqfd vfio_iommu_type1 vfio xt_CHECKSUM iptable_mangle ipt_MASQUERADE iptable_nat nf_nat_ipv4 nf_nat xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ipt_REJECT nf_reject_ipv4 xt_tcpudp bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter bpfilter amdgpu chash gpu_sched nls_iso8859_1 snd_hda_codec_hdmi radeon snd_hda_intel snd_hda_codec snd_hda_core snd_hwdep snd_pcm via_cputemp hwmon_vid kvm_intel kvm snd_seq_midi ttm snd_seq_midi_event irqbypass snd_rawmidi crct10dif_pclmul crc32_pclmul drm_kms_helper ghash_clmulni_intel snd_seq pcbc aesni_intel aes_x86_64 crypto_simd drm snd_seq_device cryptd glue_helper joydev snd_timer input_leds serio_raw snd i2c_algo_bit fb_sys_fops syscopyarea video sysfillrect sysimgblt
[ 3843.586463] soundcore mac_hid sch_fq_codel parport_pc ppdev lp parport ip_tables x_tables autofs4 hid_generic ahci psmouse r8169 usbhid hid libahci realtek
[ 3843.600637] CR2: 0000000000000000
[ 3843.604108] ---[ end trace 65d72623b84bf7a3 ]---
[ 3843.608839] RIP: 0010:vfio_device_put+0xa5/0x140 [vfio]
[ 3843.614209] Code: 1c 4e ce 48 8b 73 28 48 8b 53 20 48 c7 c7 48 54 61 c0 e8 51 1c 4e ce 48 8b 53 20 48 8b 43 28 48 c7 c7 80 54 61 c0 48 89 42 08 <48> 89 10 48 b8 00 01 00 00 00 00 ad de 48 89 43 20 48 05 00 01 00
[ 3843.633419] RSP: 0018:ffff8fde2210bc28 EFLAGS: 00010282
[ 3843.638816] RAX: 0000000000000000 RBX: ffff8fde1a59ed00 RCX: 0000000000000000
[ 3843.646133] RDX: ffff8fdde27d6820 RSI: ffff8fde2fa96438 RDI: ffffffffc0615480
[ 3843.653551] RBP: ffff8fde2210bc48 R08: 0000000000000c90 R09: 3d7478656e2c303d
[ 3843.661304] R10: ffff8fdde2284520 R11: 3032383664373265 R12: ffff8fdde27d6800
[ 3843.668540] R13: ffff8fde1a59ed20 R14: ffff8fdde27d6800 R15: 0000000000000000
[ 3843.676029] FS: 00007f91fe3d6700(0000) GS:ffff8fde2fa80000(0000) knlGS:0000000000000000
[ 3843.684398] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 3843.690269] CR2: 0000000000000000 CR3: 0000000425fec005 CR4: 0000000000160ee0
Thanks
weitaowang
保密声明：
本邮件含有保密或专有信息，仅供指定收件人使用。严禁对本邮件或其内容做任何未经授权的查阅、使用、复制或转发。
CONFIDENTIAL NOTE:
This email contains confidential or legally privileged information and is for the sole use of its intended recipient. Any unauthorized review, use, copying or forwarding of this email or the content of this email is strictly prohibited.
