Hi Alex, Cornelia, > -----Original Message----- > From: Cornelia Huck <cohuck@xxxxxxxxxx> > Sent: Wednesday, May 22, 2019 3:25 PM > To: Parav Pandit <parav@xxxxxxxxxxxx> > Cc: kvm@xxxxxxxxxxxxxxx; linux-kernel@xxxxxxxxxxxxxxx; > kwankhede@xxxxxxxxxx; alex.williamson@xxxxxxxxxx; cjia@xxxxxxxxxx > Subject: Re: [PATCHv3 1/3] vfio/mdev: Improve the create/remove sequence > > On Thu, 16 May 2019 18:30:32 -0500 > Parav Pandit <parav@xxxxxxxxxxxx> wrote: > > > This patch addresses below two issues and prepares the code to address > > 3rd issue listed below. > > > > 1. mdev device is placed on the mdev bus before it is created in the > > vendor driver. Once a device is placed on the mdev bus without > > creating its supporting underlying vendor device, mdev driver's probe() > gets triggered. > > However there isn't a stable mdev available to work on. > > > > create_store() > > mdev_create_device() > > device_register() > > ... > > vfio_mdev_probe() > > [...] > > parent->ops->create() > > vfio_ap_mdev_create() > > mdev_set_drvdata(mdev, matrix_mdev); > > /* Valid pointer set above */ > > > > Due to this way of initialization, mdev driver who wants to use the > > mdev, doesn't have a valid mdev to work on. > > > > 2. Current creation sequence is, > > parent->ops_create() > > groups_register() > > > > Remove sequence is, > > parent->ops->remove() > > groups_unregister() > > > > However, remove sequence should be exact mirror of creation sequence. > > Once this is achieved, all users of the mdev will be terminated first > > before removing underlying vendor device. > > (Follow standard linux driver model). > > At that point vendor's remove() ops shouldn't fail because taking the > > device off the bus should terminate any usage. > > > > 3. When remove operation fails, mdev sysfs removal attempts to add the > > file back on already removed device. Following call trace [1] is observed. > > > > [1] call trace: > > kernel: WARNING: CPU: 2 PID: 9348 at fs/sysfs/file.c:327 > > sysfs_create_file_ns+0x7f/0x90 > > kernel: CPU: 2 PID: 9348 Comm: bash Kdump: loaded Not tainted > > 5.1.0-rc6-vdevbus+ #6 > > kernel: Hardware name: Supermicro SYS-6028U-TR4+/X10DRU-i+, BIOS 2.0b > > 08/09/2016 > > kernel: RIP: 0010:sysfs_create_file_ns+0x7f/0x90 > > kernel: Call Trace: > > kernel: remove_store+0xdc/0x100 [mdev] > > kernel: kernfs_fop_write+0x113/0x1a0 > > kernel: vfs_write+0xad/0x1b0 > > kernel: ksys_write+0x5a/0xe0 > > kernel: do_syscall_64+0x5a/0x210 > > kernel: entry_SYSCALL_64_after_hwframe+0x49/0xbe > > > > Therefore, mdev core is improved in following ways. > > > > 1. Split the device registration/deregistration sequence so that some > > things can be done between initialization of the device and hooking it > > up to the bus respectively after deregistering it from the bus but > > before giving up our final reference. > > In particular, this means invoking the ->create and ->remove callbacks > > in those new windows. This gives the vendor driver an initialized mdev > > device to work with during creation. > > At the same time, a bus driver who wish to bind to mdev driver also > > s/who wish/that wishes/ > > > gets initialized mdev device. > > > > This follows standard Linux kernel bus and device model. > > > > 2. During remove flow, first remove the device from the bus. This > > ensures that any bus specific devices are removed. > > Once device is taken off the mdev bus, invoke remove() of mdev from > > the vendor driver. > > > > 3. The driver core device model provides way to register and auto > > unregister the device sysfs attribute groups at dev->groups. > > Make use of dev->groups to let core create the groups and eliminate > > code to avoid explicit groups creation and removal. > > > > To ensure, that new sequence is solid, a below stack dump of a process > > is taken who attempts to remove the device while device is in use by > > vfio driver and user application. > > This stack dump validates that vfio driver guards against such device > > removal when device is in use. > > > > cat /proc/21962/stack > > [<0>] vfio_del_group_dev+0x216/0x3c0 [vfio] [<0>] > > mdev_remove+0x21/0x40 [mdev] [<0>] > > device_release_driver_internal+0xe8/0x1b0 > > [<0>] bus_remove_device+0xf9/0x170 > > [<0>] device_del+0x168/0x350 > > [<0>] mdev_device_remove_common+0x1d/0x50 [mdev] [<0>] > > mdev_device_remove+0x8c/0xd0 [mdev] [<0>] remove_store+0x71/0x90 > > [mdev] [<0>] kernfs_fop_write+0x113/0x1a0 [<0>] vfs_write+0xad/0x1b0 > > [<0>] ksys_write+0x5a/0xe0 [<0>] do_syscall_64+0x5a/0x210 [<0>] > > entry_SYSCALL_64_after_hwframe+0x49/0xbe > > [<0>] 0xffffffffffffffff > > > > This prepares the code to eliminate calling device_create_file() in > > subsquent patch. > > > > Signed-off-by: Parav Pandit <parav@xxxxxxxxxxxx> > > --- > > drivers/vfio/mdev/mdev_core.c | 94 +++++++++----------------------- > > drivers/vfio/mdev/mdev_private.h | 2 +- > > drivers/vfio/mdev/mdev_sysfs.c | 2 +- > > 3 files changed, 27 insertions(+), 71 deletions(-) > > Personally, I'd do a more compact patch description, but there's nothing > really wrong with yours, either. > > Patch also seems sane to me, although I'd probably have merged this and > the next patch. But no reason to quibble further. > > Reviewed-by: Cornelia Huck <cohuck@xxxxxxxxxx> I missed to add your RB signature in v4. If you send all 3 or just fist 2 of them in 5.2-rc, please include Cornelia's RB tag.
