> -----Original Message----- > From: Brett Creeley [mailto:brett.creeley@xxxxxxx] > Sent: 12 October 2023 00:01 > To: jgg@xxxxxxxx; yishaih@xxxxxxxxxx; Shameerali Kolothum Thodi > <shameerali.kolothum.thodi@xxxxxxxxxx>; kevin.tian@xxxxxxxxx; > alex.williamson@xxxxxxxxxx; dan.carpenter@xxxxxxxxxx > Cc: kvm@xxxxxxxxxxxxxxx; linux-kernel@xxxxxxxxxxxxxxx; > brett.creeley@xxxxxxx; shannon.nelson@xxxxxxx > Subject: [PATCH v2 vfio 2/3] pds/vfio: Fix mutex lock->magic != lock warning > > The following BUG was found when running on a kernel with > CONFIG_DEBUG_MUTEXES=y set: > > DEBUG_LOCKS_WARN_ON(lock->magic != lock) > RIP: 0010:mutex_trylock+0x10d/0x120 > Call Trace: > <TASK> > ? __warn+0x85/0x140 > ? mutex_trylock+0x10d/0x120 > ? report_bug+0xfc/0x1e0 > ? handle_bug+0x3f/0x70 > ? exc_invalid_op+0x17/0x70 > ? asm_exc_invalid_op+0x1a/0x20 > ? mutex_trylock+0x10d/0x120 > ? mutex_trylock+0x10d/0x120 > pds_vfio_reset+0x3a/0x60 [pds_vfio_pci] > pci_reset_function+0x4b/0x70 > reset_store+0x5b/0xa0 > kernfs_fop_write_iter+0x137/0x1d0 > vfs_write+0x2de/0x410 > ksys_write+0x5d/0xd0 > do_syscall_64+0x3b/0x90 > entry_SYSCALL_64_after_hwframe+0x6e/0xd8 > > As shown, lock->magic != lock. This is because > mutex_init(&pds_vfio->state_mutex) is called in the VFIO open path. So, > if a reset is initiated before the VFIO device is opened the mutex will > have never been initialized. Fix this by calling > mutex_init(&pds_vfio->state_mutex) in the VFIO init path. > > Also, don't destroy the mutex on close because the device may > be re-opened, which would cause mutex to be uninitialized. Fix this by > implementing a driver specific vfio_device_ops.release callback that > destroys the mutex before calling vfio_pci_core_release_dev(). > > Signed-off-by: Brett Creeley <brett.creeley@xxxxxxx> > Reviewed-by: Shannon Nelson <shannon.nelson@xxxxxxx> Reviewed-by: Shameer Kolothum <shameerali.kolothum.thodi@xxxxxxxxxx> Looks like mutex destruction logic needs to be added to HiSilicon driver as well. >From a quick look mlx5 also doesn't destroy the state_mutex. Thanks, Shameer > --- > drivers/vfio/pci/pds/vfio_dev.c | 15 ++++++++++++--- > 1 file changed, 12 insertions(+), 3 deletions(-) > > diff --git a/drivers/vfio/pci/pds/vfio_dev.c b/drivers/vfio/pci/pds/vfio_dev.c > index c351f588fa13..306b1c25f016 100644 > --- a/drivers/vfio/pci/pds/vfio_dev.c > +++ b/drivers/vfio/pci/pds/vfio_dev.c > @@ -155,6 +155,7 @@ static int pds_vfio_init_device(struct vfio_device > *vdev) > > pds_vfio->vf_id = vf_id; > > + mutex_init(&pds_vfio->state_mutex); > spin_lock_init(&pds_vfio->reset_lock); > > vdev->migration_flags = VFIO_MIGRATION_STOP_COPY | > VFIO_MIGRATION_P2P; > @@ -170,6 +171,16 @@ static int pds_vfio_init_device(struct vfio_device > *vdev) > return 0; > } > > +static void pds_vfio_release_device(struct vfio_device *vdev) > +{ > + struct pds_vfio_pci_device *pds_vfio = > + container_of(vdev, struct pds_vfio_pci_device, > + vfio_coredev.vdev); > + > + mutex_destroy(&pds_vfio->state_mutex); > + vfio_pci_core_release_dev(vdev); > +} > + > static int pds_vfio_open_device(struct vfio_device *vdev) > { > struct pds_vfio_pci_device *pds_vfio = > @@ -181,7 +192,6 @@ static int pds_vfio_open_device(struct vfio_device > *vdev) > if (err) > return err; > > - mutex_init(&pds_vfio->state_mutex); > pds_vfio->state = VFIO_DEVICE_STATE_RUNNING; > pds_vfio->deferred_reset_state = VFIO_DEVICE_STATE_RUNNING; > > @@ -201,14 +211,13 @@ static void pds_vfio_close_device(struct > vfio_device *vdev) > pds_vfio_put_save_file(pds_vfio); > pds_vfio_dirty_disable(pds_vfio, true); > mutex_unlock(&pds_vfio->state_mutex); > - mutex_destroy(&pds_vfio->state_mutex); > vfio_pci_core_close_device(vdev); > } > > static const struct vfio_device_ops pds_vfio_ops = { > .name = "pds-vfio", > .init = pds_vfio_init_device, > - .release = vfio_pci_core_release_dev, > + .release = pds_vfio_release_device, > .open_device = pds_vfio_open_device, > .close_device = pds_vfio_close_device, > .ioctl = vfio_pci_core_ioctl, > -- > 2.17.1
