On Wed, Mar 18, 2020 at 04:02:42PM +0000, Wan, Kaike wrote: > > > > From: Jason Gunthorpe <jgg@xxxxxxxx> > > Sent: Wednesday, March 18, 2020 9:32 AM > > To: Dalessandro, Dennis <dennis.dalessandro@xxxxxxxxx> > > Cc: dledford@xxxxxxxxxx; linux-rdma@xxxxxxxxxxxxxxx; Marciniszyn, Mike > > <mike.marciniszyn@xxxxxxxxx>; Wan, Kaike <kaike.wan@xxxxxxxxx> > > Subject: Re: [PATCH for-next 3/3] IB/hfi1: Use the ibdev in hfi1_devdata as > > the parent of cdev > > > > On Mon, Mar 16, 2020 at 05:05:07PM -0400, Dennis Dalessandro wrote: > > > From: Kaike Wan <kaike.wan@xxxxxxxxx> > > > > > > This patch is implemented to address the concerns raised in: > > > https://marc.info/?l=linux-rdma&m=158101337614772&w=2 > > > > > > The hfi1 driver dynammically allocates a struct device to represent > > > the cdev in sysfs and devtmpfs (/dev/hfi1_x). On the other hand, the > > > hfi1_devdata already contains a struct device in its ibdev field > > > (hfi1_devdata.verbs_dev.rdi.ibdev.dev), and it is therefore possible > > > to eliminate the dynamical allocation when creating the cdev. Since > > > each device could be added to the sysfs only once and the function > > > device_add() is already called for the ibdev in ib_register_device(), > > > the function cdev_device_add() could not be used to create the cdev, > > > even though the hfi1_devdata contains both cdev and ibdev in the same > > > structure. > > > > > > This patch eliminates the dynamic allocation by creating the cdev > > > first, setting up the ibdev, and then calling the ib_register_device() > > > to add the device to sysfs and devtmpfs. > > > > What do the sysfs paths for the cdev look like now? > > ls -l /sys/dev/char/243:0 > lrwxrwxrwx 1 root root 0 Mar 15 14:30 /sys/dev/char/243:0 -> ../../devices/pci0000:00/0000:00:02.0/0000:02:00.0/infiniband/hfi1_0 > > It points back to the IB device (hfi1_0 ). > > Before this change, it pointed back to a virtual device: > > ls /sys/dev/char/243:0 -l > lrwxrwxrwx 1 root root 0 Mar 18 11:52 /sys/dev/char/243:0 -> ../../devices/virtual/hfi1_user/hfi1_0 Great, yes this looks right to me So this came up due to PSM having problems.. The right way for PSM to work is now to find the hfi1_0 devices under /sys/class/hfi1_user/* and then map then back to RDMA devices and the physical card by doing realpath and learning the '/sys/pci0000:00/0000:00:02.0/0000:02:00.0/infiniband/XXX/' path Yes? Jason
