On 3/1/2017 5:32 PM, Alex Rosenbaum wrote:
> On Thu, Mar 2, 2017 at 12:07 AM, Doug Ledford <dledford@xxxxxxxxxx> wrote:
>> Personally I would use a thread in the library that would block on
>> inotify events from the sysfs directory. Any time it got woke up, it
>> would process the remove/add event, update the device list, create a
>> notification if the app had registered a notification handler and queue
>> that to the app, then go back to sleep on the inotify events in the
>> sysfs dir. The processing of the event and the notification would be
>> totally disjoint. The app need not process events, even if it
>> registered and event handler, in order to get the new device list.
>
> I actually did look at inotify() for this problem and agree that this
> is can be a good option.
> There are some issues with it. You get a few more events then you
> really need (some extra context switches). It does not map all sysfs
> cleanly, there are some limitation. It will not wakeup on
> /sys/class/infiniband_verbs/*, but will catch the
> /dev/infiniband/uverbes% new instances.
>
> Then there is a question of which context/thread? library vs applications
On further reflection, I think inotify is out of the question. The
problem is that our initial setup is done via systemd on most modern
systems, and under systemd the rdma.init kernel setup script is a
oneshot script. The udev rules for various rdma capable hardware all
call out for the rdma.init service to be run. This is fine at bootup.
However, once we are up and running, I don't think a hotplug of any
module driver that services only the ethernet portion of a two part
driver (cxgb*/hns/bnxt) will trigger a load of the corresponding RDMA
portion of the driver. I think we need to monitor udev events, we
specifically need to trap for all add events for class network, then
need to detect if the device added belongs to one of the known paired
ethernet/RDMA drivers, and if the RDMA module didn't get loaded, load
it. That doesn't work if we just watch /dev/infiniband/uverbs* or
/sys/class/infiniband_verbs/* as these devices will never show up until
someone loads the proper module.
--
Doug Ledford <dledford@xxxxxxxxxx>
GPG Key ID: B826A3330E572FDD
Key fingerprint = AE6B 1BDA 122B 23B4 265B 1274 B826 A333 0E57 2FDD
Attachment:
signature.asc
Description: OpenPGP digital signature
