On Wed, 2017-01-11 at 11:10 -0700, Jason Gunthorpe wrote: > On Tue, Jan 10, 2017 at 03:57:52PM -0800, Tadeusz Struk wrote: > > > > We can fix this by enforcing the correct port order at module load > > time. > > To reorder the ports to match the numbering labels on the back of > > the > > device we need to delay registering devices with the ib_core until > > we > > Sorry, no way - this is horrifying. Agree. > If you need stable names for RDMA devices then you need to add proper > infrastructure to the kernel to rename RDMA devices from user space > via udev. ala netdev. This has its own problems in this particular case, namely having to ship files that know which parts are effected and then modify the names. Or requiring that users create all of the rename rules when they really shouldn't be bothered with anything. Although, the module option to turn this fix off for existing clusters that have been wired up wrong is just as bad as creating persistent naming rules... > or change the ib_core to allow your driver to specify the full name > and manage things in your driver. > > No way on this insane block probe approach. Isn't there already code in the code device layer to handle this kind of thing? I seem to recall backporting it from upstream to a rhel kernel many years ago. Lemme go look... OK, sure, as far as the reordering stuff is concerned, all you need to do is to make use of the EPROBE_DEFER return option to your PCI probe routine. That way, when you get the probe for the out of order port, the first time you pass EPROBE_DEFER as your return, then you get the second port, your register it with the IB layer which will make it appear as the first port (optionally, and as I haven't tried this I don't know if it will work or if it's necessary, you can save the pointer to the first port's device struct off, and when you get the second one, you can tell the driver layer to splice the second port in front of the first port in the device child list on the parent device, but I think this is optional and really has no lasting effect on the outcome), then later the first port gets retried and ends up being the second port. So, scrap all of this hand done junk and use the provided infrastructure as it was intended to be used. I *really* don't want to do a kernel module option for this either. Do you know for a fact that this is wired up wrong in the field and the people can't just swap hfi1_0<->hfi1_1 in their config files and have it all work without being recabled? -- Doug Ledford <dledford@xxxxxxxxxx> GPG KeyID: B826A3330E572FDD Key fingerprint = AE6B 1BDA 122B 23B4 265B 1274 B826 A333 0E57 2FDD
Attachment:
signature.asc
Description: This is a digitally signed message part
