On 10/17/2016 2:20 PM, Jarod Wilson wrote:
> On Mon, Oct 17, 2016 at 11:46:11AM -0600, Jason Gunthorpe wrote:
>> On Mon, Oct 17, 2016 at 12:22:21PM -0400, Jarod Wilson wrote:
>>>>> diff --git a/glue/redhat/ibacm.service b/glue/redhat/ibacm.service
>>>>> new file mode 100644
>>>>> index 0000000..1cd031a
>>>>> +++ b/glue/redhat/ibacm.service
>>>>
>>>> Can we just put this in ibacm/ ?
>>>
>>> Probably.
>>
>> Okay, the only thing I really don't like being upstream is the
>> opensm.service..
>>
>> Do you know why acm needs that?
>
> I think Doug already attempted to address this elsewhere in the thread,
> and he'd know better than me.
Since it got snipped, I'm pretty sure it just lists opensm in the Wants=
tag. With systemd, that's a soft dependency. Only if opensm is
installed and configured to start anyway does systemd then order this
item after opensm. Since you need opensm for links to come up anyway,
it makes sense for the order of startup for RDMA related items to be:
rdma-core
\-opensm
\-Everything else
Because of that, I think I had opensm listed as a want in pretty much
everything, but as already stated, it's a soft dependency and systemd
ignores it if you don't have opensm configured to run on that machine.
>>>>> +SUBSYSTEM=="module", KERNEL=="cxgb*", ACTION=="add", TAG+="systemd", ENV{SYSTEMD_WANTS}="rdma.service"
>>>>> +SUBSYSTEM=="module", KERNEL=="ib_*", ACTION=="add", TAG+="systemd", ENV{SYSTEMD_WANTS}="rdma.service"
>>>>> +SUBSYSTEM=="module", KERNEL=="mlx*", ACTION=="add", TAG+="systemd", ENV{SYSTEMD_WANTS}="rdma.service"
>>>>> +SUBSYSTEM=="module", KERNEL=="iw_*", ACTION=="add", TAG+="systemd", ENV{SYSTEMD_WANTS}="rdma.service"
>>>>> +SUBSYSTEM=="module", KERNEL=="be2net", ACTION=="add", TAG+="systemd", ENV{SYSTEMD_WANTS}="rdma.service"
>>>>> +SUBSYSTEM=="module", KERNEL=="enic", ACTION=="add", TAG+="systemd", ENV{SYSTEMD_WANTS}="rdma.service"
>>>>
>>>> Also cross distro
>>>
>>> Yeah, these are definitely prime candidates for being cross-distro. And
>>> really, I was thinking maybe these should be part of the core upstream
>>> udev/systemd rules set, rather than something we ship here.
>>
>> Okay. The trick will be to standardize the systemd_wants name ..
>
> Perhaps rdma-core should go with rdma-core.service? We were shipping a
> package called 'rdma' that carried that.
Whatever the service file is, that's what the name is. I'm partial to
just leaving it as rdma.service. The -core suffix doesn't add anything
of value IMO, and we really are initializing the entire rdma stack minus
just those upper layer protocols that have their own setup.
>>>>> +# When we detect a new verbs device is added to the system, set the node
>>>>> +# description on that device
>>>>> +# If rdma-ndd is installed, defer the setting of the node description to it.
>>>>> +SUBSYSTEM=="infiniband", KERNEL=="*", ACTION=="add", TEST!="/usr/sbin/rdma-ndd", RUN+="/bin/bash -c 'sleep 1; echo -n `hostname -s` %k > /sys/class/infiniband/%k/node_desc'"
>>>>
>>>> Shouldn't this udev drop-in by in the rdma-ndd package?
>>>
>>> Honestly don't even have a clue what rdma-ndd is. :) Don't remember what
>>> Doug said about this one...
>>
>> Guess I didn't read closely enough, this is used if rdma-ndd is not
>> installed..
A bit of a chicken and egg issue here isn't there? Wasn't rdma-ndd part
of ibutils? And since ibutils isn't in rdma-core, how do we know if
it's installed?
>> So something like that should be upstream, but the 'sleep 1' is ugly -
>> this should probably be a systemd service that runs after
>> network-online.target not as a script run from udev?
>>
>> rdma-ndd dynamically sets the NodeDescription to the hostname in the
>> adaptor for the subnet manager/tools to ready. I guess this hunk is
>> setting the NodeDescription one-shot at boot..
>
> Rather than having this janky udev rule, what if we simply made rdma-ndd
> part of what's installed with rdma-core, rather than something found in
> yet another infiniband package? (Looks like it's in infiniband-diags,
> wasn't even aware rdma-ndd existed until looking at this here).
It's probably a good candidate to be pulled back to rdma-core. Ira?
--
Doug Ledford <dledford@xxxxxxxxxx>
GPG Key ID: 0E572FDD
Attachment:
signature.asc
Description: OpenPGP digital signature
