On 08/05/18 05:11 PM, Alex Williamson wrote: > On to the implementation details... I already mentioned the BDF issue > in my other reply. If we had a way to persistently identify a device, > would we specify the downstream points at which we want to disable ACS > or the endpoints that we want to connect? The latter has a problem > that the grouping upstream of an endpoint is already set by the time we > discover the endpoint, so we might need to unwind to get the grouping > correct. The former might be more difficult for users to find the > necessary nodes, but easier for the kernel to deal with during > discovery. I was envisioning the former with kernel helping by printing a dmesg in certain circumstances to help with figuring out which devices need to be specified. Specifying a list of endpoints on the command line and having the kernel try to figure out which downstream ports need to be adjusted while we are in the middle of enumerating the bus is, like you said, a nightmare. > A runtime, sysfs approach has some benefits here, > especially in identifying the device assuming we're ok with leaving > the persistence problem to userspace tools. I'm still a little fond of > the idea of exposing an acs_flags attribute for devices in sysfs where > a write would do a soft unplug and re-add of all affected devices to > automatically recreate the proper grouping. Any dynamic change in > routing and grouping would require all DMA be re-established anyway and > a soft hotplug seems like an elegant way of handling it. Thanks, This approach sounds like it has a lot more issues to contend with: For starters, a soft unplug/re-add of all the devices behind a switch is going to be difficult if a lot of those devices have had drivers installed and their respective resources are now mounted or otherwise in use. Then, do we have to redo a the soft-replace every time we change the ACS bit for every downstream port? That could mean you have to do dozens soft-replaces before you have all the ACS bits set which means you have a storm of drivers being added and removed. This would require some kind of fancy custom setup software that runs at just the right time in the boot sequence or a lot of work on the users part to unbind all the resources, setup the ACS bits and then rebind everything (assuming the soft re-add doesn't rebind it every time you adjust one ACS bit). Ugly. IMO, if we need to do the sysfs approach then we need to be able to adjust the groups dynamically in a sensible way and not through the large hammer that is soft-replaces. I think this would be great but I don't think we will be tackling that with this patch set. Logan -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
