On Fri, 2020-11-20 at 18:02 -0800, Saravana Kannan wrote: > The current implementation of fw_devlink is very inefficient because it > tries to get away without creating fwnode links in the name of saving > memory usage. Past attempts to optimize runtime at the cost of memory > usage were blocked with request for data showing that the optimization > made significant improvement for real world scenarios. > > We have those scenarios now. There have been several reports of boot > time increase in the order of seconds in this thread [1]. Several OEMs > and SoC manufacturers have also privately reported significant > (350-400ms) increase in boot time due to all the parsing done by > fw_devlink. > > So this patch uses all the setup done by the previous patches in this > series to refactor fw_devlink to be more efficient. Most of the code has > been moved out of firmware specific (DT mostly) code into driver core. > > This brings the following benefits: > - Instead of parsing the device tree multiple times during bootup, > fw_devlink parses each fwnode node/property only once and creates > fwnode links. The rest of the fw_devlink code then just looks at these > fwnode links to do rest of the work. > > - Makes it much easier to debug probe issue due to fw_devlink in the > future. fw_devlink=on blocks the probing of devices if they depend on > a device that hasn't been added yet. With this refactor, it'll be very > easy to tell what that device is because we now have a reference to > the fwnode of the device. > > - Much easier to add fw_devlink support to ACPI and other firmware > types. A refactor to move the common bits from DT specific code to > driver core was in my TODO list as a prerequisite to adding ACPI > support to fw_devlink. This series gets that done. > > [1] - https://lore.kernel.org/linux-omap/ea02f57e-871d-cd16-4418-c1da4bbc4696@xxxxxx/ > Signed-off-by: Saravana Kannan <saravanak@xxxxxxxxxx> > Tested-by: Laurent Pinchart <laurent.pinchart@xxxxxxxxxxxxxxxx> > Tested-by: Grygorii Strashko <grygorii.strashko@xxxxxx> Reverting this commit and its dependency: 2d09e6eb4a6f driver core: Delete pointless parameter in fwnode_operations.add_links from today's linux-next fixed a boot crash on an arm64 Thunder X2 server. .config: https://cailca.coding.net/public/linux/mm/git/files/master/arm64.config [ 57.413929][ T1] ACPI: 5 ACPI AML tables successfully acquired and loaded [ 60.571643][ T1] ACPI: Interpreter enabled [ 60.576104][ T1] ACPI: Using GIC for interrupt routing [ 60.582474][ T1] ACPI: MCFG table detected, 1 entries [ 60.588051][ T1] ACPI: IORT: SMMU-v3[402300000] Mapped to Proximity domain 0 [ 60.601374][ T1] Unable to handle kernel paging request at virtual address dfff800000000000 [ 60.610146][ T1] Mem abort info: [ 60.613694][ T1] ESR = 0x96000004 [ 60.617496][ T1] EC = 0x25: DABT (current EL), IL = 32 bits [ 60.623616][ T1] SET = 0, FnV = 0 [ 60.627420][ T1] EA = 0, S1PTW = 0 [ 60.631304][ T1] Data abort info: [ 60.634957][ T1] ISV = 0, ISS = 0x00000004 [ 60.639546][ T1] CM = 0, WnR = 0 [ 60.643255][ T1] [dfff800000000000] address between user and kernel address ranges [ 60.651226][ T1] Internal error: Oops: 96000004 [#1] SMP [ 60.656864][ T1] Modules linked in: [ 60.660658][ T1] CPU: 38 PID: 1 Comm: swapper/0 Tainted: G W 5.10.0-rc7-next-20201211 #2 [ 60.670424][ T1] Hardware name: HPE Apollo 70 /C01_APACHE_MB , BIOS L50_5.13_1.16 07/29/2020 [ 60.680979][ T1] pstate: 10400009 (nzcV daif +PAN -UAO -TCO BTYPE=--) [ 60.687757][ T1] pc : device_add+0xf60/0x16b0 [ 60.692430][ T1] lr : device_add+0xf08/0x16b0 [ 60.697098][ T1] sp : ffff0000063bf7d0 [ 60.701147][ T1] x29: ffff0000063bf7d0 x28: ffff00001f760810 [ 60.707226][ T1] x27: fffffffffffffff8 x26: ffff00001f760858 [ 60.713304][ T1] x25: ffff00001f760c58 x24: fffffffffffffff8 [ 60.719381][ T1] x23: 1fffe00003eec10b x22: ffff8000190d0260 [ 60.725458][ T1] x21: ffff800011dba708 x20: 0000000000000000 [ 60.731535][ T1] x19: 1fffe00000c77f10 x18: 1fffe001cf0d53ed [ 60.737616][ T1] x17: 0000000000000000 x16: 1ffff000033676f1 [ 60.743709][ T1] x15: 0000000000000000 x14: ffff800011731e34 [ 60.749786][ T1] x13: ffff700003217fb9 x12: 1ffff00003217fb8 [ 60.755864][ T1] x11: 1ffff00003217fb8 x10: ffff700003217fb8 [ 60.761940][ T1] x9 : dfff800000000000 x8 : ffff8000190bfdc7 [ 60.768017][ T1] x7 : 0000000000000001 x6 : ffff700003217fb9 [ 60.774094][ T1] x5 : ffff700003217fb9 x4 : ffff000006324a80 [ 60.780170][ T1] x3 : 1fffe00000c64951 x2 : 0000000000000000 [ 60.786247][ T1] x1 : 0000000000000000 x0 : dfff800000000000 [ 60.792324][ T1] Call trace: [ 60.795495][ T1] device_add+0xf60/0x16b0 __fw_devlink_link_to_consumers at drivers/base/core.c:1583 (inlined by) fw_devlink_link_device at drivers/base/core.c:1726 (inlined by) device_add at drivers/base/core.c:3088 [ 60.799813][ T1] platform_device_add+0x274/0x628 [ 60.804833][ T1] acpi_iort_init+0x9d8/0xc50 [ 60.809415][ T1] acpi_init+0x45c/0x4e8 [ 60.813556][ T1] do_one_initcall+0x170/0xb70 [ 60.818224][ T1] kernel_init_freeable+0x6a8/0x734 [ 60.823332][ T1] kernel_init+0x18/0x12c [ 60.827566][ T1] ret_from_fork+0x10/0x1c [ 60.831890][ T1] Code: f2fbffe0 d100205b d343fc41 aa1b03f8 (38e06820) [ 60.838756][ T1] ---[ end trace fa5c8ce17a226d83 ]--- [ 60.844127][ T1] Kernel panic - not syncing: Oops: Fatal exception [ 60.850881][ T1] SMP: stopping secondary CPUs [ 60.855733][ T1] ---[ end Kernel panic - not syncing: Oops: Fatal exception ]--- > --- > drivers/base/core.c | 325 ++++++++++++++++++++++++++++++----------- > include/linux/device.h | 5 - > 2 files changed, 238 insertions(+), 92 deletions(-) > > diff --git a/drivers/base/core.c b/drivers/base/core.c > index 1873cecb0cc4..9edf9084fc98 100644 > --- a/drivers/base/core.c > +++ b/drivers/base/core.c > @@ -46,8 +46,6 @@ early_param("sysfs.deprecated", sysfs_deprecated_setup); > #endif > > /* Device links support. */ > -static LIST_HEAD(wait_for_suppliers); > -static DEFINE_MUTEX(wfs_lock); > static LIST_HEAD(deferred_sync); > static unsigned int defer_sync_state_count = 1; > static DEFINE_MUTEX(fwnode_link_lock); > @@ -803,74 +801,6 @@ struct device_link *device_link_add(struct device *consumer, > } > EXPORT_SYMBOL_GPL(device_link_add); > > -/** > - * device_link_wait_for_supplier - Add device to wait_for_suppliers list > - * @consumer: Consumer device > - * > - * Marks the @consumer device as waiting for suppliers to become available by > - * adding it to the wait_for_suppliers list. The consumer device will never be > - * probed until it's removed from the wait_for_suppliers list. > - * > - * The caller is responsible for adding the links to the supplier devices once > - * they are available and removing the @consumer device from the > - * wait_for_suppliers list once links to all the suppliers have been created. > - * > - * This function is NOT meant to be called from the probe function of the > - * consumer but rather from code that creates/adds the consumer device. > - */ > -static void device_link_wait_for_supplier(struct device *consumer, > - bool need_for_probe) > -{ > - mutex_lock(&wfs_lock); > - list_add_tail(&consumer->links.needs_suppliers, &wait_for_suppliers); > - consumer->links.need_for_probe = need_for_probe; > - mutex_unlock(&wfs_lock); > -} > - > -static void device_link_wait_for_mandatory_supplier(struct device *consumer) > -{ > - device_link_wait_for_supplier(consumer, true); > -} > - > -static void device_link_wait_for_optional_supplier(struct device *consumer) > -{ > - device_link_wait_for_supplier(consumer, false); > -} > - > -/** > - * device_link_add_missing_supplier_links - Add links from consumer devices to > - * supplier devices, leaving any > - * consumer with inactive suppliers on > - * the wait_for_suppliers list > - * > - * Loops through all consumers waiting on suppliers and tries to add all their > - * supplier links. If that succeeds, the consumer device is removed from > - * wait_for_suppliers list. Otherwise, they are left in the wait_for_suppliers > - * list. Devices left on the wait_for_suppliers list will not be probed. > - * > - * The fwnode add_links callback is expected to return 0 if it has found and > - * added all the supplier links for the consumer device. It should return an > - * error if it isn't able to do so. > - * > - * The caller of device_link_wait_for_supplier() is expected to call this once > - * it's aware of potential suppliers becoming available. > - */ > -static void device_link_add_missing_supplier_links(void) > -{ > - struct device *dev, *tmp; > - > - mutex_lock(&wfs_lock); > - list_for_each_entry_safe(dev, tmp, &wait_for_suppliers, > - links.needs_suppliers) { > - int ret = fwnode_call_int_op(dev->fwnode, add_links, dev); > - if (!ret) > - list_del_init(&dev->links.needs_suppliers); > - else if (ret != -ENODEV) > - dev->links.need_for_probe = false; > - } > - mutex_unlock(&wfs_lock); > -} > - > #ifdef CONFIG_SRCU > static void __device_link_del(struct kref *kref) > { > @@ -1195,9 +1125,8 @@ void device_links_driver_bound(struct device *dev) > * the device links it needs to or make new device links as it needs > * them. So, it no longer needs to wait on any suppliers. > */ > - mutex_lock(&wfs_lock); > - list_del_init(&dev->links.needs_suppliers); > - mutex_unlock(&wfs_lock); > + if (dev->fwnode && dev->fwnode->dev == dev) > + fwnode_links_purge_suppliers(dev->fwnode); > device_remove_file(dev, &dev_attr_waiting_for_supplier); > > device_links_write_lock(); > @@ -1488,10 +1417,6 @@ static void device_links_purge(struct device *dev) > if (dev->class == &devlink_class) > return; > > - mutex_lock(&wfs_lock); > - list_del(&dev->links.needs_suppliers); > - mutex_unlock(&wfs_lock); > - > /* > * Delete all of the remaining links from this device to any other > * devices (either consumers or suppliers). > @@ -1561,19 +1486,246 @@ static void fw_devlink_parse_fwtree(struct fwnode_handle *fwnode) > fw_devlink_parse_fwtree(child); > } > > -static void fw_devlink_link_device(struct device *dev) > +/** > + * fw_devlink_create_devlink - Create a device link from a consumer to fwnode > + * @con - Consumer device for the device link > + * @sup_handle - fwnode handle of supplier > + * > + * This function will try to create a device link between the consumer device > + * @con and the supplier device represented by @sup_handle. > + * > + * The supplier has to be provided as a fwnode because incorrect cycles in > + * fwnode links can sometimes cause the supplier device to never be created. > + * This function detects such cases and returns an error if it cannot create a > + * device link from the consumer to a missing supplier. > + * > + * Returns, > + * 0 on successfully creating a device link > + * -EINVAL if the device link cannot be created as expected > + * -EAGAIN if the device link cannot be created right now, but it may be > + * possible to do that in the future > + */ > +static int fw_devlink_create_devlink(struct device *con, > + struct fwnode_handle *sup_handle, u32 flags) > +{ > + struct device *sup_dev; > + int ret = 0; > + > + sup_dev = get_dev_from_fwnode(sup_handle); > + if (sup_dev) { > + /* > + * If this fails, it is due to cycles in device links. Just > + * give up on this link and treat it as invalid. > + */ > + if (!device_link_add(con, sup_dev, flags)) > + ret = -EINVAL; > + > + goto out; > + } > + > + /* > + * DL_FLAG_SYNC_STATE_ONLY doesn't block probing and supports > + * cycles. So cycle detection isn't necessary and shouldn't be > + * done. > + */ > + if (flags & DL_FLAG_SYNC_STATE_ONLY) > + return -EAGAIN; > + > + /* > + * If we can't find the supplier device from its fwnode, it might be > + * due to a cyclic dependency between fwnodes. Some of these cycles can > + * be broken by applying logic. Check for these types of cycles and > + * break them so that devices in the cycle probe properly. > + * > + * If the supplier's parent is dependent on the consumer, then > + * the consumer-supplier dependency is a false dependency. So, > + * treat it as an invalid link. > + */ > + sup_dev = fwnode_get_next_parent_dev(sup_handle); > + if (sup_dev && device_is_dependent(con, sup_dev)) { > + dev_dbg(con, "Not linking to %pfwP - False link\n", > + sup_handle); > + ret = -EINVAL; > + } else { > + /* > + * Can't check for cycles or no cycles. So let's try > + * again later. > + */ > + ret = -EAGAIN; > + } > + > +out: > + put_device(sup_dev); > + return ret; > +} > + > +/** > + * __fw_devlink_link_to_consumers - Create device links to consumers of a device > + * @dev - Device that needs to be linked to its consumers > + * > + * This function looks at all the consumer fwnodes of @dev and creates device > + * links between the consumer device and @dev (supplier). > + * > + * If the consumer device has not been added yet, then this function creates a > + * SYNC_STATE_ONLY link between @dev (supplier) and the closest ancestor device > + * of the consumer fwnode. This is necessary to make sure @dev doesn't get a > + * sync_state() callback before the real consumer device gets to be added and > + * then probed. > + * > + * Once device links are created from the real consumer to @dev (supplier), the > + * fwnode links are deleted. > + */ > +static void __fw_devlink_link_to_consumers(struct device *dev) > +{ > + struct fwnode_handle *fwnode = dev->fwnode; > + struct fwnode_link *link, *tmp; > + > + list_for_each_entry_safe(link, tmp, &fwnode->consumers, s_hook) { > + u32 dl_flags = fw_devlink_get_flags(); > + struct device *con_dev; > + bool own_link = true; > + int ret; > + > + con_dev = get_dev_from_fwnode(link->consumer); > + /* > + * If consumer device is not available yet, make a "proxy" > + * SYNC_STATE_ONLY link from the consumer's parent device to > + * the supplier device. This is necessary to make sure the > + * supplier doesn't get a sync_state() callback before the real > + * consumer can create a device link to the supplier. > + * > + * This proxy link step is needed to handle the case where the > + * consumer's parent device is added before the supplier. > + */ > + if (!con_dev) { > + con_dev = fwnode_get_next_parent_dev(link->consumer); > + /* > + * However, if the consumer's parent device is also the > + * parent of the supplier, don't create a > + * consumer-supplier link from the parent to its child > + * device. Such a dependency is impossible. > + */ > + if (con_dev && > + fwnode_is_ancestor_of(con_dev->fwnode, fwnode)) { > + put_device(con_dev); > + con_dev = NULL; > + } else { > + own_link = false; > + dl_flags = DL_FLAG_SYNC_STATE_ONLY; > + } > + } > + > + if (!con_dev) > + continue; > + > + ret = fw_devlink_create_devlink(con_dev, fwnode, dl_flags); > + put_device(con_dev); > + if (!own_link || ret == -EAGAIN) > + continue; > + > + list_del(&link->s_hook); > + list_del(&link->c_hook); > + kfree(link); > + } > +} > + > +/** > + * __fw_devlink_link_to_suppliers - Create device links to suppliers of a device > + * @dev - The consumer device that needs to be linked to its suppliers > + * @fwnode - Root of the fwnode tree that is used to create device links > + * > + * This function looks at all the supplier fwnodes of fwnode tree rooted at > + * @fwnode and creates device links between @dev (consumer) and all the > + * supplier devices of the entire fwnode tree at @fwnode. > + * > + * The function creates normal (non-SYNC_STATE_ONLY) device links between @dev > + * and the real suppliers of @dev. Once these device links are created, the > + * fwnode links are deleted. When such device links are successfully created, > + * this function is called recursively on those supplier devices. This is > + * needed to detect and break some invalid cycles in fwnode links. See > + * fw_devlink_create_devlink() for more details. > + * > + * In addition, it also looks at all the suppliers of the entire fwnode tree > + * because some of the child devices of @dev that have not been added yet > + * (because @dev hasn't probed) might already have their suppliers added to > + * driver core. So, this function creates SYNC_STATE_ONLY device links between > + * @dev (consumer) and these suppliers to make sure they don't execute their > + * sync_state() callbacks before these child devices have a chance to create > + * their device links. The fwnode links that correspond to the child devices > + * aren't delete because they are needed later to create the device links > + * between the real consumer and supplier devices. > + */ > +static void __fw_devlink_link_to_suppliers(struct device *dev, > + struct fwnode_handle *fwnode) > { > - int fw_ret; > + bool own_link = (dev->fwnode == fwnode); > + struct fwnode_link *link, *tmp; > + struct fwnode_handle *child = NULL; > + u32 dl_flags; > + > + if (own_link) > + dl_flags = fw_devlink_get_flags(); > + else > + dl_flags = DL_FLAG_SYNC_STATE_ONLY; > > - device_link_add_missing_supplier_links(); > + list_for_each_entry_safe(link, tmp, &fwnode->suppliers, c_hook) { > + int ret; > + struct device *sup_dev; > + struct fwnode_handle *sup = link->supplier; > + > + ret = fw_devlink_create_devlink(dev, sup, dl_flags); > + if (!own_link || ret == -EAGAIN) > + continue; > > - if (fw_devlink_flags && fwnode_has_op(dev->fwnode, add_links)) { > - fw_ret = fwnode_call_int_op(dev->fwnode, add_links, dev); > - if (fw_ret == -ENODEV && !fw_devlink_is_permissive()) > - device_link_wait_for_mandatory_supplier(dev); > - else if (fw_ret) > - device_link_wait_for_optional_supplier(dev); > + list_del(&link->s_hook); > + list_del(&link->c_hook); > + kfree(link); > + > + /* If no device link was created, nothing more to do. */ > + if (ret) > + continue; > + > + /* > + * If a device link was successfully created to a supplier, we > + * now need to try and link the supplier to all its suppliers. > + * > + * This is needed to detect and delete false dependencies in > + * fwnode links that haven't been converted to a device link > + * yet. See comments in fw_devlink_create_devlink() for more > + * details on the false dependency. > + * > + * Without deleting these false dependencies, some devices will > + * never probe because they'll keep waiting for their false > + * dependency fwnode links to be converted to device links. > + */ > + sup_dev = get_dev_from_fwnode(sup); > + __fw_devlink_link_to_suppliers(sup_dev, sup_dev->fwnode); > + put_device(sup_dev); > } > + > + /* > + * Make "proxy" SYNC_STATE_ONLY device links to represent the needs of > + * all the descendants. This proxy link step is needed to handle the > + * case where the supplier is added before the consumer's parent device > + * (@dev). > + */ > + while ((child = fwnode_get_next_available_child_node(fwnode, child))) > + __fw_devlink_link_to_suppliers(dev, child); > +} > + > +static void fw_devlink_link_device(struct device *dev) > +{ > + struct fwnode_handle *fwnode = dev->fwnode; > + > + if (!fw_devlink_flags) > + return; > + > + fw_devlink_parse_fwtree(fwnode); > + > + mutex_lock(&fwnode_link_lock); > + __fw_devlink_link_to_consumers(dev); > + __fw_devlink_link_to_suppliers(dev, fwnode); > + mutex_unlock(&fwnode_link_lock); > } > > /* Device links support end. */ > @@ -2431,7 +2583,6 @@ void device_initialize(struct device *dev) > #endif > INIT_LIST_HEAD(&dev->links.consumers); > INIT_LIST_HEAD(&dev->links.suppliers); > - INIT_LIST_HEAD(&dev->links.needs_suppliers); > INIT_LIST_HEAD(&dev->links.defer_sync); > dev->links.status = DL_DEV_NO_DRIVER; > } > diff --git a/include/linux/device.h b/include/linux/device.h > index 1e771ea4dca6..89bb8b84173e 100644 > --- a/include/linux/device.h > +++ b/include/linux/device.h > @@ -351,18 +351,13 @@ enum dl_dev_state { > * struct dev_links_info - Device data related to device links. > * @suppliers: List of links to supplier devices. > * @consumers: List of links to consumer devices. > - * @needs_suppliers: Hook to global list of devices waiting for suppliers. > * @defer_sync: Hook to global list of devices that have deferred sync_state. > - * @need_for_probe: If needs_suppliers is on a list, this indicates if the > - * suppliers are needed for probe or not. > * @status: Driver status information. > */ > struct dev_links_info { > struct list_head suppliers; > struct list_head consumers; > - struct list_head needs_suppliers; > struct list_head defer_sync; > - bool need_for_probe; > enum dl_dev_state status; > }; >
