Andrei Vagin <avagin@xxxxxxxxxxxxx> writes: > On Thu, Oct 13, 2016 at 10:49:38AM -0500, Eric W. Biederman wrote: >> Andrei Vagin <avagin@xxxxxxxxxx> writes: >> >> > From: Andrey Vagin <avagin@xxxxxxxxxx> >> > >> > The operation of destroying netns is heavy and it is executed under >> > net_mutex. If many namespaces are destroyed concurrently, net_mutex can >> > be locked for a long time. It is impossible to create a new netns during >> > this period of time. >> >> This may be the right approach or at least the right approach to bound >> net_mutex hold times but I have to take exception to calling network >> namespace cleanup heavy. >> >> The only particularly time consuming operation I have ever found are calls to >> synchronize_rcu/sycrhonize_sched/synchronize_net. > > I booted the kernel with maxcpus=1, in this case these functions work > very fast and the problem is there any way. > > Accoding to perf, we spend a lot of time in kobject_uevent: > > - 99.96% 0.00% kworker/u4:1 [kernel.kallsyms] [k] unregister_netdevice_many ▒ > - unregister_netdevice_many ◆ > - 99.95% rollback_registered_many ▒ > - 99.64% netdev_unregister_kobject ▒ > - 33.43% netdev_queue_update_kobjects ▒ > - 33.40% kobject_put ▒ > - kobject_release ▒ > + 33.37% kobject_uevent ▒ > + 0.03% kobject_del ▒ > + 0.03% sysfs_remove_group ▒ > - 33.13% net_rx_queue_update_kobjects ▒ > - kobject_put ▒ > - kobject_release ▒ > + 33.11% kobject_uevent ▒ > + 0.01% kobject_del ▒ > 0.00% rx_queue_release ▒ > - 33.08% device_del ▒ > + 32.75% kobject_uevent ▒ > + 0.17% device_remove_attrs ▒ > + 0.07% dpm_sysfs_remove ▒ > + 0.04% device_remove_class_symlinks ▒ > + 0.01% kobject_del ▒ > + 0.01% device_pm_remove ▒ > + 0.01% sysfs_remove_file_ns ▒ > + 0.00% klist_del ▒ > + 0.00% driver_deferred_probe_del ▒ > 0.00% cleanup_glue_dir.isra.14.part.15 ▒ > 0.00% to_acpi_device_node ▒ > 0.00% sysfs_remove_group ▒ > 0.00% klist_del ▒ > 0.00% device_remove_attrs ▒ > + 0.26% call_netdevice_notifiers_info ▒ > + 0.04% rtmsg_ifinfo_build_skb ▒ > + 0.01% rtmsg_ifinfo_send ▒ > 0.00% dev_uc_flush ▒ > 0.00% netif_reset_xps_queues_gt > > Someone can listen these uevents, so we can't stop sending them without > breaking backward compatibility. We can try to optimize > kobject_uevent... Oh that is a surprise. We can definitely skip genenerating uevents for network namespaces that are exiting because by definition no one can see those network namespaces. If a socket existed that could see those uevents it would hold a reference to the network namespace and as such the network namespace could not exit. That sounds like it is worth investigating a little more deeply. I am surprised that allocation and freeing is so heavy we are spending lots of time doing that. On the other hand kobj_bcast_filter is very dumb and very late so I expect something can be moved earlier and make that code cheaper with the tiniest bit of work. Eric _______________________________________________ Containers mailing list Containers@xxxxxxxxxxxxxxxxxxxxxxxxxx https://lists.linuxfoundation.org/mailman/listinfo/containers
