On Tue, Dec 03, 2024 at 09:30:09PM +0800, Kefeng Wang wrote: > > > On 2024/12/3 21:10, zuoze wrote: > > > > > > 在 2024/12/3 20:39, Uladzislau Rezki 写道: > > > On Tue, Dec 03, 2024 at 07:23:44PM +0800, zuoze wrote: > > > > We have implemented host-guest communication based on the TUN device > > > > using XSK[1]. The hardware is a Kunpeng 920 machine (ARM architecture), > > > > and the operating system is based on the 6.6 LTS version with kernel > > > > version 6.6. The specific stack for hotspot collection is as follows: > > > > > > > > - 100.00% 0.00% vhost-12384 [unknown] [k] 0000000000000000 > > > > - ret_from_fork > > > > - 99.99% vhost_task_fn > > > > - 99.98% 0xffffdc59f619876c > > > > - 98.99% handle_rx_kick > > > > - 98.94% handle_rx > > > > - 94.92% tun_recvmsg > > > > - 94.76% tun_do_read > > > > - 94.62% tun_put_user_xdp_zc > > > > - 63.53% __check_object_size > > > > - 63.49% __check_object_size.part.0 > > > > find_vmap_area > > > > - 30.02% _copy_to_iter > > > > __arch_copy_to_user > > > > - 2.27% get_rx_bufs > > > > - 2.12% vhost_get_vq_desc > > > > 1.49% __arch_copy_from_user > > > > - 0.89% peek_head_len > > > > 0.54% xsk_tx_peek_desc > > > > - 0.68% vhost_add_used_and_signal_n > > > > - 0.53% eventfd_signal > > > > eventfd_signal_mask > > > > - 0.94% handle_tx_kick > > > > - 0.94% handle_tx > > > > - handle_tx_copy > > > > - 0.59% vhost_tx_batch.constprop.0 > > > > 0.52% tun_sendmsg > > > > > > > > It can be observed that most of the overhead is concentrated in the > > > > find_vmap_area function. > > > > > > > I see. Yes, it is pretty contented, since you run the v6.6 kernel. There > > > was a work that tends to improve it to mitigate a vmap lock contention. > > > See it here: https://lwn.net/Articles/956590/ > > > > > > The work was taken in the v6.9 kernel: > > > > > > <snip> > > > commit 38f6b9af04c4b79f81b3c2a0f76d1de94b78d7bc > > > Author: Uladzislau Rezki (Sony) <urezki@xxxxxxxxx> > > > Date: Tue Jan 2 19:46:23 2024 +0100 > > > > > > mm: vmalloc: add va_alloc() helper > > > > > > Patch series "Mitigate a vmap lock contention", v3. > > > > > > 1. Motivation > > > ... > > > <snip> > > > > > > Could you please try the v6.9 kernel on your setup? > > > > > > How to solve it, probably, it can be back-ported to the v6.6 kernel. > > > > All the vmalloc-related optimizations have already been merged into 6.6, > > including the set of optimization patches you suggested. Thank you very > > much for your input. > > > > It is unclear, we have backported the vmalloc optimization into our 6.6 > kernel before, so the above stack already with those patches and even > with those optimization, the find_vmap_area() is still the hotpots. > > Could you please check that all below patches are in your v6.6 kernel? <snip> commit 8be4d46e12af32342569840d958272dbb3be3f4c Author: Uladzislau Rezki (Sony) <urezki@xxxxxxxxx> Date: Wed Jan 24 19:09:20 2024 +0100 mm: vmalloc: refactor vmalloc_dump_obj() function commit 15e02a39fb6b43f37100563c6a246252d5d1e6da Author: Uladzislau Rezki (Sony) <urezki@xxxxxxxxx> Date: Wed Jan 24 19:09:19 2024 +0100 mm: vmalloc: improve description of vmap node layer commit 7679ba6b36dbb300b757b672d6a32a606499e14b Author: Uladzislau Rezki (Sony) <urezki@xxxxxxxxx> Date: Tue Jan 2 19:46:33 2024 +0100 mm: vmalloc: add a shrinker to drain vmap pools commit 8f33a2ff307248c3e55a7696f60b3658b28edb57 Author: Uladzislau Rezki (Sony) <urezki@xxxxxxxxx> Date: Tue Jan 2 19:46:32 2024 +0100 mm: vmalloc: set nr_nodes based on CPUs in a system commit 8e1d743f2c2671aa54f6f91a2b33823f92512870 Author: Uladzislau Rezki (Sony) <urezki@xxxxxxxxx> Date: Tue Jan 2 19:46:31 2024 +0100 mm: vmalloc: support multiple nodes in vmallocinfo commit 53becf32aec1c8049b854f0c31a11df5ed75df6f Author: Uladzislau Rezki (Sony) <urezki@xxxxxxxxx> Date: Tue Jan 2 19:46:30 2024 +0100 mm: vmalloc: support multiple nodes in vread_iter commit 96aa8437d169b8e030a98e2b74fd9a8ee9d3be7e Author: Uladzislau Rezki (Sony) <urezki@xxxxxxxxx> Date: Fri Feb 2 20:06:28 2024 +0100 mm: vmalloc: add a scan area of VA only once commit 72210662c5a2b6005f6daea7fe293a0dc573e1a5 Author: Uladzislau Rezki (Sony) <urezki@xxxxxxxxx> Date: Tue Jan 2 19:46:29 2024 +0100 mm: vmalloc: offload free_vmap_area_lock lock commit 282631cb2447318e2a55b41a665dbe8571c46d70 Author: Uladzislau Rezki (Sony) <urezki@xxxxxxxxx> Date: Tue Jan 2 19:46:28 2024 +0100 mm: vmalloc: remove global purge_vmap_area_root rb-tree commit 55c49fee57af99f3c663e69dedc5b85e691bbe50 Author: Baoquan He <bhe@xxxxxxxxxx> Date: Tue Jan 2 19:46:27 2024 +0100 mm/vmalloc: remove vmap_area_list commit d093602919ad5908532142a048539800fa94a0d1 Author: Uladzislau Rezki (Sony) <urezki@xxxxxxxxx> Date: Tue Jan 2 19:46:26 2024 +0100 mm: vmalloc: remove global vmap_area_root rb-tree commit 7fa8cee003166ef6db0bba70d610dbf173543811 Author: Uladzislau Rezki (Sony) <urezki@xxxxxxxxx> Date: Tue Jan 2 19:46:25 2024 +0100 mm: vmalloc: move vmap_init_free_space() down in vmalloc.c commit 5b75b8e1b9040b34f43809b1948eaa4e83e39112 Author: Uladzislau Rezki (Sony) <urezki@xxxxxxxxx> Date: Tue Jan 2 19:46:24 2024 +0100 mm: vmalloc: rename adjust_va_to_fit_type() function commit 38f6b9af04c4b79f81b3c2a0f76d1de94b78d7bc Author: Uladzislau Rezki (Sony) <urezki@xxxxxxxxx> Date: Tue Jan 2 19:46:23 2024 +0100 mm: vmalloc: add va_alloc() helper <snip> Thank you! -- Uladzislau Rezki
