> > On Tue, Dec 03, 2024 at 07:23:44PM +0800, zuoze wrote: > > > We have implemented host-guest communication based on the TUN device > > > using XSK[1]. The hardware is a Kunpeng 920 machine (ARM architecture), > > > and the operating system is based on the 6.6 LTS version with kernel > > > version 6.6. The specific stack for hotspot collection is as follows: > > > > > > - 100.00% 0.00% vhost-12384 [unknown] [k] 0000000000000000 > > > - ret_from_fork > > > - 99.99% vhost_task_fn > > > - 99.98% 0xffffdc59f619876c > > > - 98.99% handle_rx_kick > > > - 98.94% handle_rx > > > - 94.92% tun_recvmsg > > > - 94.76% tun_do_read > > > - 94.62% tun_put_user_xdp_zc > > > - 63.53% __check_object_size > > > - 63.49% __check_object_size.part.0 > > > find_vmap_area > > > - 30.02% _copy_to_iter > > > __arch_copy_to_user > > > - 2.27% get_rx_bufs > > > - 2.12% vhost_get_vq_desc > > > 1.49% __arch_copy_from_user > > > - 0.89% peek_head_len > > > 0.54% xsk_tx_peek_desc > > > - 0.68% vhost_add_used_and_signal_n > > > - 0.53% eventfd_signal > > > eventfd_signal_mask > > > - 0.94% handle_tx_kick > > > - 0.94% handle_tx > > > - handle_tx_copy > > > - 0.59% vhost_tx_batch.constprop.0 > > > 0.52% tun_sendmsg > > > > > > It can be observed that most of the overhead is concentrated in the > > > find_vmap_area function. > > > > > I see. Yes, it is pretty contented, since you run the v6.6 kernel. There > > was a work that tends to improve it to mitigate a vmap lock contention. > > See it here: https://lwn.net/Articles/956590/ > > > > The work was taken in the v6.9 kernel: > > > > <snip> > > commit 38f6b9af04c4b79f81b3c2a0f76d1de94b78d7bc > > Author: Uladzislau Rezki (Sony) <urezki@xxxxxxxxx> > > Date: Tue Jan 2 19:46:23 2024 +0100 > > > > mm: vmalloc: add va_alloc() helper > > > > Patch series "Mitigate a vmap lock contention", v3. > > > > 1. Motivation > > ... > > <snip> > > > > Could you please try the v6.9 kernel on your setup? > > > > How to solve it, probably, it can be back-ported to the v6.6 kernel. > > All the vmalloc-related optimizations have already been merged into 6.6, > including the set of optimization patches you suggested. Thank you very > much for your input. > Do you mean that the perf-data that you posted to this email thread, contains the: Patch series "Mitigate a vmap lock contention", v3. from the v6.9? and you still see a high contention? Thank you! -- Uladzislau Rezki
