On Thu, Sep 08, 2022 at 05:35:21PM +0200, Jiri Wiesner wrote:
> There is an alternative design where you could increase kd->est_max_count for all kthreads once all of the available kthreads have kd->est_max_count estimators. Nevertheless, there would also have to be a limit to the value of kd->est_max_count. Imagine the estimation during a single tick would take so long that the gap variable in ip_vs_estimation_kthread() would become negative. You would need to have circa 250,000 estimators per kthread. Since you are already measuring the timeout you need for schedule_timeout() in ip_vs_estimation_kthread(), it should be possible to set the kd->est_max_count limit based on the maximum processing time per chain. It could be half a IPVS_EST_TICK, for example.
>
> But it seems to me that the alternative design - increasing kd->est_max_count - should have some support in what is used in production. Are there servers with more than 983,040 estimators (which would be IPVS_EST_MAX_COUNT * 30 kthreads) or even one third of that?
I did some profiling (but could have just looked at top, actually) of a kthread with IPVS_EST_MAX_COUNT estimators for 100 seconds:
# Samples: 4K of event 'bus-cycles'
# Event count (approx.): 125024900
# Overhead Period Command Shared Object Symbol
# ........ ............ ............... ................. .........................................
#
76.44% 95570475 ipvs-e:0:0 [kernel.kallsyms] [k] ip_vs_estimation_kthread
8.75% 10935925 ipvs-e:0:0 [kernel.kallsyms] [k] _find_next_bit
3.18% 3978975 swapper [kernel.kallsyms] [k] intel_idle
1.00% 1251250 ipvs-e:0:0 [kernel.kallsyms] [k] _raw_spin_lock_bh
0.36% 450450 swapper [kernel.kallsyms] [k] _raw_spin_lock
0.36% 450450 swapper [kernel.kallsyms] [k] update_rq_clock
The bus-cycles event on this particular machine makes 25,000,000 events per second. Based on the period in the profile, the CPU utilization for various functions is:
ip_vs_estimation_kthread: 95570475/100/25000000*100 = 3.82%
_find_next_bit: 10935925/100/25000000*100 = 0.44%
_raw_spin_lock_bh: 1251250/100/25000000*100 = 0.05%
The kthread could definitely utilize the CPU more, which is an argument for increasing kd->est_max_count.
--
Jiri Wiesner
SUSE Labs
