Hello, On Fri, 12 Aug 2022, Jiri Wiesner wrote: > The calculation of rate estimates for IPVS services and destinations will > cause an increase in scheduling latency to hundreds of milliseconds when > the number of estimators reaches tens of thousands or more. This issue has > been reported upstream [1]. Design changes to the algorithm to compute the > estimates were proposed in the same email thread. > > By implementing some of the proposed design changes, this patch seeks to > address the latency issue by dividing the estimators into groups for which > estimates are calculated in a 2-second interval (same as before). Each of > the groups is processed once in each 2-second interval. Instead of > allocating an array of lists, groups are identified by their group_id, > which has the advantage that estimators can stay in the same list to which > they have been added by ip_vs_start_estimator(). The implementation of > estimator grouping is able to scale up with an increasing number of > estimators as well as scale down when estimators are being removed. > The changes to group size can be monitored with dynamic debugging: > echo 'file net/netfilter/ipvs/ip_vs_est.c +pfl' >> /sys/kernel/debug/dynamic_debug/control > > Rebalacing of estimator groups is implemented and can be triggered only > after all the calculations for a 2-second interval have finished. After a > limit is exceeded, adding or removing estimators will triger rebalacing, > which will cause estimates to be inaccurate in the next 2-second interval. > For example, removing estimators that results in the removal of an entire > group will shorten the time interval used for computing rates, which will > lead to the rates being underestimated in the next 2-second interval. > > Testing was carried out on a 2-socket machine with Intel Xeon Gold 6326 > CPUs (64 logical CPUs). Tests with up to 600,000 estimators were > successfully completed. The expectation is that, given the current default > limits, the implementation can handle 150,000 estimators on most machines > in use today. In a test with 100,000 estimators, the default group size of > 1024 estimators resulted in the processing time for one group to be circa > 2.3 milliseconds and a timer period of 5 jiffies. Despite estimators being > added or removed throughout most of the test, the overhead of > ip_vs_estimator_rebalance() was less than 10% of the overhead of > estimation_timer(): > 7.66% 124093 swapper [kernel.kallsyms] [k] intel_idle > 2.86% 14296 ipvsadm [kernel.kallsyms] [k] native_queued_spin_lock_slowpath > 2.64% 16827 ipvsadm [kernel.kallsyms] [k] ip_vs_genl_parse_service > 2.15% 18457 ipvsadm libc-2.31.so [.] _dl_addr > 2.08% 4562 ipvsadm [kernel.kallsyms] [k] ip_vs_genl_dump_services > 2.06% 18326 ipvsadm ld-2.31.so [.] do_lookup_x > 1.78% 17251 swapper [kernel.kallsyms] [k] estimation_timer > ... > 0.14% 855 swapper [kernel.kallsyms] [k] ip_vs_estimator_rebalance > > The intention is to develop this RFC patch into a short series addressing > the design changes proposed in [1]. Also, after moving the rate estimation > out of softirq context, the whole estimator list could be processed > concurrently - more than one work item would be used. > > [1] https://lore.kernel.org/netdev/D25792C1-1B89-45DE-9F10-EC350DC04ADC@xxxxxxxxx > > Signed-off-by: Jiri Wiesner <jwiesner@xxxxxxx> Other developers tried solutions with workqueues but so far we don't see any results. Give me some days, may be I can come up with solution that uses kthread(s) to allow later nice/cpumask cfg tuning and to avoid overload of the system workqueues. Regards -- Julian Anastasov <ja@xxxxxx>
