Hello, On Sat, 22 Oct 2022, Jiri Wiesner wrote: > On Sun, Oct 16, 2022 at 03:21:10PM +0300, Julian Anastasov wrote: > > > It is not a problem to add some wait_event_idle_timeout > > calls to sleep before/between tests if the system is so busy > > on boot that it can even disturb our tests with disabled BHs. > > That is definitely not the case. When I get the underestimated max chain length: > > [ 130.699910][ T2564] IPVS: Registered protocols (TCP, UDP, SCTP, AH, ESP) > > [ 130.707580][ T2564] IPVS: Connection hash table configured (size=4096, memory=32Kbytes) > > [ 130.716633][ T2564] IPVS: ipvs loaded. > > [ 130.723423][ T2570] IPVS: [wlc] scheduler registered. > > [ 130.731071][ T477] IPVS: starting estimator thread 0... > > [ 130.737169][ T2571] IPVS: calc: chain_max=12, single est=7379ns, diff=7379, loops=1, ntest=3 > > [ 130.746673][ T2571] IPVS: dequeue: 81ns > > [ 130.750988][ T2571] IPVS: using max 576 ests per chain, 28800 per kthread > > [ 132.678012][ T2571] IPVS: tick time: 5930ns for 64 CPUs, 2 ests, 1 chains, chain_max=576 > the system is idle, not running any workload and the booting sequence has finished. Hm, can it be some cpufreq/ondemand issue causing this? Test can be affected by CPU speed. > > We have 2 seconds > > for the first tests, so we can add some gaps. If you want to > > test it you can add some schedule_timeout(HZ/10) between the > > 3 tests (total 300ms delay of the real estimation). > > schedule_timeout(HZ/10) does not make the thread sleep - the function returns 25, which is the value of the timeout passed to it. msleep(100) works, though: Hm, yes. Due to the RUNNING state, msleep works for testing. > > kworker/0:0-eve 7 [000] 70.223673: sched:sched_waking: comm=ipvs-e:0:0 pid=8927 prio=120 target_cpu=051 > > swapper 0 [051] 70.223770: sched:sched_switch: prev_comm=swapper/51 prev_pid=0 prev_prio=120 prev_state=R ==> next_comm=ipvs-e:0:0 next_pid=8927 next_prio=120 > > ipvs-e:0:0 8927 [051] 70.223786: sched:sched_stat_runtime: comm=ipvs-e:0:0 pid=8927 runtime=26009 [ns] vruntime=2654620258 [ns] > > ipvs-e:0:0 8927 [051] 70.223787: sched:sched_switch: prev_comm=ipvs-e:0:0 prev_pid=8927 prev_prio=120 prev_state=D ==> next_comm=swapper/51 next_pid=0 next_prio=120 > > swapper 0 [051] 70.331221: sched:sched_waking: comm=ipvs-e:0:0 pid=8927 prio=120 target_cpu=051 > > swapper 0 [051] 70.331234: sched:sched_switch: prev_comm=swapper/51 prev_pid=0 prev_prio=120 prev_state=R ==> next_comm=ipvs-e:0:0 next_pid=8927 next_prio=120 > > ipvs-e:0:0 8927 [051] 70.331241: sched:sched_stat_runtime: comm=ipvs-e:0:0 pid=8927 runtime=11064 [ns] vruntime=2654631322 [ns] > > ipvs-e:0:0 8927 [051] 70.331242: sched:sched_switch: prev_comm=ipvs-e:0:0 prev_pid=8927 prev_prio=120 prev_state=D ==> next_comm=swapper/51 next_pid=0 next_prio=120 > > swapper 0 [051] 70.439220: sched:sched_waking: comm=ipvs-e:0:0 pid=8927 prio=120 target_cpu=051 > > swapper 0 [051] 70.439235: sched:sched_switch: prev_comm=swapper/51 prev_pid=0 prev_prio=120 prev_state=R ==> next_comm=ipvs-e:0:0 next_pid=8927 next_prio=120 > > ipvs-e:0:0 8927 [051] 70.439242: sched:sched_stat_runtime: comm=ipvs-e:0:0 pid=8927 runtime=10324 [ns] vruntime=2654641646 [ns] > > ipvs-e:0:0 8927 [051] 70.439243: sched:sched_switch: prev_comm=ipvs-e:0:0 prev_pid=8927 prev_prio=120 prev_state=D ==> next_comm=swapper/51 next_pid=0 next_prio=120 > > swapper 0 [051] 70.547220: sched:sched_waking: comm=ipvs-e:0:0 pid=8927 prio=120 target_cpu=051 > > swapper 0 [051] 70.547235: sched:sched_switch: prev_comm=swapper/51 prev_pid=0 prev_prio=120 prev_state=R ==> next_comm=ipvs-e:0:0 next_pid=8927 next_prio=120 > > ipvs-e:0:0 8927 [051] 70.556717: sched:sched_stat_runtime: comm=ipvs-e:0:0 pid=8927 runtime=9486028 [ns] vruntime=2664127674 [ns] > > ipvs-e:0:0 8927 [051] 70.561134: sched:sched_waking: comm=in:imklog pid=2210 prio=120 target_cpu=039 > > ipvs-e:0:0 8927 [051] 70.561136: sched:sched_waking: comm=systemd-journal pid=1161 prio=120 target_cpu=001 > > ipvs-e:0:0 8927 [051] 70.561138: sched:sched_stat_runtime: comm=ipvs-e:0:0 pid=8927 runtime=4421889 [ns] vruntime=2668549563 [ns] > > ipvs-e:0:0 8927 [051] 70.568867: sched:sched_waking: comm=in:imklog pid=2210 prio=120 target_cpu=039 > > ipvs-e:0:0 8927 [051] 70.568868: sched:sched_waking: comm=systemd-journal pid=1161 prio=120 target_cpu=001 > > ipvs-e:0:0 8927 [051] 70.568870: sched:sched_stat_runtime: comm=ipvs-e:0:0 pid=8927 runtime=7731843 [ns] vruntime=2676281406 [ns] > > ipvs-e:0:0 8927 [051] 70.568878: sched:sched_stat_runtime: comm=ipvs-e:0:0 pid=8927 runtime=8169 [ns] vruntime=2676289575 [ns] > > ipvs-e:0:0 8927 [051] 70.568880: sched:sched_switch: prev_comm=ipvs-e:0:0 prev_pid=8927 prev_prio=120 prev_state=I ==> next_comm=swapper/51 next_pid=0 next_prio=120 > > swapper 0 [051] 70.611220: sched:sched_waking: comm=ipvs-e:0:0 pid=8927 prio=120 target_cpu=051 > > swapper 0 [051] 70.611239: sched:sched_switch: prev_comm=swapper/51 prev_pid=0 prev_prio=120 prev_state=R ==> next_comm=ipvs-e:0:0 next_pid=8927 next_prio=120 > > ipvs-e:0:0 8927 [051] 70.611243: sched:sched_stat_runtime: comm=ipvs-e:0:0 pid=8927 runtime=10196 [ns] vruntime=2676299771 [ns] > > ipvs-e:0:0 8927 [051] 70.611245: sched:sched_switch: prev_comm=ipvs-e:0:0 prev_pid=8927 prev_prio=120 prev_state=I ==> next_comm=swapper/51 next_pid=0 next_prio=120 > > swapper 0 [051] 70.651220: sched:sched_waking: comm=ipvs-e:0:0 pid=8927 prio=120 target_cpu=051 > > swapper 0 [051] 70.651239: sched:sched_switch: prev_comm=swapper/51 prev_pid=0 prev_prio=120 prev_state=R ==> next_comm=ipvs-e:0:0 next_pid=8927 next_prio=120 > > ipvs-e:0:0 8927 [051] 70.651243: sched:sched_stat_runtime: comm=ipvs-e:0:0 pid=8927 runtime=10985 [ns] vruntime=2676310756 [ns] > > ipvs-e:0:0 8927 [051] 70.651245: sched:sched_switch: prev_comm=ipvs-e:0:0 prev_pid=8927 prev_prio=120 prev_state=I ==> next_comm=swapper/51 next_pid=0 next_prio=120 > After adding msleep(), I have rebooted 3 times and added a service. The max chain length was always at the optimal value - around 35. I think more tests on other architecture would be needed. I can test on ARM next week. Then I'll add such pause between the tests in the next version. Let me know if you see any problems with different NUMA configurations due to the chosen cache_factor. For now, I don't have a good idea how to change the algorithm to use feedback from real estimation without complicating it further. The only way to safely change the chain limit immediately is as it is implemented now: stop tasks, reallocate, relink and start tasks. If we want to do it without stopping tasks, it violates the RCU-list rules: we can not relink entries without RCU grace period. So, we have the following options: 1. Use this algorithm if it works in different configurations 2. Use this algorithm but trigger recalculation (stop, relink, start) if a kthread with largest number of entries detects big difference for chain_max 3. Implement different data structure to store estimators Currently, the problem comes from the fact that we store estimators in chains. We should cut these chains if chain_max should be reduced. Second option would be to put estimators in ptr arrays but then there is a problem with fragmentation on add/del and as result, slower walking. Arrays probably can allow the limit used for cond_resched, that is now chain_max, to be applied without relinking entries. To summarize, the goals are: - allocations for linking estimators should not be large (many pages), prefer to allocate in small steps - due to RCU-list rules we can not relink without task stop+start - real estimation should give more accurate values for the parameters: cond_resched rate - fast lock-free walking of estimators by kthreads - fast add/del of estimators, by netlink - if possible, a way to avoid estimations for estimators that are not updated, eg. added service/dest but no traffic - fast and safe way to apply a new chain_max or similar parameter for cond_resched rate. If possible, without relinking. stop+start can be slow too. While finishing this posting, I'm investigating the idea to use structures without chains (no relinking), without chain_len, tick_len, etc. But let me first see if such idea can work... Regards -- Julian Anastasov <ja@xxxxxx>
