On 17-Feb-23 11:33 AM, Huang, Ying wrote: > Bharata B Rao <bharata@xxxxxxx> writes: > >> On 14-Feb-23 10:25 AM, Bharata B Rao wrote: >>> On 13-Feb-23 12:00 PM, Huang, Ying wrote: >>>>> I have a microbenchmark where two sets of threads bound to two >>>>> NUMA nodes access the two different halves of memory which is >>>>> initially allocated on the 1st node. >>>>> >>>>> On a two node Zen4 system, with 64 threads in each set accessing >>>>> 8G of memory each from the initial allocation of 16G, I see that >>>>> IBS driven NUMA balancing (i,e., this patchset) takes 50% less time >>>>> to complete a fixed number of memory accesses. This could well >>>>> be the best case and real workloads/benchmarks may not get this much >>>>> uplift, but it does show the potential gain to be had. >>>> >>>> Can you find a way to show the overhead of the original implementation >>>> and your method? Then we can compare between them? Because you think >>>> the improvement comes from the reduced overhead. >>> >>> Sure, will measure the overhead. >> >> I used ftrace function_graph tracer to measure the amount of time (in us) >> spent in fault handling and task_work handling in both the methods when >> the above mentioned benchmark was running. >> >> Default IBS >> Fault handling 29879668.71 1226770.84 >> Task work handling 24878.894 10635593.82 >> Sched switch handling 78159.846 >> >> Total 29904547.6 11940524.51 > > Thanks! You have shown the large overhead difference between the > original method and your method. Can you show the number of the pages > migrated too? I think the overhead / page can be a good overhead > indicator too. > > Can it be translated to the performance improvement? Per my > understanding, the total overhead is small compared with total run time. I captured some of the numbers that you wanted for two different runs. The first case shows the data for a short run (less number of memory access iterations) and the second one is for a long run (more number of iterations) Short-run ========= Time taken or overhead (us) for fault, task_work and sched_switch handling Default IBS Fault handling 29017953.99 1196828.67 Task work handling 10354.40 10356778.53 Sched switch handling 56572.21 Total overhead 29028308.39 11610179.41 Benchmark score(us) 194050290 53963650 numa_pages_migrated 2097256 662755 Overhead / page 13.84 17.51 Pages migrated per sec 72248.64 57083.95 Default ------- Total Min Max Avg do_numa_page 29017953.99 0.1 307.63 15.97 task_numa_work 10354.40 2.86 4573.60 175.50 Total 29028308.39 IBS --- Total Min Max Avg ibs_overflow_handler 1196828.67 0.15 100.28 1.26 task_ibs_access_work 10356778.53 0.21 10504.14 28.42 hw_access_sched_in 56572.21 0.15 16.94 1.45 Total 11610179.41 Long-run ======== Time taken or overhead (us) for fault, task_work and sched_switch handling Default IBS Fault handling 27437756.73 901406.37 Task work handling 1741.66 4902935.32 Sched switch handling 100590.33 Total overhead 27439498.38 5904932.02 Benchmark score(us) 306786210.0 153422489.0 numa_pages_migrated 2097218 1746099 Overhead / page 13.08 3.38 Pages migrated per sec 6836.08 11380.98 Default ------- Total Min Max Avg do_numa_page 27437756.73 0.08 363.475 15.03 task_numa_work 1741.66 3.294 1200.71 42.48 Total 27439498.38 IBS --- Total Min Max Avg ibs_overflow_handler 901406.37 0.15 95.51 1.06 task_ibs_access_work 4902935.32 0.22 11013.68 9.64 hw_access_sched_in 100590.33 0.14 91.97 1.52 Total 5904932.02 Regards, Bharata.
