On Sun, Apr 23, 2023 at 09:55:24AM +0800, Hou Tao wrote: > > > >> ./bench htab-mem --use-case $name --max-entries 16384 \ > >> --full 50 -d 7 -w 3 --producers=8 --prod-affinity=0-7 > >> > >> | name | loop (k/s) | average memory (MiB) | peak memory (MiB) | > >> | -- | -- | -- | -- | > >> | no_op | 1129 | 1.15 | 1.15 | > >> | overwrite | 24.37 | 2.07 | 2.97 | > >> | batch_add_batch_del | 10.58 | 2.91 | 3.36 | > >> | add_del_on_diff_cpu | 13.14 | 380.66 | 633.99 | > > large mem for diff_cpu case needs to be investigated. > The main reason is that tasks-trace RCU GP is slow and there is only one > inflight free callback, so the CPUs which only do element addition will allocate > new memory from slab continuously and the CPUs which only do element deletion > will free these elements continuously through call_tasks_trace_rcu(), but due to > the slowness of tasks-trace RCU GP, these freed elements could not be freed back > to slab subsystem timely. I see. Now it makes sense. It's slow call_tasks_trace_rcu and not at all "memory can never be reused." Please explain things clearly in commit log. > >> +{ > >> + __u64 *value; > >> + > >> + if (ctx->from >= ctx->max) > >> + return 1; > >> + > >> + value = bpf_map_lookup_elem(&array, &ctx->from); > >> + if (value) > >> + bpf_map_update_elem(&htab, &ctx->from, value, flags); > > What is a point of doing lookup from giant array of en element with zero value > > to copy it into htab? > > Why not to use single zero inited elem for all htab ops? > I want to check how does the different size of value effect the benchmark > result, so I choose a variable-size value. Not following. All elements of the array have the same size. Are you saying you were not able to figure out how to supply a single 'value' of variable size? Try array of max_entries=1. Do not do unnecessary and confusing bpf_map_lookup_elem(&array, &ctx->from);. > > > > Each loop will run 16k times and every time you step += 4. > > So 3/4 of these 16k runs it will be hitting if (ctx->from >= ctx->max) condition. > > What are you measuring? > As explained in the commit message, I am trying to let different deletion and > deletion CPU pairs operate on the different subsets of hash-table elements. > Assuming there are 16 elements in the htab and there are 8 CPUs and 8 threads, > the following is the operation subset for each CPU: > > CPU 0: 0 4 8 12 (do deletion) > CPU 1: 0 4 8 12 (do addition) > > CPU 2: 1 5 9 13 > CPU 3: 1 5 9 13 > > CPU 4: 2 6 10 14 > CPU 5: 2 6 10 14 > > CPU 6: 3 7 11 15 > CPU 7: 3 7 11 15 That part is clear, but > >> + __sync_fetch_and_add(&loop_cnt, 1); this doesn't match the rest. loop_cnt is inremented 4 times faster. So it's not comparable to other tests.