On Wed 18-07-18 16:29:20, Bruce Merry wrote: > On 18 July 2018 at 12:42, Michal Hocko <mhocko@xxxxxxxxxx> wrote: > > [CC some more people] > > > > On Tue 17-07-18 21:23:07, Andrew Morton wrote: > >> (cc linux-mm) > >> > >> On Tue, 3 Jul 2018 08:43:23 +0200 Bruce Merry <bmerry@xxxxxxxxx> wrote: > >> > >> > Hi > >> > > >> > I've run into an odd performance issue in the kernel, and not being a > >> > kernel dev or knowing terribly much about cgroups, am looking for > >> > advice on diagnosing the problem further (I discovered this while > >> > trying to pin down high CPU load in cadvisor). > >> > > >> > On some machines in our production system, cat > >> > /sys/fs/cgroup/memory/memory.stat is extremely slow (500ms on one > >> > machine), while on other nominally identical machines it is fast > >> > (2ms). > > > > Could you try to use ftrace to see where the time is spent? > > Thanks for looking into this. I'm not familiar with ftrace. Can you > give me a specific command line to run? Based on "perf record cat > /sys/fs/cgroup/memory/memory.stat"/"perf report", I see the following: > > 42.09% cat [kernel.kallsyms] [k] memcg_stat_show > 29.19% cat [kernel.kallsyms] [k] memcg_sum_events.isra.22 > 12.41% cat [kernel.kallsyms] [k] mem_cgroup_iter > 5.42% cat [kernel.kallsyms] [k] _find_next_bit > 4.14% cat [kernel.kallsyms] [k] css_next_descendant_pre > 3.44% cat [kernel.kallsyms] [k] find_next_bit > 2.84% cat [kernel.kallsyms] [k] mem_cgroup_node_nr_lru_pages I would just use perf record as you did. How long did the call take? Also is the excessive time an outlier or a more consistent thing? If the former does perf record show any difference? > > memory_stat_show should only scale with the depth of the cgroup > > hierarchy for memory.stat to get cumulative numbers. All the rest should > > be simply reads of gathered counters. There is no locking involved in > > the current kernel. What is the kernel version you are using, btw? > > Ubuntu 16.04 with kernel 4.13.0-41-generic (so presumably includes > some Ubuntu special sauce). Do you see the same whe running with the vanilla kernel? -- Michal Hocko SUSE Labs
