On Wed, Feb 23, 2022 at 9:07 AM Daniel Dao <dqminh@xxxxxxxxxxxxxx> wrote: > > On Wed, Feb 23, 2022 at 4:00 PM Shakeel Butt <shakeelb@xxxxxxxxxx> wrote: > > > > Can you share a bit more detail on your hardware configuration (num of > > > cpus) and if possible the flamegraph? > > > > > We have a mix of 96 and 128 cpus. I'm not yet sure if it's possible to share > the flamegraphs. We may have to come back to that later if necessary. > Can you please first confirm if you are running 5.15.19 kernel? If that is the case then it means the machine is seeing (nr_cpus * 32) stat updates. Couple of questions: 1) Did you see any call stack other than xfs read where expensive mem_cgroup_flush_stats() is showing up? 2) Can you please use the similar bpf+kprobe tracing for the memcg_rstat_updated() (or __mod_memcg_lruvec_state()) to find the source of frequent stat updates. 3) I am still pondering why disabling swap resolves the issue for you. Is that only for a workload different from xfs read? 4) Can you please also trace cgroup_rstat_flush_locked()?
