On Wed, May 15, 2024 at 10:26:31AM -0400, Waiman Long wrote: > > On 5/14/24 23:59, Waiman Long wrote: > > On 5/14/24 15:25, Dan Schatzberg wrote: > > > Hi Waiman, > > > > > > I've noticed that on recent kernels io.stat metrics don't propagate > > > all the way up the hierarchy. Specifically, io.stat metrics of some > > > leaf cgroup will be propagated to the parent, but not its grandparent. > > > > > > For a simple repro, run the following: > > > > > > systemd-run --slice test-test dd if=/dev/urandom of=/tmp/test > > > bs=4096 count=1 > > > > > > Then: > > > > > > cat /sys/fs/cgroup/test.slice/test-test.slice/io.stat > > > > > > Shows the parent cgroup stats and I see wbytes=4096 but the > > > grandparent cgroup: > > > > > > cat /sys/fs/cgroup/test.slice/io.stat > > > > > > shows no writes. > > > > > > I believe this was caused by the change in "blk-cgroup: Optimize > > > blkcg_rstat_flush()". When blkcg_rstat_flush is called on the parent > > > cgroup, it exits early because the lockless list is empty since the > > > parent cgroup never issued writes itself (e.g. in > > > blk_cgroup_bio_start). However, in doing so it never propagated stats > > > to its parent. > > > > > > Can you confirm if my understanding of the logic here is correct and > > > advise on a fix? > > > > Yes, I believe your analysis is correct. Thanks for spotting this iostat > > propagation problem. > > > > I am working on a fix to address this problem and will post a patch once > > I have finished my testing. > > Actually, I can only reproduce the issue with a 3-level > (child-parent-grandparent) cgroup hierarchy below the root cgroup. The dd > command is run test.slice/test-test.slice. So both test.slice/io.stat and > test.slice/test-test.slice/io.stat are properly updated. That's correct, this repros with a 3-level cgroup hierarchy (or more). systemd-run should create an ephemeral .scope cgroup under test-test.slice and then delete it when the dd command finishes. So test.slice/test-test.slice was the parent (2nd level) and test.slice is the grandparent.
