On Wed, Mar 27, 2024 at 11:57:45PM +0800, Kemeng Shi wrote: > v1->v2: > -Send cleanup to wq_monitor.py separately. > -Add patch to avoid use after free of bdi. > -Rename wb_calc_cg_thresh to cgwb_calc_thresh as Tejun suggested. > -Use rcu walk to avoid use after free. > -Add debug output to each related patches. > > This series tries to improve visilibity of writeback. Patch 1 make > /sys/kernel/debug/bdi/xxx/stats show writeback info of whole bdi > instead of only writeback info in root cgroup. Patch 2 add a new > debug file /sys/kernel/debug/bdi/xxx/wb_stats to show per wb writeback > info. Patch 4 add wb_monitor.py to monitor basic writeback info > of running system, more info could be added on demand. Rest patches > are some random cleanups. More details can be found in respective > patches. Thanks! > This series is on top of patchset [1]. > > [1] https://lore.kernel.org/lkml/20240123183332.876854-1-shikemeng@xxxxxxxxxxxxxxx/T/#mc6455784a63d0f8aa1a2f5aff325abcdf9336b76 Not bad I've been trying to improve our ability to debug latency issues - stalls of all sorts. While you're looking at all this code, do you think you could find some places to collect useful latency numbers? fs/bcachefs/time_stats.c has some code that's going to be moving out to lib/ at some point, after I switch it to MAD; if you could hook that up as well to a few points we could see at a glance if there are stalls happening in the writeback path. > > Following domain hierarchy is tested: > global domain (320G) > / \ > cgroup domain1(10G) cgroup domain2(10G) > | | > bdi wb1 wb2 > > /* all writeback info of bdi is successfully collected */ > # cat /sys/kernel/debug/bdi/252:16/stats: > BdiWriteback: 448 kB > BdiReclaimable: 1303904 kB > BdiDirtyThresh: 189914124 kB > DirtyThresh: 195337564 kB > BackgroundThresh: 32516508 kB > BdiDirtied: 3591392 kB > BdiWritten: 2287488 kB > BdiWriteBandwidth: 322248 kBps > b_dirty: 0 > b_io: 0 > b_more_io: 2 > b_dirty_time: 0 > bdi_list: 1 > state: 1 > > /* per wb writeback info is collected */ > # cat /sys/kernel/debug/bdi/252:16/wb_stats: > cat wb_stats > WbCgIno: 1 > WbWriteback: 0 kB > WbReclaimable: 0 kB > WbDirtyThresh: 0 kB > WbDirtied: 0 kB > WbWritten: 0 kB > WbWriteBandwidth: 102400 kBps > b_dirty: 0 > b_io: 0 > b_more_io: 0 > b_dirty_time: 0 > state: 1 > WbCgIno: 4284 > WbWriteback: 448 kB > WbReclaimable: 818944 kB > WbDirtyThresh: 3096524 kB > WbDirtied: 2266880 kB > WbWritten: 1447936 kB > WbWriteBandwidth: 214036 kBps > b_dirty: 0 > b_io: 0 > b_more_io: 1 > b_dirty_time: 0 > state: 5 > WbCgIno: 4325 > WbWriteback: 224 kB > WbReclaimable: 819392 kB > WbDirtyThresh: 2920088 kB > WbDirtied: 2551808 kB > WbWritten: 1732416 kB > WbWriteBandwidth: 201832 kBps > b_dirty: 0 > b_io: 0 > b_more_io: 1 > b_dirty_time: 0 > state: 5 > > /* monitor writeback info */ > # ./wb_monitor.py 252:16 -c > writeback reclaimable dirtied written avg_bw > 252:16_1 0 0 0 0 102400 > 252:16_4284 672 820064 9230368 8410304 685612 > 252:16_4325 896 819840 10491264 9671648 652348 > 252:16 1568 1639904 19721632 18081952 1440360 > > > writeback reclaimable dirtied written avg_bw > 252:16_1 0 0 0 0 102400 > 252:16_4284 672 820064 9230368 8410304 685612 > 252:16_4325 896 819840 10491264 9671648 652348 > 252:16 1568 1639904 19721632 18081952 1440360 > ... > > Kemeng Shi (6): > writeback: protect race between bdi release and bdi_debug_stats_show > writeback: collect stats of all wb of bdi in bdi_debug_stats_show > writeback: support retrieving per group debug writeback stats of bdi > writeback: add wb_monitor.py script to monitor writeback info on bdi > writeback: rename nr_reclaimable to nr_dirty in balance_dirty_pages > writeback: define GDTC_INIT_NO_WB to null > > include/linux/writeback.h | 1 + > mm/backing-dev.c | 203 ++++++++++++++++++++++++++++++---- > mm/page-writeback.c | 31 ++++-- > tools/writeback/wb_monitor.py | 172 ++++++++++++++++++++++++++++ > 4 files changed, 378 insertions(+), 29 deletions(-) > create mode 100644 tools/writeback/wb_monitor.py > > -- > 2.30.0 >
