on 3/29/2024 3:24 AM, Kent Overstreet wrote: > On Wed, Mar 27, 2024 at 11:57:45PM +0800, Kemeng Shi wrote: >> v1->v2: >> -Send cleanup to wq_monitor.py separately. >> -Add patch to avoid use after free of bdi. >> -Rename wb_calc_cg_thresh to cgwb_calc_thresh as Tejun suggested. >> -Use rcu walk to avoid use after free. >> -Add debug output to each related patches. >> >> This series tries to improve visilibity of writeback. Patch 1 make >> /sys/kernel/debug/bdi/xxx/stats show writeback info of whole bdi >> instead of only writeback info in root cgroup. Patch 2 add a new >> debug file /sys/kernel/debug/bdi/xxx/wb_stats to show per wb writeback >> info. Patch 4 add wb_monitor.py to monitor basic writeback info >> of running system, more info could be added on demand. Rest patches >> are some random cleanups. More details can be found in respective >> patches. Thanks! >> This series is on top of patchset [1]. >> >> [1] https://lore.kernel.org/lkml/20240123183332.876854-1-shikemeng@xxxxxxxxxxxxxxx/T/#mc6455784a63d0f8aa1a2f5aff325abcdf9336b76 > > Not bad > Hi Kent, > I've been trying to improve our ability to debug latency issues - stalls > of all sorts. While you're looking at all this code, do you think you > could find some places to collect useful latency numbers? I would like to do it to collect more useful info for writeback. > > fs/bcachefs/time_stats.c has some code that's going to be moving out to > lib/ at some point, after I switch it to MAD; if you could hook that up > as well to a few points we could see at a glance if there are stalls > happening in the writeback path. I see that Tejun recommend to use bpf. I don't know much about bpf and new approach in time_stats.c. For me personly, I think that it's better to use the new approach after the work of moving code to lib/ is merged. Then I would like to submit a patchset to discuss of using it in writeback. Would this make sense to you. Look forward to your reply! Thanks, Kemeng > >> >> Following domain hierarchy is tested: >> global domain (320G) >> / \ >> cgroup domain1(10G) cgroup domain2(10G) >> | | >> bdi wb1 wb2 >> >> /* all writeback info of bdi is successfully collected */ >> # cat /sys/kernel/debug/bdi/252:16/stats: >> BdiWriteback: 448 kB >> BdiReclaimable: 1303904 kB >> BdiDirtyThresh: 189914124 kB >> DirtyThresh: 195337564 kB >> BackgroundThresh: 32516508 kB >> BdiDirtied: 3591392 kB >> BdiWritten: 2287488 kB >> BdiWriteBandwidth: 322248 kBps >> b_dirty: 0 >> b_io: 0 >> b_more_io: 2 >> b_dirty_time: 0 >> bdi_list: 1 >> state: 1 >> >> /* per wb writeback info is collected */ >> # cat /sys/kernel/debug/bdi/252:16/wb_stats: >> cat wb_stats >> WbCgIno: 1 >> WbWriteback: 0 kB >> WbReclaimable: 0 kB >> WbDirtyThresh: 0 kB >> WbDirtied: 0 kB >> WbWritten: 0 kB >> WbWriteBandwidth: 102400 kBps >> b_dirty: 0 >> b_io: 0 >> b_more_io: 0 >> b_dirty_time: 0 >> state: 1 >> WbCgIno: 4284 >> WbWriteback: 448 kB >> WbReclaimable: 818944 kB >> WbDirtyThresh: 3096524 kB >> WbDirtied: 2266880 kB >> WbWritten: 1447936 kB >> WbWriteBandwidth: 214036 kBps >> b_dirty: 0 >> b_io: 0 >> b_more_io: 1 >> b_dirty_time: 0 >> state: 5 >> WbCgIno: 4325 >> WbWriteback: 224 kB >> WbReclaimable: 819392 kB >> WbDirtyThresh: 2920088 kB >> WbDirtied: 2551808 kB >> WbWritten: 1732416 kB >> WbWriteBandwidth: 201832 kBps >> b_dirty: 0 >> b_io: 0 >> b_more_io: 1 >> b_dirty_time: 0 >> state: 5 >> >> /* monitor writeback info */ >> # ./wb_monitor.py 252:16 -c >> writeback reclaimable dirtied written avg_bw >> 252:16_1 0 0 0 0 102400 >> 252:16_4284 672 820064 9230368 8410304 685612 >> 252:16_4325 896 819840 10491264 9671648 652348 >> 252:16 1568 1639904 19721632 18081952 1440360 >> >> >> writeback reclaimable dirtied written avg_bw >> 252:16_1 0 0 0 0 102400 >> 252:16_4284 672 820064 9230368 8410304 685612 >> 252:16_4325 896 819840 10491264 9671648 652348 >> 252:16 1568 1639904 19721632 18081952 1440360 >> ... >> >> Kemeng Shi (6): >> writeback: protect race between bdi release and bdi_debug_stats_show >> writeback: collect stats of all wb of bdi in bdi_debug_stats_show >> writeback: support retrieving per group debug writeback stats of bdi >> writeback: add wb_monitor.py script to monitor writeback info on bdi >> writeback: rename nr_reclaimable to nr_dirty in balance_dirty_pages >> writeback: define GDTC_INIT_NO_WB to null >> >> include/linux/writeback.h | 1 + >> mm/backing-dev.c | 203 ++++++++++++++++++++++++++++++---- >> mm/page-writeback.c | 31 ++++-- >> tools/writeback/wb_monitor.py | 172 ++++++++++++++++++++++++++++ >> 4 files changed, 378 insertions(+), 29 deletions(-) >> create mode 100644 tools/writeback/wb_monitor.py >> >> -- >> 2.30.0 >> >
