On 1/31/19 2:26 AM, Jan Kara wrote: > Hi! > > On Thu 31-01-19 10:03:34, Xiaoguang Wang wrote: >>> Currently in blk_throtl_bio(), if one bio exceeds its throtl_grp's bps >>> or iops limit, this bio will be queued throtl_grp's throtl_service_queue, >>> then obviously mm subsys will submit more pages, even underlying device >>> can not handle these io requests, also this will make large amount of pages >>> entering writeback prematurely, later if some process writes some of these >>> pages, it will wait for long time. >>> >>> I have done some tests: one process does buffered writes on a 1GB file, >>> and make this process's blkcg max bps limit be 10MB/s, I observe this: >>> #cat /proc/meminfo | grep -i back >>> Writeback: 900024 kB >>> WritebackTmp: 0 kB >>> >>> I think this Writeback value is just too big, indeed many bios have been >>> queued in throtl_grp's throtl_service_queue, if one process try to write >>> the last bio's page in this queue, it will call wait_on_page_writeback(page), >>> which must wait the previous bios to finish and will take long time, we >>> have also see 120s hung task warning in our server. >>> >>> To fix this issue, we can simply limit throtl_service_queue's max queued >>> bios, currently we limit it to throtl_grp's bps_limit or iops limit, if it >>> still exteeds, we just sleep for a while. >> Ping :) >> >> The fix method in this patch is not good, I had written a new patch that >> uses wait queue, but do you think this is a blk-throttle design issue and >> needs fixing? thanks. > > Well, essentially this is a priority inversion issue where low-priority > process submits writes and higher priority process blocks on those, isn't > it? I think the blk-wbt throttling was meant to address these issues by > throttling the process already when submitting bios (i.e. something similar > to what you propose in your patch). I'll defer to Jens as a maintainer > whether he wants to redirect users to blk-wbt or whether improving > blk-throttle to avoid similar issues is desirable. Jens? I think that blk-throttle usage should be phased out and we can hopefully remove it at some point. I also don't think that there's a large use base of it, which is good, but does seem active on the Alibaba front. -- Jens Axboe
