> On Thu, 24 Oct 2024 14:09:54 +0800 Jim Zhao <jimzhao.ai@xxxxxxxxx> wrote: > > > On Wed, 23 Oct 2024 18:00:32 +0800 Jim Zhao <jimzhao.ai@xxxxxxxxx> wrote: > > > > > > With the strictlimit flag, wb_thresh acts as a hard limit in > > > > balance_dirty_pages() and wb_position_ratio(). When device write > > > > operations are inactive, wb_thresh can drop to 0, causing writes to > > > > be blocked. The issue occasionally occurs in fuse fs, particularly > > > > with network backends, the write thread is blocked frequently during > > > > a period. To address it, this patch raises the minimum wb_thresh to a > > > > controllable level, similar to the non-strictlimit case. > > > > > Please tell us more about the userspace-visible effects of this. It > > > *sounds* like a serious (but occasional) problem, but that is unclear. > > > > > And, very much relatedly, do you feel this fix is needed in earlier > > > (-stable) kernels? > > > > The problem exists in two scenarios: > > 1. FUSE Write Transition from Inactive to Active > > > > sometimes, active writes require several pauses to ramp up to the appropriate wb_thresh. > > As shown in the trace below, both bdi_setpoint and task_ratelimit are 0, means wb_thresh is 0. > > The dd process pauses multiple times before reaching a normal state. > > > > dd-1206590 [003] .... 62988.324049: balance_dirty_pages: bdi 0:51: limit=295073 setpoint=259360 dirty=454 bdi_setpoint=0 bdi_dirty=32 dirty_ratelimit=18716 task_ratelimit=0 dirtied=32 dirtied_pause=32 paused=0 pause=4 period=4 think=0 cgroup_ino=1 > > dd-1206590 [003] .... 62988.332063: balance_dirty_pages: bdi 0:51: limit=295073 setpoint=259453 dirty=454 bdi_setpoint=0 bdi_dirty=33 dirty_ratelimit=18716 task_ratelimit=0 dirtied=1 dirtied_pause=0 paused=0 pause=4 period=4 think=4 cgroup_ino=1 > > dd-1206590 [003] .... 62988.340064: balance_dirty_pages: bdi 0:51: limit=295073 setpoint=259526 dirty=454 bdi_setpoint=0 bdi_dirty=34 dirty_ratelimit=18716 task_ratelimit=0 dirtied=1 dirtied_pause=0 paused=0 pause=4 period=4 think=4 cgroup_ino=1 > > dd-1206590 [003] .... 62988.348061: balance_dirty_pages: bdi 0:51: limit=295073 setpoint=259531 dirty=489 bdi_setpoint=0 bdi_dirty=35 dirty_ratelimit=18716 task_ratelimit=0 dirtied=1 dirtied_pause=0 paused=0 pause=4 period=4 think=4 cgroup_ino=1 > > dd-1206590 [003] .... 62988.356063: balance_dirty_pages: bdi 0:51: limit=295073 setpoint=259531 dirty=490 bdi_setpoint=0 bdi_dirty=36 dirty_ratelimit=18716 task_ratelimit=0 dirtied=1 dirtied_pause=0 paused=0 pause=4 period=4 think=4 cgroup_ino=1 > > ... > > > > 2. FUSE with Unstable Network Backends and Occasional Writes > > Not easy to reproduce, but when it occurs in this scenario, > > it causes the write thread to experience more pauses and longer durations. > Thanks, but it's still unclear how this impacts our users. How lenghty > are these pauses? The length is related to device writeback bandwidth. Under normal bandwidth, each pause may last around 4ms in several times as shown in the trace above(5 times). In extreme cases, fuse with unstable network backends, if pauses occur frequently and bandwidth is low, each pause can exceed 10ms, the total duration of pauses can accumulate to second. Thnaks, Jim Zhao
