On Tue 06-03-18 20:28:59, Tetsuo Handa wrote: > Laura Abbott wrote: > > On 02/26/2018 06:28 AM, Michal Hocko wrote: > > > On Fri 23-02-18 11:51:41, Laura Abbott wrote: > > >> Hi, > > >> > > >> The Fedora arm-32 build VMs have a somewhat long standing problem > > >> of hanging when running mkfs.ext4 with a bunch of processes stuck > > >> in D state. This has been seen as far back as 4.13 but is still > > >> present on 4.14: > > >> > > > [...] > > >> This looks like everything is blocked on the writeback completing but > > >> the writeback has been throttled. According to the infra team, this problem > > >> is _not_ seen without LPAE (i.e. only 4G of RAM). I did see > > >> https://patchwork.kernel.org/patch/10201593/ but that doesn't seem to > > >> quite match since this seems to be completely stuck. Any suggestions to > > >> narrow the problem down? > > > > > > How much dirtyable memory does the system have? We do allow only lowmem > > > to be dirtyable by default on 32b highmem systems. Maybe you have the > > > lowmem mostly consumed by the kernel memory. Have you tried to enable > > > highmem_is_dirtyable? > > > > > > > Setting highmem_is_dirtyable did fix the problem. The infrastructure > > people seemed satisfied enough with this (and are happy to have the > > machines back). > > That's good. > > > I'll see if they are willing to run a few more tests > > to get some more state information. > > Well, I'm far from understanding what is happening in your case, but I'm > interested in other threads which were trying to allocate memory. Therefore, > I appreciate if they can take SysRq-m + SysRq-t than SysRq-w (as described > at http://akari.osdn.jp/capturing-kernel-messages.html ). > > Code which assumes that kswapd can make progress can get stuck when kswapd > is blocked somewhere. And wbt_wait() seems to change behavior based on > current_is_kswapd(). If everyone is waiting for kswapd but kswapd cannot > make progress, I worry that it leads to hangups like your case. Tetsuo, could you stop this finally, pretty please? This is a well known limitation of 32b architectures with more than 4G. The lowmem can only handle 896MB of memory and that can be filled up with other kernel allocations. Stalled writeback is _usually_ a result of only little dirtyable memory which is left in the lowmem. We cannot simply allow highmem to be dirtyable by default due to reasons explained in other email. I can imagine that it is hard for you to grasp that not everything is "silent hang during OOM" but there are other things going on in the VM. -- Michal Hocko SUSE Labs
