Hi folks, Willy reported generic/530 had started hanging on his test machines and I've tried to reproduce the problem he reported. While I haven't reproduced the exact hang he's been having, I've found a couple of others while running g/530 in a tight loop on a couple of test machines. The first two patches are a result of a hang documented in patch 1. The change to run the log worker earlier is defensive, but serves to break generic log space deadlocks during intent and unlinked inode recovery as it does at normal runtime. This doesn't fix the problem, just adds a layer of protection that means stuff that gets stuck on pinned buffers, push hangs, etc only stays hung up for 30s at most. The second patch fixes the hang that results from delwri buffer pushing racing with modifications that pin the buffer (i.e. transaction commit) and then require access to it again soon after. The buffer is locked by delwri submission that is waiting for it to be unpinned, but the processes that might be able to trigger an unpin are blocked waiting for the buffer lock itself. This happens during log recovery when processing unlinked inodes that hit the same inode cluster buffer. The third patch is for log recovery hangs I've been seeing that occur after unlinked inode recovery has completed and the AIL is being pushed out. The trigger may be unique to the highly modified kernel I was running (and mitigated to a 30s delay to log recovery completion in g/530 by the first patch in the series), but I have occasionally seen period hangs in xfs_ail_push_all_sync() in the past where the AIL has not been fully emptied but it is sleeping without making progress. Hence I think the problem is a real one, just I don't have a way of reproducing it reliably an unmodified kernel. Willy, can you see if these patches fix the problem you are seeing? If not, I still think they stand alone as necessary fixes, but I'll have to keep digging to find out why you are seeing hangs in g/530. Cheers, Dave.
