Chris Mason reported recently that a concurent stress test (basically copying the linux kernel tree 20 times, verifying md5sums and deleting it in a loop concurrently) under low memory conditions was triggering the OOM killer muchmore easily than for btrfs. Turns out there are two main problems. The first is that unlinked inodes were not being reclaimed fast enough, leading to the OOM being declared when there are large numbers of reclaimable inodes still around. The second was that atime updates due to the verify step were creating large numbers of dirty inodes at the VFS level that were not being written back and hence made reclaimable before the system declared OOM and killed stuff. The first problem is fixed by making background inode reclaim more frequent and faster, kicking background reclaim from the inode cache shrinker so that when memory is low we always have background inode reclaim in progress, and finally making the shrinker reclaim scan block waiting on inodes to reclaim. This last step throttles memory reclaim to the speed at which we can reclaim inodes, a key step needed to prevent inodes from reclaim declaring OOM while there are still reclaimable inodes around. The background inode reclaim prevents this synchronous flush from finding dirty inodes and block on them in most cases and hence prevents performance regressions in more common workloads due to reclaim stalls. To enable this new functionality, the xfssyncd thread is replaced with a workqueue and the existing xfssyncd work replaced with a global workqueue. Hence all filesystems will share the same workqueue and we remove allt eh xfssyncd threads from the system. The ENOSPC inode flush is converted to use the workqueue, and optimised to only allow a single flush at a time. This significant speeds up ENOSPC processing under concurrent workloads as it removes all the unnecessary scanning that every single ENOSPC event currently queues to the xfssyncd. Finally, a new reinode reclaim worker is added to the workqueue that runs 5x more frequently that the xfssyncd to do the background inode reclaim scan. The second problem is fixed simply by making the XFS inode cache shrinker kick the bdi flusher to write back inodes if the bdi flusher is not already active. This ensures that in low memory situations we are always actively writing back inodes that are dirty at the VFS level and hence preventing them from building up in an unreclaimable state. Once again this does not affect performance in non-memory constrained situations. The result is not yet perfect - the stress test still triggers the OOM killer somewhere between 3-6 hours into the test on a CONFIG_XFS_DEBUG kernel with lockdep enabled (so inodes consume roughly 2x the memory of a production kernel), though this is a marked improvement. The OOM kill trigger appears to be a different one to the above two, so expect more patches to address that soon. _______________________________________________ xfs mailing list xfs@xxxxxxxxxxx http://oss.sgi.com/mailman/listinfo/xfs
