On Wed, Aug 21, 2013 at 11:24:58AM -0400, Josef 'Jeff' Sipek wrote: > We've started experimenting with larger directory block sizes to avoid > directory fragmentation. Everything seems to work fine, except that the log > is spammed with these lovely debug messages: > > XFS: possible memory allocation deadlock in kmem_alloc (mode:0x250) > > From looking at the code, it looks like that each of those messages (there > are thousands) equates to 100 trips through the loop. My guess is that the > larger blocks require multi-page allocations which are harder to satisfy. > This is with 3.10 kernel. No, larger blocks simply require more single pages. The buffer cache does not require multi-page allocation at all. So, mode = 0x250, which means ___GFP_NOWARN | ___GFP_IO | ___GFP_WAIT which is also known as a GFP_NOFS allocation context. So, it's entirely possible that your memory is full of cached filesystem data and metadata, and the allocation that needs more can't reclaim them. > The hardware is something like (I can find out the exact config is you want): > > 32 cores > 128 GB RAM > LSI 9271-8i RAID (one big RAID-60 with 36 disks, partitioned) > > As I hinted at earlier, we end up with pretty big directories. We can > semi-reliably trigger this when we run rsync on the data between two > (identical) hosts over 10GbitE. > > # xfs_info /dev/sda9 > meta-data=/dev/sda9 isize=256 agcount=6, agsize=268435455 blks > = sectsz=512 attr=2 > data = bsize=4096 blocks=1454213211, imaxpct=5 > = sunit=0 swidth=0 blks > naming =version 2 bsize=65536 ascii-ci=0 > log =internal bsize=4096 blocks=521728, version=2 > = sectsz=512 sunit=0 blks, lazy-count=1 > realtime =none extsz=4096 blocks=0, rtextents=0 > > /proc/slabinfo: https://www.copy.com/s/1x1yZFjYO2EI/slab.txt Hmmm. You're using filestreams. That's unusual. Only major slab cache is the buffer_head slab, with ~12 million active bufferheads. So, that means you've got at least 47-48GB of data in the page cache..... And there's only ~35000 xfs_buf items in the slab, so the metadata cache isn't very big, and reclaim from that isn't a problem, nor the inode caches as there's only 130,000 cached inodes. > sysrq m output: https://www.copy.com/s/mYfMYfJJl2EB/sysrq-m.txt 27764401 total pagecache pages which indicates that you've got close to 110GB of pages in the page cache. Hmmm, and 24-25GB of dirty pages in memory. You know, I'd be suspecting a memory reclaim problem here to do with having large amounts of dirty memory in the page cache. I don't think the underlying cause is going to be the filesystem code, as the warning should never be emitted if memory reclaim is making progress. Perhaps you could try lowering all the dirty memory thresholds to see if that allows memory reclaim to make more progress because there are fewer dirty pages in memory... Cheers, Dave. -- Dave Chinner david@xxxxxxxxxxxxx _______________________________________________ xfs mailing list xfs@xxxxxxxxxxx http://oss.sgi.com/mailman/listinfo/xfs
