On 11/4/19 6:01 PM, Darrick J. Wong wrote: > On Mon, Nov 04, 2019 at 03:38:12PM -0800, Chris Holcombe wrote: >> After upgrading from scientific linux 6 -> centos 7 i'm starting to >> see a sharp uptick in dmesg lines about xfs having a possible memory >> allocation deadlock. All the searching I did through previous mailing >> list archives and blog posts show all pointing to large files having >> too many extents. >> I don't think that is the case with these servers so I'm reaching out >> in the hopes of getting an answer to what is going on. The largest >> file sizes I can find on the servers are roughly 15GB with maybe 9 >> extents total. The vast majority small with only a few extents. >> I've setup a cron job to drop the cache every 5 minutes which is >> helping but not eliminating the problem. These servers are dedicated >> to storing data that is written through nginx webdav. AFAIK nginx >> webdav put does not use sparse files. >> >> Some info about the servers this issue is occurring on: >> >> nginx is writing to 82TB filesystems: >> xfs_info /dev/sdb1 >> meta-data=/dev/sdb1 isize=512 agcount=82, agsize=268435424 blks >> = sectsz=4096 attr=2, projid32bit=1 >> = crc=1 finobt=0 spinodes=0 >> data = bsize=4096 blocks=21973302784, imaxpct=1 >> = sunit=16 swidth=144 blks >> naming =version 2 bsize=65536 ascii-ci=0 ftype=1 >> log =internal bsize=4096 blocks=521728, version=2 >> = sectsz=4096 sunit=1 blks, lazy-count=1 >> realtime =none extsz=4096 blocks=0, rtextents=0 >> >> xfs_db -r /dev/sdb1 >> xfs_db> frag >> actual 6565, ideal 5996, fragmentation factor 8.67% >> Note, this number is largely meaningless. >> Files on this filesystem average 1.09 extents per file >> >> I see dmesg lines with various size numbers in the line: >> [6262080.803537] XFS: nginx(2514) possible memory allocation deadlock >> size 50184 in kmem_alloc (mode:0x250) > > Full kernel logs, please. There's not enough info here to tell what's > trying to grab a 50K memory buffer. a kernel uname -a would be groovy too so we have some idea what you're running. -Eric > --D >
