After upgrading from scientific linux 6 -> centos 7 i'm starting to
see a sharp uptick in dmesg lines about xfs having a possible memory
allocation deadlock. All the searching I did through previous mailing
list archives and blog posts show all pointing to large files having
too many extents.
I don't think that is the case with these servers so I'm reaching out
in the hopes of getting an answer to what is going on. The largest
file sizes I can find on the servers are roughly 15GB with maybe 9
extents total. The vast majority small with only a few extents.
I've setup a cron job to drop the cache every 5 minutes which is
helping but not eliminating the problem. These servers are dedicated
to storing data that is written through nginx webdav. AFAIK nginx
webdav put does not use sparse files.
Some info about the servers this issue is occurring on:
nginx is writing to 82TB filesystems:
xfs_info /dev/sdb1
meta-data=/dev/sdb1 isize=512 agcount=82, agsize=268435424 blks
= sectsz=4096 attr=2, projid32bit=1
= crc=1 finobt=0 spinodes=0
data = bsize=4096 blocks=21973302784, imaxpct=1
= sunit=16 swidth=144 blks
naming =version 2 bsize=65536 ascii-ci=0 ftype=1
log =internal bsize=4096 blocks=521728, version=2
= sectsz=4096 sunit=1 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
xfs_db -r /dev/sdb1
xfs_db> frag
actual 6565, ideal 5996, fragmentation factor 8.67%
Note, this number is largely meaningless.
Files on this filesystem average 1.09 extents per file
I see dmesg lines with various size numbers in the line:
[6262080.803537] XFS: nginx(2514) possible memory allocation deadlock
size 50184 in kmem_alloc (mode:0x250)
Typical extents for the largest files on the filesystem are:
find /mnt/jbod/ -type f -size +15G -printf '%s %p\n' -exec xfs_bmap
-vp {} \; | tee extents
17093242444 /mnt/jbod/boxfiler3038-sdb1/data/220190411/ephemeral/2019-08-12/18/0f6bee4d6ee0136af3b58eef611e2586.enc
/mnt/jbod/boxfiler3038-sdb1/data/220190411/ephemeral/2019-08-12/18/0f6bee4d6ee0136af3b58eef611e2586.enc:
EXT: FILE-OFFSET BLOCK-RANGE AG AG-OFFSET
TOTAL FLAGS
0: [0..1919]: 51660187008..51660188927 24
(120585600..120587519) 1920 00010
1: [1920..8063]: 51660189056..51660195199 24
(120587648..120593791) 6144 00011
2: [8064..4194175]: 51660210816..51664396927 24
(120609408..124795519) 4186112 00001
3: [4194176..11552759]: 51664560768..51671919351 24
(124959360..132317943) 7358584 00101
4: [11552760..33385239]: 51678355840..51700188319 24
(138754432..160586911) 21832480 00111
Memory size:
free -m
total used free shared buff/cache available
Mem: 64150 6338 421 2 57390 57123
Swap: 2047 6 2041
cat /etc/redhat-release
CentOS Linux release 7.6.1810 (Core)
cat /proc/buddyinfo
Node 0, zone DMA 0 0 1 0 1 0 0
0 0 1 3
Node 0, zone DMA32 31577 88 2 0 0 0 0
0 0 0 0
Node 0, zone Normal 33331 3323 582 87 0 0 0
0 0 0 0
Node 1, zone Normal 51121 6343 822 77 1 0 0
0 0 0 0
tuned-adm shows 'balanced' as the current tuning profile.
Thanks for your help!
