Hello,
we have in the last few weeks always problems with the XFS file
system.
XFS: possible memory allocation deadlock in kmem_alloc (mode:
0x250)
After a certain time there is a kernel crash which looks as
follows:
September 23 10:17:57 hostname kernel: [1282081.010814] XFS:
possible memory
allocation deadlock in kmem_alloc (mode: 0x250)
September 23 10:17:59 hostname kernel: [1282082.913437] XFS:
possible memory
allocation deadlock in kmem_alloc (mode: 0x250)
September 23 10:18:01 hostname kernel: [1282084.908582] XFS:
possible memory
allocation deadlock in kmem_alloc (mode: 0x250)
September 23 10:18:03 hostname kernel: [1282086.691791] XFS:
possible memory
allocation deadlock in kmem_alloc (mode: 0x250)
September 23 10:18:05 hostname kernel: [1282088.691319] XFS:
possible memory
allocation deadlock in kmem_alloc (mode: 0x250)
September 23 10:18:07 hostname kernel: [1282090.690182] XFS:
possible memory
allocation deadlock in kmem_alloc (mode: 0x250)
September 23 10:18:09 hostname kernel: [1282092.673355] XFS:
possible memory
allocation deadlock in kmem_alloc (mode: 0x250)
September 23 10:18:11 hostname kernel: [1282094.596504] XFS:
possible memory
allocation deadlock in kmem_alloc (mode: 0x250)
September 23 10:18:13 hostname kernel: [1282096.595971] XFS:
possible memory
allocation deadlock in kmem_alloc (mode: 0x250)
September 23 10:18:15 hostname kernel: [1282098.595301] XFS:
possible memory
allocation deadlock in kmem_alloc (mode: 0x250)
September 23 10:18:17 hostname kernel: [1282100.594232] XFS:
possible memory
allocation deadlock in kmem_alloc (mode: 0x250)
September 23 10:18:19 hostname kernel: [1282102.433262] XFS:
possible memory
allocation deadlock in kmem_alloc (mode: 0x250)
September 23 10:18:21 hostname kernel: [1282104.412432] XFS:
possible memory
allocation deadlock in kmem_alloc (mode: 0x250)
September 23 10:18:23 hostname kernel: [1282106.411598] XFS:
possible memory
allocation deadlock in kmem_alloc (mode: 0x250)
September 23 10:18:25 hostname kernel: [1282108.310740] XFS:
possible memory
allocation deadlock in kmem_alloc (mode: 0x250)
September 23 10:18:27 hostname kernel: [1282110.289892] XFS:
possible memory
allocation deadlock in kmem_alloc (mode: 0x250)
September 23 10:18:29 hostname kernel: [1282112.289106] XFS:
possible memory
allocation deadlock in kmem_alloc (mode: 0x250)
September 23 10:18:31 hostname kernel: [1282114.288247] XFS:
possible memory
allocation deadlock in kmem_alloc (mode: 0x250)
September 23 10:18:33 hostname kernel: [1282116.215953] XFS:
possible memory
allocation deadlock in kmem_alloc (mode: 0x250)
September 23 10:18:35 hostname kernel: [1282118.198488] XFS:
possible memory
allocation deadlock in kmem_alloc (mode: 0x250)
September 23 10:18:37 hostname kernel: [1282120.193686] XFS:
possible memory
allocation deadlock in kmem_alloc (mode: 0x250)
September 23 10:18:39 hostname kernel: [1282122.288716] XFS:
possible memory
allocation deadlock in kmem_alloc (mode: 0x250)
September 23 10:18:41 hostname kernel: [1282124.215970] XFS:
possible memory
allocation deadlock in kmem_alloc (mode: 0x250)
September 23 10:18:43 hostname kernel: [1282126.307031] XFS:
possible memory
allocation deadlock in kmem_alloc (mode: 0x250)
September 23 10:18:45 hostname kernel: [1282128.306190] XFS:
possible memory
allocation deadlock in kmem_alloc (mode: 0x250)
September 23 10:18:47 hostname kernel: [1282130.405310] XFS:
possible memory
allocation deadlock in kmem_alloc (mode: 0x250)
September 23 10:18:49 hostname kernel: [1282132.076611] XFS:
possible memory
allocation deadlock in kmem_alloc (mode: 0x250)
September 23 10:18:51 hostname kernel: [1282134.027790] XFS:
possible memory
allocation deadlock in kmem_alloc (mode: 0x250)
September 23 10:18:53 hostname kernel: [1282136.122911] XFS:
possible memory
allocation deadlock in kmem_alloc (mode: 0x250)
September 23 10:18:55 hostname kernel: [1282138.218032] XFS:
possible memory
allocation deadlock in kmem_alloc (mode: 0x250)
September 23 10:18:57 hostname kernel: [1282140.145365] XFS:
possible memory
allocation deadlock in kmem_alloc (mode: 0x250)
September 23 10:18:59 hostname kernel: [1282142.140528] XFS:
possible memory
allocation deadlock in kmem_alloc (mode: 0x250)
September 23 10:19:01 hostname kernel: [1282144.091708] XFS:
possible memory
allocation deadlock in kmem_alloc (mode: 0x250)
September 23 10:19:01 hostname kernel: [1282144.223485] INFO:
task kswapd0: 87
blocked for more than 120 seconds.
September 23 10:19:01 hostname kernel: [1282144.224156] Not
tainted 3.13.0-34-generic
# 60-Ubuntu ~ precise1
September 23 10:19:01 hostname kernel: [1282144.224837] "echo
0> / proc
/ sys / kernel / hung_task_timeout_secs" disables this message.
September 23 10:19:01 hostname kernel: [1282144.225597] kswapd0
D
ffff88031662a020 0 87 2 0x00000000
September 23 10:19:01 hostname kernel: [1282144.225609]
ffff88019500d8f8
0000000000000002 ffff880199d94440 ffff88019500dfd8
September 23 10:19:01 hostname kernel: [1282144.225621]
0000000000014440
0000000000014440 ffff880316cd2fe0 ffff880316f3afe0
September 23 10:19:01 hostname kernel: [1282144.225629]
ffff88019500d970
ffff880314119128 ffff88031411912c 00000000ffffffff
September 23 10:19:01 hostname kernel: [1282144.225638] Call
Trace:
September 23 10:19:01 hostname kernel: [1282144.225658]
[<ffffffff8175b8b9>] schedule + 0x29 / 0x70
September 23 10:19:01 hostname kernel: [1282144.225669]
[<ffffffff8175bbde>] schedule_preempt_disabled + 0xe /
0x10
September 23 10:19:01 hostname kernel: [1282144.225680]
[<ffffffff8175da14>] __mutex_lock_slowpath + 0x114 / 0x1b0
September 23 10:19:01 hostname kernel: [1282144.225771]
[<ffffffffa0218291>]? xfs_perag_get_tag + 0x41 / 0xf0
[xfs]
September 23 10:19:01 hostname kernel: [1282144.225781]
[<ffffffff8175dad3>] mutex_lock + 0x23 / 0x37
September 23 10:19:01 hostname kernel: [1282144.225832]
[<ffffffffa01c7540>] xfs_reclaim_inodes_ag + 0x2e0 / 0x3a0
[xfs]
September 23 10:19:01 hostname kernel: [1282144.225843]
[<ffffffff8101d0b9>]? sched_clock + 0x9 / 0x10
September 23 10:19:01 hostname kernel: [1282144.225852]
[<ffffffff81042c9b>]? native_smp_send_reschedule + 0x4b /
0x60
September 23 10:19:01 hostname kernel: [1282144.225862]
[<ffffffff8109cf7d>]? ttwu_queue + 0x8D / 0xD0
September 23 10:19:01 hostname kernel: [1282144.225871]
[<ffffffff8109f860>]?
try_to_wake_up + 0x190 / 0x210
September 23 10:19:01 hostname kernel: [1282144.225879]
[<ffffffff8109f947>]? wake_up_process + 0x27 / 0x50
uname -a
hostname 3.13.0-34-generic # 60-Ubuntu SMP Wed precise1 ~ August
13 15:55:33
UTC 2014 x86_64 x86_64 x86_64 GNU / Linux
xfs_repair -V
xfs_repair Version 3.1.7
7 CPUs -> cat /proc/cpuinfo
processor : 0
vendor_id : AuthenticAMD
cpu family : 16
model : 4
model name : Quad-Core AMD Opteron(tm) Processor 2376
stepping : 2
microcode : 0x1000086
cpu MHz : 800.000
cache size : 512 KB
physical id : 0
siblings : 4
core id : 0
cpu cores : 4
apicid : 0
initial apicid : 0
fpu : yes
fpu_exception : yes
cpuid level : 5
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep
mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall
nx mmxext fxsr_opt pdpe1gb rdtscp lm 3dnowext 3dnow constant_tsc
rep_good nopl nonstop_tsc extd_apicid pni monitor cx16 popcnt
lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse
3dnowprefetch osvw ibs skinit wdt hw_pstate npt lbrv svm_lock
nrip_save
bogomips : 4600.18
TLB size : 1024 4K pages
clflush size : 64
cache_alignment : 64
address sizes : 48 bits physical, 48 bits virtual
power management: ts ttp tm stc 100mhzsteps hwpstate
cat /proc/meminfo
MemTotal: 12304884 kB
MemFree: 151816 kB
Buffers: 1076 kB
Cached: 10328496 kB
SwapCached: 0 kB
Active: 2658232 kB
Inactive: 7766540 kB
Active(anon): 42644 kB
Inactive(anon): 53636 kB
Active(file): 2615588 kB
Inactive(file): 7712904 kB
Unevictable: 0 kB
Mlocked: 0 kB
SwapTotal: 4878268 kB
SwapFree: 4878268 kB
Dirty: 1220 kB
Writeback: 0 kB
AnonPages: 95728 kB
Mapped: 32208 kB
Shmem: 352 kB
Slab: 1412904 kB
SReclaimable: 1188188 kB
SUnreclaim: 224716 kB
KernelStack: 3512 kB
PageTables: 28816 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 11030708 kB
Committed_AS: 949792 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 310312 kB
VmallocChunk: 34353030140 kB
HardwareCorrupted: 0 kB
AnonHugePages: 12288 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
DirectMap4k: 75520 kB
DirectMap2M: 3069952 kB
DirectMap1G: 9437184 kB
cat /proc/mounts
rootfs / rootfs rw 0 0
sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0
proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0
udev /dev devtmpfs
rw,relatime,size=6108796k,nr_inodes=1527199,mode=755 0 0
devpts /dev/pts devpts
rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0
tmpfs /run tmpfs
rw,nosuid,noexec,relatime,size=1230492k,mode=755 0 0
/dev/disk/by-uuid/d6821e50-e82b-448d-bc03-9c286ef59c8f / xfs
rw,relatime,attr2,inode64,noquota 0 0
none /sys/fs/fuse/connections fusectl rw,relatime 0 0
none /sys/kernel/debug debugfs rw,relatime 0 0
none /sys/kernel/security securityfs rw,relatime 0 0
none /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k
0 0
none /run/shm tmpfs rw,nosuid,nodev,relatime 0 0
/dev/sdc1 /extern/data0 xfs rw,relatime,attr2,inode64,noquota 0
0
rpc_pipefs /run/rpc_pipefs rpc_pipefs rw,relatime 0 0
nfsd /proc/fs/nfsd nfsd rw,relatime 0 0
cat /proc/partitions
major minor #blocks name
11 0 1048575 sr0
8 0 245175336 sda
8 1 97654784 sda1
8 2 4882432 sda2
8 16 245175336 sdb
8 17 97654784 sdb1
8 18 4882432 sdb2
9 2 97589120 md2
9 1 4878272 md1
8 32 27345461248 sdc
8 33 27345461214 sdc1
xfs_info /dev/md2
Metadaten
=/dev/disk/by-uuid/d6821e50-e82b-448d-bc03-9c286ef59c8f
isize=256 agcount=4, agsize=6099320 blks
= sectsz=512 attr=2
Daten = bsize=4096 Blöcke=24397280,
imaxpct=25
= sunit=0 swidth=0 blks
Benennung =Version 2 bsize=4096 ascii-ci=0
Protokoll =Intern bsize=4096 Blöcke=11912,
Version=2
= sectsz=512 sunit=0 blks,
lazy-count=1
Echtzeit =keine extsz=4096 Blöcke=0,
rtextents=0
xfs_info /dev/md2
Metadaten
=/dev/disk/by-uuid/d6821e50-e82b-448d-bc03-9c286ef59c8f
isize=256 agcount=4, agsize=6099320 blks
= sectsz=512 attr=2
Daten = bsize=4096 Blöcke=24397280,
imaxpct=25
= sunit=0 swidth=0 blks
Benennung =Version 2 bsize=4096 ascii-ci=0
Protokoll =Intern bsize=4096 Blöcke=11912,
Version=2
= sectsz=512 sunit=0 blks,
lazy-count=1
Echtzeit =keine extsz=4096 Blöcke=0,
rtextents=0
xfs_info /dev/sdc1
Metadaten =/dev/sdc1 isize=256 agcount=26,
agsize=268435455 blks
= sectsz=512 attr=2
Daten = bsize=4096
Blöcke=6836365303, imaxpct=5
= sunit=0 swidth=0 blks
Benennung =Version 2 bsize=4096 ascii-ci=0
Protokoll =Intern bsize=4096 Blöcke=32768,
Version=2
= sectsz=512 sunit=0 blks,
lazy-count=0
Echtzeit =keine extsz=4096 Blöcke=0,
rtextents=0
Can you help us?
Best regards,
--
Mit freundlichen Grüssen,
Thomas Stagge
--------------------------------------
Thomas Stagge, Dipl.Ing.(FH)
System Administrator
OPTIMARE Systems GmbH
Am Luneort 15a
27572 Bremerhaven, Germany
T: +49 471 48361-27
F: +49 471 48361-11
M: +49 172 8698575
thomas.stagge@xxxxxxxxxxx
Internet: http://www.optimare.de
CEO: Dipl.-Ing. Hans J. Stahl; Sitz der Gesellschaft: Bremerhaven; Registergericht: Amtsgericht Bremen HRB 28618 HB
