Ilya, further to your email I have switched back to the 3.18 kernel that you've sent and I got similar looking dmesg output as I had on the 3.17 kernel. Please find it attached for your reference. As before, this is the command I've ran on the client:
time dd if=/dev/zero of=4G00 bs=4M count=5K oflag=direct & time dd if=/dev/zero of=4G11 bs=4M count=5K oflag=direct &time dd if=/dev/zero of=4G22 bs=4M count=5K oflag=direct &time dd if=/dev/zero of=4G33 bs=4M count=5K oflag=direct & time dd if=/dev/zero of=4G44 bs=4M count=5K oflag=direct & time dd if=/dev/zero of=4G55 bs=4M count=5K oflag=direct &time dd if=/dev/zero of=4G66 bs=4M count=5K oflag=direct &time dd if=/dev/zero of=4G77 bs=4M count=5K oflag=direct &
Andrei
time dd if=/dev/zero of=4G00 bs=4M count=5K oflag=direct & time dd if=/dev/zero of=4G11 bs=4M count=5K oflag=direct &time dd if=/dev/zero of=4G22 bs=4M count=5K oflag=direct &time dd if=/dev/zero of=4G33 bs=4M count=5K oflag=direct & time dd if=/dev/zero of=4G44 bs=4M count=5K oflag=direct & time dd if=/dev/zero of=4G55 bs=4M count=5K oflag=direct &time dd if=/dev/zero of=4G66 bs=4M count=5K oflag=direct &time dd if=/dev/zero of=4G77 bs=4M count=5K oflag=direct &
Andrei
From: "Ilya Dryomov" <ilya.dryomov@xxxxxxxxxxx>
To: "Gregory Farnum" <greg@xxxxxxxxxxx>
Cc: "Andrei Mikhailovsky" <andrei@xxxxxxxxxx>, "ceph-users" <ceph-users@xxxxxxxxxxxxxx>
Sent: Sunday, 30 November, 2014 7:47:48 AM
Subject: Re: Giant + nfs over cephfs hang tasks
On Sun, Nov 30, 2014 at 1:19 AM, Gregory Farnum <greg@xxxxxxxxxxx> wrote:
> Ilya, do you have a ticket reference for the bug?
Opened a ticket, assigned to myself.
http://tracker.ceph.com/issues/10208
> Andrei, we run NFS tests on CephFS in our nightlies and it does pretty well
> so in the general case we expect it to work. Obviously not at the moment
> with whatever bug Ilya is looking at, though. ;)
This is most probably a libceph issue - both krbd and kcephfs are
affected. I've been tracking it under the general "io hang" umbrella,
which spreads over a couple existing tickets. Definitely not a
nfs-on-cephfs problem, Greg.
Thanks,
Ilya
Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.879504] libceph: osd6 down Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.880376] libceph: osd6 up Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.880381] libceph: osd1 down Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.880384] libceph: osd5 down Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.880387] libceph: osd8 down Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.881265] libceph: osd3 down Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.882067] libceph: osd5 up Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.882071] libceph: osd8 up Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.882485] libceph: osd1 up Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.882489] libceph: osd3 up Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.883303] libceph: osd5 down Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.884122] libceph: osd5 up Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.885344] libceph: osd2 down Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.885349] libceph: osd7 down Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.885777] libceph: osd1 down Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.885781] libceph: osd4 down Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.885783] libceph: osd6 down Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.885786] libceph: osd8 down Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.886202] libceph: osd7 up Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.886206] libceph: osd8 up Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.886209] libceph: osd0 down Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.886639] libceph: osd2 up Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.887027] libceph: osd6 up Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.887410] libceph: osd0 up Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.887414] libceph: osd1 up Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.888265] libceph: osd4 up Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.889041] libceph: osd0 down Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.889045] libceph: osd1 down Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.889047] libceph: osd4 down Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.889050] libceph: osd7 down Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.889053] libceph: osd8 down Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.889439] libceph: osd2 down Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.889442] libceph: osd3 down Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.889844] libceph: osd0 up Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.889848] libceph: osd7 up Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.890236] libceph: osd1 up Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.890240] libceph: osd4 up Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.890243] libceph: osd8 up Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.891013] libceph: osd2 up Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.891409] libceph: osd3 up Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.892181] libceph: osd4 down Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.892584] libceph: osd4 up Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.893349] libceph: osd1 down Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.894509] libceph: osd1 up Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.895657] libceph: osd3 down Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.896441] libceph: osd3 up Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.897594] libceph: osd2 down Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.897597] libceph: osd6 down Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.897980] libceph: osd4 down Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.897983] libceph: osd7 down Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.898367] libceph: osd2 up Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.898370] libceph: osd4 up Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.898373] libceph: osd6 up Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.898376] libceph: osd8 down Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.899149] libceph: osd7 up Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.899153] libceph: osd8 up Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.899914] libceph: osd5 down Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.900682] libceph: osd5 up Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.901841] libceph: osd2 down Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.902599] libceph: osd2 up Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.902602] libceph: osd1 down Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.903746] libceph: osd1 up Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.904943] libceph: osd7 down Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.905743] libceph: osd7 up Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.906902] libceph: osd0 down Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.906906] libceph: osd6 down Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.907666] libceph: osd6 up Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.908163] libceph: osd0 up Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.910061] libceph: osd0 down Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.910064] libceph: osd7 down Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.910515] libceph: osd1 down Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.911335] libceph: osd0 up Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.912332] libceph: osd1 up Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.912360] libceph: osd7 up Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.914213] libceph: osd2 down Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.914976] libceph: osd2 up Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.916586] libceph: osd8 down Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.917349] libceph: osd8 up Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.918111] libceph: osd4 down Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.918897] libceph: osd4 up Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.920065] libceph: osd0 down Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.920069] libceph: osd4 down Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.920072] libceph: osd5 down Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.920952] libceph: osd6 down Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.921749] libceph: osd5 up Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.922511] libceph: osd8 down Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.922892] libceph: osd4 up Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.922895] libceph: osd6 up Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.923302] libceph: osd0 up Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.924231] libceph: osd8 up Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.925422] libceph: osd0 down Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.925426] libceph: osd4 down Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.925428] libceph: osd5 down Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.926188] libceph: osd4 up Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.926571] libceph: osd0 up Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.926574] libceph: osd5 up Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.927349] libceph: osd6 down Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.927731] libceph: osd6 up Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.928633] libceph: osd3 down Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.928637] libceph: osd4 down Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.929515] libceph: osd5 down Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.930345] libceph: osd3 up Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.930349] libceph: osd4 up Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.930352] libceph: osd5 up Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.931681] libceph: osd8 down Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.932575] libceph: osd8 up Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.933900] libceph: osd6 down Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.934305] libceph: osd6 up Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.935294] libceph: osd0 down Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.935298] libceph: osd6 down Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.936191] libceph: osd0 up Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.936618] libceph: osd6 up Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.938264] libceph: osd4 down Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.939065] libceph: osd4 up Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.940250] libceph: osd6 down Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.941052] libceph: osd6 up Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.942186] libceph: osd5 down Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.942968] libceph: osd0 down Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.942972] libceph: osd3 down Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.943430] libceph: osd0 up Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.943434] libceph: osd5 up Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.943881] libceph: osd3 up Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.945017] libceph: osd3 down Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.945848] libceph: osd3 up Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.946981] libceph: osd0 down Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.947779] libceph: osd0 up Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.948580] libceph: osd3 down Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.948584] libceph: osd4 down Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.948591] libceph: osd5 down Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.949547] libceph: osd3 up Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.949552] libceph: osd7 down Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.949554] libceph: osd8 down Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.949995] libceph: osd4 up Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.949999] libceph: osd5 up Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.950828] libceph: osd7 up Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.950832] libceph: osd8 up Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.952601] libceph: osd10 down Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.953080] libceph: osd16 down Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.953674] libceph: osd13 down Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.953679] libceph: osd15 down Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.954166] libceph: osd10 up Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.954170] libceph: osd16 up Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.955060] libceph: osd13 up Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.955064] libceph: osd15 up Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.955867] libceph: osd4 down Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.955871] libceph: osd5 down Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.956688] libceph: osd5 up Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.957487] libceph: osd4 up Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.958677] libceph: osd6 down Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.959452] libceph: osd6 up Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.960634] libceph: osd0 down Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.960638] libceph: osd4 down Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.960641] libceph: osd5 down Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.960644] libceph: osd6 down Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.961081] libceph: osd0 up Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.961086] libceph: osd4 up Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.961492] libceph: osd5 up Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.961496] libceph: osd6 up Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.962639] libceph: osd5 down Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.962643] libceph: osd6 down Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.963465] libceph: osd5 up Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.963938] libceph: osd6 up Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.965169] libceph: osd4 down Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.966377] libceph: osd4 up Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.967882] libceph: osd0 down Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.968689] libceph: osd0 up Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.969850] libceph: osd5 down Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.970648] libceph: osd5 up Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.971424] libceph: osd0 down Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.972233] libceph: osd0 up Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.972616] libceph: osd6 down Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.973394] libceph: osd6 up Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.974573] libceph: osd4 down Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.975329] libceph: osd4 up Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.976161] libceph: osd8 down Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.976948] libceph: osd8 up Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.977787] libceph: osd4 down Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.978187] libceph: osd4 up Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.978191] libceph: osd0 down Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.978194] libceph: osd6 down Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.978197] libceph: osd8 down Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.978601] libceph: osd0 up Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.979361] libceph: osd8 up Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.979763] libceph: osd6 up Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.980906] libceph: osd0 down Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.980910] libceph: osd8 down Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.981666] libceph: osd6 down Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.982046] libceph: osd8 up Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.982450] libceph: osd0 up Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.983256] libceph: osd6 up Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.984109] libceph: osd8 down Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.984490] libceph: osd6 down Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.984894] libceph: osd8 up Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.985651] libceph: osd6 up Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.986793] libceph: osd0 down Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.986797] libceph: osd3 down Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.986799] libceph: osd4 down Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.987182] libceph: osd0 up Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.987186] libceph: osd4 up Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.987570] libceph: osd3 up Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.989227] libceph: osd0 down Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.989990] libceph: osd0 up Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.989993] libceph: osd1 down Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.989996] libceph: osd2 down Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.989999] libceph: osd7 down Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.990827] libceph: osd2 up Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.991240] libceph: osd1 up Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77672.991244] libceph: osd7 up Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.238455] kworker/9:1: page allocation failure: order:5, mode:0x206050 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.238471] CPU: 9 PID: 28159 Comm: kworker/9:1 Tainted: G E 3.18.0-rc6-ceph-00024-g72ca172 #1 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.238479] Hardware name: Supermicro X9DRH-7TF/7F/iTF/iF/X9DRH-7TF/7F/iTF/iF, BIOS 3.0b 04/28/2014 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.238500] Workqueue: ceph-msgr con_work [libceph] Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.238507] 0000000000000005 ffff88010e5bf478 ffffffff81733b38 ffff88067fcce2d8 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.238515] 0000000000206050 ffff88010e5bf508 ffffffff81167f8e 0000000000000000 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.238523] 0000000400000000 ffff88067fff9f48 0000001fffffffe0 0000000000000010 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.238532] Call Trace: Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.238544] [<ffffffff81733b38>] dump_stack+0x4e/0x68 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.238554] [<ffffffff81167f8e>] warn_alloc_failed+0xee/0x140 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.238562] [<ffffffff8116bc72>] __alloc_pages_nodemask+0x962/0xba0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.238570] [<ffffffff810ca408>] ? lock_timer_base.isra.43+0x38/0x70 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.238577] [<ffffffff8116bef9>] alloc_kmem_pages_node+0x49/0x60 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.238585] [<ffffffff8173095d>] kmalloc_large_node+0x28/0x56 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.238595] [<ffffffff811bd09c>] __kmalloc_node_track_caller+0x1ec/0x230 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.238606] [<ffffffff816162d7>] ? __alloc_skb+0x87/0x2a0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.238612] [<ffffffff81615e95>] __kmalloc_reserve.isra.52+0x95/0xa0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.238618] [<ffffffff816162d7>] __alloc_skb+0x87/0x2a0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.238627] [<ffffffff816748f1>] sk_stream_alloc_skb+0x41/0x130 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.238632] [<ffffffff81675364>] tcp_sendmsg+0x3b4/0xd10 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.238639] [<ffffffff816a14b5>] ? inet_sendmsg+0x5/0x110 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.238645] [<ffffffff816a157f>] inet_sendmsg+0xcf/0x110 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.238650] [<ffffffff816a14b5>] ? inet_sendmsg+0x5/0x110 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.238656] [<ffffffff8160d67f>] sock_sendmsg+0xaf/0xc0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.238664] [<ffffffff8173cf59>] ? _raw_spin_unlock_bh+0x39/0x40 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.238670] [<ffffffff81675083>] ? tcp_sendmsg+0xd3/0xd10 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.238675] [<ffffffff8160d6d0>] kernel_sendmsg+0x40/0x60 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.238681] [<ffffffff8160fe4b>] sock_no_sendpage+0x9b/0xb0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.238687] [<ffffffff81674a37>] tcp_sendpage+0x57/0x590 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.238692] [<ffffffff816a15c5>] ? inet_sendpage+0x5/0x130 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.238702] [<ffffffffa051f1a5>] ? con_work+0x55/0x2cc0 [libceph] Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.238708] [<ffffffff816a16a2>] inet_sendpage+0xe2/0x130 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.238713] [<ffffffff816a15c5>] ? inet_sendpage+0x5/0x130 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.238720] [<ffffffff8160a72b>] kernel_sendpage+0x1b/0x30 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.238730] [<ffffffffa051e9db>] ceph_tcp_sendpage+0x5b/0x110 [libceph] Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.238740] [<ffffffffa051fc48>] con_work+0xaf8/0x2cc0 [libceph] Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.238750] [<ffffffff810801af>] ? finish_task_switch+0x3f/0x120 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.238757] [<ffffffff8107408a>] ? process_one_work+0x24a/0x530 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.238762] [<ffffffff81073f83>] ? process_one_work+0x143/0x530 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.238768] [<ffffffff81073f01>] ? process_one_work+0xc1/0x530 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.238773] [<ffffffff81073ffc>] process_one_work+0x1bc/0x530 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.238778] [<ffffffff81073f83>] ? process_one_work+0x143/0x530 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.238784] [<ffffffff8107475c>] worker_thread+0x11c/0x4b0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.238790] [<ffffffff81074640>] ? rescuer_thread+0x290/0x290 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.238796] [<ffffffff8107a664>] kthread+0xe4/0x100 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.238803] [<ffffffff8107a580>] ? flush_kthread_worker+0xf0/0xf0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.238810] [<ffffffff8173d66c>] ret_from_fork+0x7c/0xb0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.238815] [<ffffffff8107a580>] ? flush_kthread_worker+0xf0/0xf0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.238820] Mem-Info: Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.238824] Node 0 DMA per-cpu: Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.238829] CPU 0: hi: 0, btch: 1 usd: 0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.238834] CPU 1: hi: 0, btch: 1 usd: 0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.238838] CPU 2: hi: 0, btch: 1 usd: 0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.238842] CPU 3: hi: 0, btch: 1 usd: 0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.238846] CPU 4: hi: 0, btch: 1 usd: 0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.238850] CPU 5: hi: 0, btch: 1 usd: 0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.238854] CPU 6: hi: 0, btch: 1 usd: 0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.238858] CPU 7: hi: 0, btch: 1 usd: 0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.238862] CPU 8: hi: 0, btch: 1 usd: 0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.238866] CPU 9: hi: 0, btch: 1 usd: 0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.238870] CPU 10: hi: 0, btch: 1 usd: 0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.238874] CPU 11: hi: 0, btch: 1 usd: 0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.238878] Node 0 DMA32 per-cpu: Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.238882] CPU 0: hi: 186, btch: 31 usd: 0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.238886] CPU 1: hi: 186, btch: 31 usd: 0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.238890] CPU 2: hi: 186, btch: 31 usd: 0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.238894] CPU 3: hi: 186, btch: 31 usd: 0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.238898] CPU 4: hi: 186, btch: 31 usd: 0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.238902] CPU 5: hi: 186, btch: 31 usd: 0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.238906] CPU 6: hi: 186, btch: 31 usd: 0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.238910] CPU 7: hi: 186, btch: 31 usd: 0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.238914] CPU 8: hi: 186, btch: 31 usd: 0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.238918] CPU 9: hi: 186, btch: 31 usd: 0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.238922] CPU 10: hi: 186, btch: 31 usd: 0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.238926] CPU 11: hi: 186, btch: 31 usd: 0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.238929] Node 0 Normal per-cpu: Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.238934] CPU 0: hi: 186, btch: 31 usd: 0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.238938] CPU 1: hi: 186, btch: 31 usd: 0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.238942] CPU 2: hi: 186, btch: 31 usd: 0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.238945] CPU 3: hi: 186, btch: 31 usd: 0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.238949] CPU 4: hi: 186, btch: 31 usd: 0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.238953] CPU 5: hi: 186, btch: 31 usd: 0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.238957] CPU 6: hi: 186, btch: 31 usd: 0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.238961] CPU 7: hi: 186, btch: 31 usd: 0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.238965] CPU 8: hi: 186, btch: 31 usd: 27 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.238969] CPU 9: hi: 186, btch: 31 usd: 0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.238973] CPU 10: hi: 186, btch: 31 usd: 1 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.238977] CPU 11: hi: 186, btch: 31 usd: 0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.238981] Node 1 Normal per-cpu: Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.238985] CPU 0: hi: 186, btch: 31 usd: 0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.238990] CPU 1: hi: 186, btch: 31 usd: 30 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.238994] CPU 2: hi: 186, btch: 31 usd: 0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.238998] CPU 3: hi: 186, btch: 31 usd: 0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.239002] CPU 4: hi: 186, btch: 31 usd: 0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.239005] CPU 5: hi: 186, btch: 31 usd: 0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.239009] CPU 6: hi: 186, btch: 31 usd: 17 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.239013] CPU 7: hi: 186, btch: 31 usd: 0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.239017] CPU 8: hi: 186, btch: 31 usd: 0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.239021] CPU 9: hi: 186, btch: 31 usd: 31 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.239025] CPU 10: hi: 186, btch: 31 usd: 30 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.239029] CPU 11: hi: 186, btch: 31 usd: 24 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.239037] active_anon:1424463 inactive_anon:368455 isolated_anon:0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.239037] active_file:1823989 inactive_file:1862701 isolated_file:0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.239037] unevictable:0 dirty:214510 writeback:167592 unstable:0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.239037] free:90876 slab_reclaimable:179326 slab_unreclaimable:69862 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.239037] mapped:9364 shmem:362 pagetables:8248 bounce:0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.239037] free_cma:0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.239056] Node 0 DMA free:96kB min:56kB low:68kB high:84kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15956kB managed:15872kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.239079] lowmem_reserve[]: 0 1907 15985 15985 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.239087] Node 0 DMA32 free:73012kB min:7140kB low:8924kB high:10708kB active_anon:113484kB inactive_anon:181596kB active_file:551852kB inactive_file:680264kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:2047564kB managed:1955580kB mlocked:0kB dirty:32676kB writeback:22336kB mapped:1952kB shmem:76kB slab_reclaimable:255108kB slab_unreclaimable:51120kB kernel_stack:24240kB pagetables:2240kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.239112] lowmem_reserve[]: 0 0 14078 14078 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.239119] Node 0 Normal free:181116kB min:52716kB low:65892kB high:79072kB active_anon:2639044kB inactive_anon:657788kB active_file:5027880kB inactive_file:5056316kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:14680064kB managed:14416320kB mlocked:0kB dirty:429032kB writeback:340064kB mapped:16376kB shmem:1012kB slab_reclaimable:262560kB slab_unreclaimable:103260kB kernel_stack:23024kB pagetables:14112kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.239146] lowmem_reserve[]: 0 0 0 0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.239152] Node 1 Normal free:109280kB min:30192kB low:37740kB high:45288kB active_anon:2945324kB inactive_anon:634436kB active_file:1716224kB inactive_file:1714224kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:8388608kB managed:8256092kB mlocked:0kB dirty:396332kB writeback:307968kB mapped:19128kB shmem:360kB slab_reclaimable:199636kB slab_unreclaimable:125052kB kernel_stack:10768kB pagetables:16640kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.239178] lowmem_reserve[]: 0 0 0 0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.239184] Node 0 DMA: 0*4kB 0*8kB 2*16kB (U) 0*32kB 1*64kB (U) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 96kB Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.239204] Node 0 DMA32: 18065*4kB (U) 0*8kB 0*16kB 17*32kB (R) 1*64kB (R) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 72868kB Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.239224] Node 0 Normal: 23632*4kB (UEM) 7313*8kB (UEM) 773*16kB (UEM) 306*32kB (UM) 104*64kB (UM) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 181848kB Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.239247] Node 1 Normal: 23702*4kB (UEM) 1316*8kB (UEM) 84*16kB (UEMR) 57*32kB (UEMR) 8*64kB (UMR) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 109016kB Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.239272] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.239278] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.239283] 3688868 total pagecache pages Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.239288] 184 pages in swap cache Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.239292] Swap cache stats: add 654, delete 470, find 20/29 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.239296] Free swap = 3143168kB Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.239299] Total swap = 3145724kB Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.239302] 6283048 pages RAM Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.239305] 0 pages HighMem/MovableOnly Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.239309] 33129 pages reserved Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.239311] 0 pages hwpoisoned Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.239337] kworker/9:1: page allocation failure: order:5, mode:0x206050 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.239344] CPU: 9 PID: 28159 Comm: kworker/9:1 Tainted: G E 3.18.0-rc6-ceph-00024-g72ca172 #1 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.239350] Hardware name: Supermicro X9DRH-7TF/7F/iTF/iF/X9DRH-7TF/7F/iTF/iF, BIOS 3.0b 04/28/2014 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.239361] Workqueue: ceph-msgr con_work [libceph] Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.239365] 0000000000000005 ffff88010e5bf5f8 ffffffff81733b38 0000000000000007 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.239373] 0000000000206050 ffff88010e5bf688 ffffffff81167f8e 0000000000000000 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.239381] 0000000400000000 ffff88067fff9f48 0000001fffffffe0 0000000000000010 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.239389] Call Trace: Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.239394] [<ffffffff81733b38>] dump_stack+0x4e/0x68 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.239400] [<ffffffff81167f8e>] warn_alloc_failed+0xee/0x140 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.239407] [<ffffffff8116bc72>] __alloc_pages_nodemask+0x962/0xba0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.239414] [<ffffffff8109cac1>] ? finish_wait+0x41/0x80 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.239421] [<ffffffff8116bef9>] alloc_kmem_pages_node+0x49/0x60 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.239427] [<ffffffff8173095d>] kmalloc_large_node+0x28/0x56 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.239433] [<ffffffff811bd09c>] __kmalloc_node_track_caller+0x1ec/0x230 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.239440] [<ffffffff816162d7>] ? __alloc_skb+0x87/0x2a0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.239445] [<ffffffff8160fcd3>] ? lock_sock_nested+0x43/0xc0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.239451] [<ffffffff81615e95>] __kmalloc_reserve.isra.52+0x95/0xa0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.239458] [<ffffffff816162d7>] __alloc_skb+0x87/0x2a0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.239464] [<ffffffff810a0000>] ? print_cfs_rq+0x5a0/0x1430 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.239470] [<ffffffff816748f1>] sk_stream_alloc_skb+0x41/0x130 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.239476] [<ffffffff81675364>] tcp_sendmsg+0x3b4/0xd10 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.239481] [<ffffffff816a14b5>] ? inet_sendmsg+0x5/0x110 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.239487] [<ffffffff816a157f>] inet_sendmsg+0xcf/0x110 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.239492] [<ffffffff816a14b5>] ? inet_sendmsg+0x5/0x110 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.239497] [<ffffffff8160d67f>] sock_sendmsg+0xaf/0xc0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.239506] [<ffffffffa051f1a5>] ? con_work+0x55/0x2cc0 [libceph] Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.239512] [<ffffffff8160d6d0>] kernel_sendmsg+0x40/0x60 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.239521] [<ffffffffa051d22b>] ceph_tcp_sendmsg+0x4b/0x60 [libceph] Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.239531] [<ffffffffa051fd91>] con_work+0xc41/0x2cc0 [libceph] Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.239537] [<ffffffff810801af>] ? finish_task_switch+0x3f/0x120 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.239543] [<ffffffff8107408a>] ? process_one_work+0x24a/0x530 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.239548] [<ffffffff81073f83>] ? process_one_work+0x143/0x530 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.239553] [<ffffffff81073f83>] ? process_one_work+0x143/0x530 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.239559] [<ffffffff81073ffc>] process_one_work+0x1bc/0x530 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.239564] [<ffffffff81073f83>] ? process_one_work+0x143/0x530 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.239570] [<ffffffff8107475c>] worker_thread+0x11c/0x4b0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.239576] [<ffffffff81074640>] ? rescuer_thread+0x290/0x290 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.239582] [<ffffffff8107a664>] kthread+0xe4/0x100 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.239588] [<ffffffff8107a580>] ? flush_kthread_worker+0xf0/0xf0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.239594] [<ffffffff8173d66c>] ret_from_fork+0x7c/0xb0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.239600] [<ffffffff8107a580>] ? flush_kthread_worker+0xf0/0xf0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.239604] Mem-Info: Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.239607] Node 0 DMA per-cpu: Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.239611] CPU 0: hi: 0, btch: 1 usd: 0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.239615] CPU 1: hi: 0, btch: 1 usd: 0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.239619] CPU 2: hi: 0, btch: 1 usd: 0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.239623] CPU 3: hi: 0, btch: 1 usd: 0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.239627] CPU 4: hi: 0, btch: 1 usd: 0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.239631] CPU 5: hi: 0, btch: 1 usd: 0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.239635] CPU 6: hi: 0, btch: 1 usd: 0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.239639] CPU 7: hi: 0, btch: 1 usd: 0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.239643] CPU 8: hi: 0, btch: 1 usd: 0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.239647] CPU 9: hi: 0, btch: 1 usd: 0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.239651] CPU 10: hi: 0, btch: 1 usd: 0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.239655] CPU 11: hi: 0, btch: 1 usd: 0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.239658] Node 0 DMA32 per-cpu: Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.239662] CPU 0: hi: 186, btch: 31 usd: 0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.239666] CPU 1: hi: 186, btch: 31 usd: 0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.239670] CPU 2: hi: 186, btch: 31 usd: 0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.239674] CPU 3: hi: 186, btch: 31 usd: 0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.239678] CPU 4: hi: 186, btch: 31 usd: 0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.239682] CPU 5: hi: 186, btch: 31 usd: 0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.239686] CPU 6: hi: 186, btch: 31 usd: 0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.239690] CPU 7: hi: 186, btch: 31 usd: 0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.239694] CPU 8: hi: 186, btch: 31 usd: 0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.239698] CPU 9: hi: 186, btch: 31 usd: 0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.239702] CPU 10: hi: 186, btch: 31 usd: 0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.239706] CPU 11: hi: 186, btch: 31 usd: 0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.239710] Node 0 Normal per-cpu: Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.239714] CPU 0: hi: 186, btch: 31 usd: 0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.239718] CPU 1: hi: 186, btch: 31 usd: 0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.239722] CPU 2: hi: 186, btch: 31 usd: 0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.239726] CPU 3: hi: 186, btch: 31 usd: 0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.239730] CPU 4: hi: 186, btch: 31 usd: 0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.239734] CPU 5: hi: 186, btch: 31 usd: 0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.239738] CPU 6: hi: 186, btch: 31 usd: 0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.239742] CPU 7: hi: 186, btch: 31 usd: 0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.239746] CPU 8: hi: 186, btch: 31 usd: 27 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.239750] CPU 9: hi: 186, btch: 31 usd: 0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.239754] CPU 10: hi: 186, btch: 31 usd: 1 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.239758] CPU 11: hi: 186, btch: 31 usd: 0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.239761] Node 1 Normal per-cpu: Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.239765] CPU 0: hi: 186, btch: 31 usd: 0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.239769] CPU 1: hi: 186, btch: 31 usd: 30 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.239773] CPU 2: hi: 186, btch: 31 usd: 0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.239777] CPU 3: hi: 186, btch: 31 usd: 0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.239781] CPU 4: hi: 186, btch: 31 usd: 0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.239785] CPU 5: hi: 186, btch: 31 usd: 0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.239789] CPU 6: hi: 186, btch: 31 usd: 17 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.239793] CPU 7: hi: 186, btch: 31 usd: 0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.239797] CPU 8: hi: 186, btch: 31 usd: 0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.239801] CPU 9: hi: 186, btch: 31 usd: 31 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.239805] CPU 10: hi: 186, btch: 31 usd: 30 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.239809] CPU 11: hi: 186, btch: 31 usd: 24 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.239816] active_anon:1424463 inactive_anon:368455 isolated_anon:0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.239816] active_file:1823989 inactive_file:1862701 isolated_file:0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.239816] unevictable:0 dirty:214510 writeback:167592 unstable:0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.239816] free:90876 slab_reclaimable:179326 slab_unreclaimable:69862 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.239816] mapped:9364 shmem:362 pagetables:8248 bounce:0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.239816] free_cma:0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.239834] Node 0 DMA free:96kB min:56kB low:68kB high:84kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15956kB managed:15872kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.239857] lowmem_reserve[]: 0 1907 15985 15985 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.239864] Node 0 DMA32 free:73012kB min:7140kB low:8924kB high:10708kB active_anon:113484kB inactive_anon:181596kB active_file:551852kB inactive_file:680264kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:2047564kB managed:1955580kB mlocked:0kB dirty:32676kB writeback:22336kB mapped:1952kB shmem:76kB slab_reclaimable:255108kB slab_unreclaimable:51120kB kernel_stack:24240kB pagetables:2240kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.239889] lowmem_reserve[]: 0 0 14078 14078 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.239896] Node 0 Normal free:181116kB min:52716kB low:65892kB high:79072kB active_anon:2639044kB inactive_anon:657788kB active_file:5027880kB inactive_file:5056316kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:14680064kB managed:14416320kB mlocked:0kB dirty:429032kB writeback:340064kB mapped:16376kB shmem:1012kB slab_reclaimable:262560kB slab_unreclaimable:103260kB kernel_stack:23024kB pagetables:14112kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.239922] lowmem_reserve[]: 0 0 0 0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.239928] Node 1 Normal free:109280kB min:30192kB low:37740kB high:45288kB active_anon:2945324kB inactive_anon:634436kB active_file:1716224kB inactive_file:1714224kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:8388608kB managed:8256092kB mlocked:0kB dirty:396332kB writeback:307968kB mapped:19128kB shmem:360kB slab_reclaimable:199636kB slab_unreclaimable:125052kB kernel_stack:10768kB pagetables:16640kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.239954] lowmem_reserve[]: 0 0 0 0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.239960] Node 0 DMA: 0*4kB 0*8kB 2*16kB (U) 0*32kB 1*64kB (U) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 96kB Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.240002] Node 0 DMA32: 18065*4kB (U) 0*8kB 0*16kB 17*32kB (R) 1*64kB (R) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 72868kB Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.240066] Node 0 Normal: 23632*4kB (UEM) 7313*8kB (UEM) 773*16kB (UEM) 306*32kB (UM) 104*64kB (UM) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 181848kB Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.240089] Node 1 Normal: 23702*4kB (UEM) 1316*8kB (UEM) 84*16kB (UEMR) 57*32kB (UEMR) 8*64kB (UMR) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 109016kB Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.240113] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.240119] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.240125] 3688868 total pagecache pages Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.240128] 184 pages in swap cache Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.240132] Swap cache stats: add 654, delete 470, find 20/29 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.240136] Free swap = 3143168kB Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.240139] Total swap = 3145724kB Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.240143] 6283048 pages RAM Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.240145] 0 pages HighMem/MovableOnly Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.240148] 33129 pages reserved Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.240151] 0 pages hwpoisoned Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.240200] kworker/9:1: page allocation failure: order:5, mode:0x206050 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.240206] CPU: 9 PID: 28159 Comm: kworker/9:1 Tainted: G E 3.18.0-rc6-ceph-00024-g72ca172 #1 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.240213] Hardware name: Supermicro X9DRH-7TF/7F/iTF/iF/X9DRH-7TF/7F/iTF/iF, BIOS 3.0b 04/28/2014 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.240227] Workqueue: ceph-msgr con_work [libceph] Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.240236] 0000000000000005 ffff88010e5bf478 ffffffff81733b38 0000000000009e27 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.240260] 0000000000206050 ffff88010e5bf508 ffffffff81167f8e 0000000000000000 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.240273] 0000000400000000 ffff88067fff9f48 0000001fffffffe0 0000000000000010 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.240280] Call Trace: Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.240286] [<ffffffff81733b38>] dump_stack+0x4e/0x68 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.240292] [<ffffffff81167f8e>] warn_alloc_failed+0xee/0x140 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.240298] [<ffffffff8116bc72>] __alloc_pages_nodemask+0x962/0xba0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.240305] [<ffffffff8116bef9>] alloc_kmem_pages_node+0x49/0x60 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.240311] [<ffffffff8173095d>] kmalloc_large_node+0x28/0x56 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.240317] [<ffffffff811bd09c>] __kmalloc_node_track_caller+0x1ec/0x230 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.240324] [<ffffffff816162d7>] ? __alloc_skb+0x87/0x2a0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.240330] [<ffffffff8160fcd3>] ? lock_sock_nested+0x43/0xc0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.240336] [<ffffffff81615e95>] __kmalloc_reserve.isra.52+0x95/0xa0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.240342] [<ffffffff816162d7>] __alloc_skb+0x87/0x2a0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.240352] [<ffffffff810a0000>] ? print_cfs_rq+0x5a0/0x1430 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.240363] [<ffffffff816748f1>] sk_stream_alloc_skb+0x41/0x130 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.240373] [<ffffffff81675364>] tcp_sendmsg+0x3b4/0xd10 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.240383] [<ffffffff816a14b5>] ? inet_sendmsg+0x5/0x110 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.240393] [<ffffffff816a157f>] inet_sendmsg+0xcf/0x110 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.240402] [<ffffffff816a14b5>] ? inet_sendmsg+0x5/0x110 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.240410] [<ffffffff810a98fd>] ? trace_hardirqs_on+0xd/0x10 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.240416] [<ffffffff8160d67f>] sock_sendmsg+0xaf/0xc0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.240421] [<ffffffff81611b23>] ? release_sock+0x193/0x1d0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.240427] [<ffffffff816732a0>] ? tcp_recvmsg+0x610/0xca0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.240433] [<ffffffff816a13a5>] ? inet_recvmsg+0x5/0x110 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.240438] [<ffffffff8160d6d0>] kernel_sendmsg+0x40/0x60 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.240444] [<ffffffff8160fe4b>] sock_no_sendpage+0x9b/0xb0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.240450] [<ffffffff81674a37>] tcp_sendpage+0x57/0x590 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.240455] [<ffffffff816a15c5>] ? inet_sendpage+0x5/0x130 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.240461] [<ffffffff816a16a2>] inet_sendpage+0xe2/0x130 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.240466] [<ffffffff816a15c5>] ? inet_sendpage+0x5/0x130 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.240472] [<ffffffff8160a72b>] kernel_sendpage+0x1b/0x30 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.240482] [<ffffffffa051e9db>] ceph_tcp_sendpage+0x5b/0x110 [libceph] Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.240495] [<ffffffffa051fc48>] con_work+0xaf8/0x2cc0 [libceph] Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.240507] [<ffffffff810801af>] ? finish_task_switch+0x3f/0x120 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.240518] [<ffffffff8107408a>] ? process_one_work+0x24a/0x530 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.240528] [<ffffffff81073f83>] ? process_one_work+0x143/0x530 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.240539] [<ffffffff81073f01>] ? process_one_work+0xc1/0x530 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.240550] [<ffffffff81073ffc>] process_one_work+0x1bc/0x530 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.240560] [<ffffffff81073f83>] ? process_one_work+0x143/0x530 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.240571] [<ffffffff8107475c>] worker_thread+0x11c/0x4b0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.240576] [<ffffffff81074640>] ? rescuer_thread+0x290/0x290 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.240582] [<ffffffff8107a664>] kthread+0xe4/0x100 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.240588] [<ffffffff8107a580>] ? flush_kthread_worker+0xf0/0xf0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.240595] [<ffffffff8173d66c>] ret_from_fork+0x7c/0xb0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.240601] [<ffffffff8107a580>] ? flush_kthread_worker+0xf0/0xf0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.240605] Mem-Info: Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.240607] Node 0 DMA per-cpu: Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.240612] CPU 0: hi: 0, btch: 1 usd: 0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.240616] CPU 1: hi: 0, btch: 1 usd: 0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.240620] CPU 2: hi: 0, btch: 1 usd: 0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.240624] CPU 3: hi: 0, btch: 1 usd: 0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.240628] CPU 4: hi: 0, btch: 1 usd: 0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.240632] CPU 5: hi: 0, btch: 1 usd: 0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.240636] CPU 6: hi: 0, btch: 1 usd: 0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.240639] CPU 7: hi: 0, btch: 1 usd: 0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.240643] CPU 8: hi: 0, btch: 1 usd: 0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.240647] CPU 9: hi: 0, btch: 1 usd: 0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.240651] CPU 10: hi: 0, btch: 1 usd: 0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.240655] CPU 11: hi: 0, btch: 1 usd: 0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.240659] Node 0 DMA32 per-cpu: Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.240663] CPU 0: hi: 186, btch: 31 usd: 0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.240667] CPU 1: hi: 186, btch: 31 usd: 0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.240677] CPU 2: hi: 186, btch: 31 usd: 0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.240683] CPU 3: hi: 186, btch: 31 usd: 0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.240691] CPU 4: hi: 186, btch: 31 usd: 0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.240699] CPU 5: hi: 186, btch: 31 usd: 0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.240705] CPU 6: hi: 186, btch: 31 usd: 0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.240710] CPU 7: hi: 186, btch: 31 usd: 0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.240714] CPU 8: hi: 186, btch: 31 usd: 0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.240719] CPU 9: hi: 186, btch: 31 usd: 0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.240723] CPU 10: hi: 186, btch: 31 usd: 0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.240727] CPU 11: hi: 186, btch: 31 usd: 0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.240730] Node 0 Normal per-cpu: Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.240734] CPU 0: hi: 186, btch: 31 usd: 0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.240738] CPU 1: hi: 186, btch: 31 usd: 0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.240742] CPU 2: hi: 186, btch: 31 usd: 0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.240746] CPU 3: hi: 186, btch: 31 usd: 0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.240750] CPU 4: hi: 186, btch: 31 usd: 0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.240754] CPU 5: hi: 186, btch: 31 usd: 0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.240758] CPU 6: hi: 186, btch: 31 usd: 0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.240762] CPU 7: hi: 186, btch: 31 usd: 0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.240766] CPU 8: hi: 186, btch: 31 usd: 27 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.240770] CPU 9: hi: 186, btch: 31 usd: 0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.240774] CPU 10: hi: 186, btch: 31 usd: 1 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.240778] CPU 11: hi: 186, btch: 31 usd: 0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.240782] Node 1 Normal per-cpu: Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.240788] CPU 0: hi: 186, btch: 31 usd: 0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.240794] CPU 1: hi: 186, btch: 31 usd: 30 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.240802] CPU 2: hi: 186, btch: 31 usd: 0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.240818] CPU 3: hi: 186, btch: 31 usd: 0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.240822] CPU 4: hi: 186, btch: 31 usd: 0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.240826] CPU 5: hi: 186, btch: 31 usd: 0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.240831] CPU 6: hi: 186, btch: 31 usd: 0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.240838] CPU 7: hi: 186, btch: 31 usd: 0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.240845] CPU 8: hi: 186, btch: 31 usd: 0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.240851] CPU 9: hi: 186, btch: 31 usd: 0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.240858] CPU 10: hi: 186, btch: 31 usd: 0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.240864] CPU 11: hi: 186, btch: 31 usd: 0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.240873] active_anon:1424463 inactive_anon:368455 isolated_anon:0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.240873] active_file:1823989 inactive_file:1862575 isolated_file:0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.240873] unevictable:0 dirty:214510 writeback:167592 unstable:0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.240873] free:91388 slab_reclaimable:179326 slab_unreclaimable:69862 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.240873] mapped:9364 shmem:362 pagetables:8248 bounce:0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.240873] free_cma:0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.240893] Node 0 DMA free:2144kB min:56kB low:68kB high:84kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15956kB managed:15872kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.240916] lowmem_reserve[]: 0 1907 15985 15985 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.240923] Node 0 DMA32 free:73012kB min:7140kB low:8924kB high:10708kB active_anon:113484kB inactive_anon:181596kB active_file:551852kB inactive_file:680264kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:2047564kB managed:1955580kB mlocked:0kB dirty:32676kB writeback:22336kB mapped:1952kB shmem:76kB slab_reclaimable:255108kB slab_unreclaimable:51120kB kernel_stack:24240kB pagetables:2240kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.240949] lowmem_reserve[]: 0 0 14078 14078 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.240956] Node 0 Normal free:181116kB min:52716kB low:65892kB high:79072kB active_anon:2639044kB inactive_anon:657788kB active_file:5027880kB inactive_file:5056316kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:14680064kB managed:14416320kB mlocked:0kB dirty:429032kB writeback:340064kB mapped:16376kB shmem:1012kB slab_reclaimable:262560kB slab_unreclaimable:103260kB kernel_stack:23024kB pagetables:14112kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.240982] lowmem_reserve[]: 0 0 0 0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.240988] Node 1 Normal free:109280kB min:30192kB low:37740kB high:45288kB active_anon:2945324kB inactive_anon:634436kB active_file:1716224kB inactive_file:1713720kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:8388608kB managed:8256092kB mlocked:0kB dirty:396332kB writeback:307968kB mapped:19128kB shmem:360kB slab_reclaimable:199636kB slab_unreclaimable:125052kB kernel_stack:10768kB pagetables:16640kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.241015] lowmem_reserve[]: 0 0 0 0 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.241037] Node 0 DMA: 0*4kB 0*8kB 2*16kB (U) 0*32kB 1*64kB (U) 0*128kB 2*256kB (UR) 1*512kB (U) 1*1024kB (R) 0*2048kB 0*4096kB = 2144kB Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.241073] Node 0 DMA32: 18065*4kB (U) 0*8kB 0*16kB 17*32kB (R) 1*64kB (R) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 72868kB Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.241094] Node 0 Normal: 23660*4kB (UEM) 7313*8kB (UEM) 773*16kB (UEM) 306*32kB (UM) 104*64kB (UM) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 181960kB Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.241117] Node 1 Normal: 23769*4kB (UEM) 1316*8kB (UEM) 84*16kB (UEMR) 57*32kB (UEMR) 8*64kB (UMR) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 109284kB Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.241143] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.241152] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.241158] 3688868 total pagecache pages Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.241161] 184 pages in swap cache Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.241165] Swap cache stats: add 654, delete 470, find 20/29 Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.241169] Free swap = 3143168kB Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.241172] Total swap = 3145724kB Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.241175] 6283048 pages RAM Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.241180] 0 pages HighMem/MovableOnly Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.241186] 33129 pages reserved Nov 30 21:21:58 arh-ibstorage1-ib kernel: [77673.241190] 0 pages hwpoisoned Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.518096] kworker/8:0: page allocation failure: order:5, mode:0x206050 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.518111] CPU: 8 PID: 13987 Comm: kworker/8:0 Tainted: G E 3.18.0-rc6-ceph-00024-g72ca172 #1 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.518119] Hardware name: Supermicro X9DRH-7TF/7F/iTF/iF/X9DRH-7TF/7F/iTF/iF, BIOS 3.0b 04/28/2014 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.518139] Workqueue: ceph-msgr con_work [libceph] Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.518146] 0000000000000005 ffff8800171cf478 ffffffff81733b38 ffff88067fc8e2d8 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.518154] 0000000000206050 ffff8800171cf508 ffffffff81167f8e 0000000000000000 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.518162] 0000000400000000 ffff88067fff9f48 0000001fffffffe0 0000000000000010 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.518170] Call Trace: Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.518181] [<ffffffff81733b38>] dump_stack+0x4e/0x68 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.518189] [<ffffffff81167f8e>] warn_alloc_failed+0xee/0x140 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.518196] [<ffffffff8116bc72>] __alloc_pages_nodemask+0x962/0xba0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.518203] [<ffffffff8173d03b>] ? _raw_spin_unlock+0x2b/0x40 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.518211] [<ffffffff810ca408>] ? lock_timer_base.isra.43+0x38/0x70 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.518218] [<ffffffff8116bef9>] alloc_kmem_pages_node+0x49/0x60 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.518225] [<ffffffff8173095d>] kmalloc_large_node+0x28/0x56 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.518234] [<ffffffff811bd09c>] __kmalloc_node_track_caller+0x1ec/0x230 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.518245] [<ffffffff816162d7>] ? __alloc_skb+0x87/0x2a0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.518251] [<ffffffff81615e95>] __kmalloc_reserve.isra.52+0x95/0xa0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.518258] [<ffffffff816162d7>] __alloc_skb+0x87/0x2a0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.518265] [<ffffffff816748f1>] sk_stream_alloc_skb+0x41/0x130 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.518271] [<ffffffff81675364>] tcp_sendmsg+0x3b4/0xd10 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.518278] [<ffffffff816a14b5>] ? inet_sendmsg+0x5/0x110 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.518284] [<ffffffff816a157f>] inet_sendmsg+0xcf/0x110 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.518289] [<ffffffff816a14b5>] ? inet_sendmsg+0x5/0x110 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.518295] [<ffffffff8160d67f>] sock_sendmsg+0xaf/0xc0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.518301] [<ffffffff8173cf59>] ? _raw_spin_unlock_bh+0x39/0x40 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.518306] [<ffffffff81675083>] ? tcp_sendmsg+0xd3/0xd10 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.518312] [<ffffffff8160d6d0>] kernel_sendmsg+0x40/0x60 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.518318] [<ffffffff8160fe4b>] sock_no_sendpage+0x9b/0xb0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.518324] [<ffffffff81674a37>] tcp_sendpage+0x57/0x590 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.518329] [<ffffffff816a15c5>] ? inet_sendpage+0x5/0x130 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.518339] [<ffffffffa051f1a5>] ? con_work+0x55/0x2cc0 [libceph] Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.518344] [<ffffffff816a16a2>] inet_sendpage+0xe2/0x130 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.518350] [<ffffffff816a15c5>] ? inet_sendpage+0x5/0x130 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.518357] [<ffffffff8160a72b>] kernel_sendpage+0x1b/0x30 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.518367] [<ffffffffa051e9db>] ceph_tcp_sendpage+0x5b/0x110 [libceph] Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.518376] [<ffffffffa051fc48>] con_work+0xaf8/0x2cc0 [libceph] Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.518386] [<ffffffff810801af>] ? finish_task_switch+0x3f/0x120 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.518393] [<ffffffff81073f83>] ? process_one_work+0x143/0x530 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.518398] [<ffffffff81073f01>] ? process_one_work+0xc1/0x530 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.518404] [<ffffffff81073ffc>] process_one_work+0x1bc/0x530 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.518409] [<ffffffff81073f83>] ? process_one_work+0x143/0x530 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.518415] [<ffffffff8107475c>] worker_thread+0x11c/0x4b0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.518420] [<ffffffff81074640>] ? rescuer_thread+0x290/0x290 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.518427] [<ffffffff8107a664>] kthread+0xe4/0x100 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.518433] [<ffffffff8107a580>] ? flush_kthread_worker+0xf0/0xf0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.518440] [<ffffffff8173d66c>] ret_from_fork+0x7c/0xb0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.518446] [<ffffffff8107a580>] ? flush_kthread_worker+0xf0/0xf0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.518450] Mem-Info: Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.518454] Node 0 DMA per-cpu: Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.518459] CPU 0: hi: 0, btch: 1 usd: 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.518463] CPU 1: hi: 0, btch: 1 usd: 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.518467] CPU 2: hi: 0, btch: 1 usd: 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.518471] CPU 3: hi: 0, btch: 1 usd: 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.518475] CPU 4: hi: 0, btch: 1 usd: 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.518479] CPU 5: hi: 0, btch: 1 usd: 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.518483] CPU 6: hi: 0, btch: 1 usd: 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.518487] CPU 7: hi: 0, btch: 1 usd: 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.518491] CPU 8: hi: 0, btch: 1 usd: 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.518495] CPU 9: hi: 0, btch: 1 usd: 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.518499] CPU 10: hi: 0, btch: 1 usd: 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.518503] CPU 11: hi: 0, btch: 1 usd: 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.518507] Node 0 DMA32 per-cpu: Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.518511] CPU 0: hi: 186, btch: 31 usd: 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.518515] CPU 1: hi: 186, btch: 31 usd: 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.518520] CPU 2: hi: 186, btch: 31 usd: 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.518524] CPU 3: hi: 186, btch: 31 usd: 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.518528] CPU 4: hi: 186, btch: 31 usd: 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.518532] CPU 5: hi: 186, btch: 31 usd: 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.518536] CPU 6: hi: 186, btch: 31 usd: 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.518540] CPU 7: hi: 186, btch: 31 usd: 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.518544] CPU 8: hi: 186, btch: 31 usd: 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.518548] CPU 9: hi: 186, btch: 31 usd: 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.518552] CPU 10: hi: 186, btch: 31 usd: 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.518556] CPU 11: hi: 186, btch: 31 usd: 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.518559] Node 0 Normal per-cpu: Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.518564] CPU 0: hi: 186, btch: 31 usd: 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.518568] CPU 1: hi: 186, btch: 31 usd: 30 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.518572] CPU 2: hi: 186, btch: 31 usd: 29 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.518576] CPU 3: hi: 186, btch: 31 usd: 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.518580] CPU 4: hi: 186, btch: 31 usd: 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.518584] CPU 5: hi: 186, btch: 31 usd: 182 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.518588] CPU 6: hi: 186, btch: 31 usd: 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.518592] CPU 7: hi: 186, btch: 31 usd: 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.518595] CPU 8: hi: 186, btch: 31 usd: 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.518599] CPU 9: hi: 186, btch: 31 usd: 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.518603] CPU 10: hi: 186, btch: 31 usd: 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.518607] CPU 11: hi: 186, btch: 31 usd: 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.518611] Node 1 Normal per-cpu: Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.518616] CPU 0: hi: 186, btch: 31 usd: 178 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.518620] CPU 1: hi: 186, btch: 31 usd: 1 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.518624] CPU 2: hi: 186, btch: 31 usd: 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.518628] CPU 3: hi: 186, btch: 31 usd: 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.518632] CPU 4: hi: 186, btch: 31 usd: 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.518636] CPU 5: hi: 186, btch: 31 usd: 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.518640] CPU 6: hi: 186, btch: 31 usd: 30 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.518644] CPU 7: hi: 186, btch: 31 usd: 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.518648] CPU 8: hi: 186, btch: 31 usd: 31 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.518652] CPU 9: hi: 186, btch: 31 usd: 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.518656] CPU 10: hi: 186, btch: 31 usd: 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.518660] CPU 11: hi: 186, btch: 31 usd: 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.518667] active_anon:1484229 inactive_anon:368455 isolated_anon:0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.518667] active_file:1797005 inactive_file:1843848 isolated_file:0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.518667] unevictable:0 dirty:242798 writeback:172347 unstable:0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.518667] free:85828 slab_reclaimable:179365 slab_unreclaimable:67585 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.518667] mapped:9742 shmem:362 pagetables:8251 bounce:0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.518667] free_cma:0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.518686] Node 0 DMA free:96kB min:56kB low:68kB high:84kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15956kB managed:15872kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.518709] lowmem_reserve[]: 0 1907 15985 15985 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.518717] Node 0 DMA32 free:77064kB min:7140kB low:8924kB high:10708kB active_anon:120388kB inactive_anon:181600kB active_file:551744kB inactive_file:668500kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:2047564kB managed:1955580kB mlocked:0kB dirty:36868kB writeback:19156kB mapped:2208kB shmem:76kB slab_reclaimable:255044kB slab_unreclaimable:51120kB kernel_stack:24272kB pagetables:2240kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.518743] lowmem_reserve[]: 0 0 14078 14078 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.518749] Node 0 Normal free:141608kB min:52716kB low:65892kB high:79072kB active_anon:2725892kB inactive_anon:657836kB active_file:4994212kB inactive_file:5063636kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:14680064kB managed:14416320kB mlocked:0kB dirty:597928kB writeback:331764kB mapped:16640kB shmem:1012kB slab_reclaimable:262920kB slab_unreclaimable:94612kB kernel_stack:23104kB pagetables:14112kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:148 all_unreclaimable? no Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.518775] lowmem_reserve[]: 0 0 0 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.518782] Node 1 Normal free:124544kB min:30192kB low:37740kB high:45288kB active_anon:3090636kB inactive_anon:634384kB active_file:1642064kB inactive_file:1643256kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:8388608kB managed:8256092kB mlocked:0kB dirty:336396kB writeback:338468kB mapped:20120kB shmem:360kB slab_reclaimable:199496kB slab_unreclaimable:124592kB kernel_stack:10704kB pagetables:16652kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.518807] lowmem_reserve[]: 0 0 0 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.518814] Node 0 DMA: 0*4kB 0*8kB 2*16kB (U) 0*32kB 1*64kB (U) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 96kB Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.518833] Node 0 DMA32: 18885*4kB (UEM) 114*8kB (UM) 6*16kB (M) 21*32kB (UMR) 2*64kB (MR) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 77348kB Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.518856] Node 0 Normal: 27508*4kB (UEM) 1270*8kB (UM) 430*16kB (UM) 310*32kB (UM) 85*64kB (UM) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 142432kB Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.518878] Node 1 Normal: 22389*4kB (UEM) 2220*8kB (UEM) 655*16kB (UMR) 178*32kB (UMR) 32*64kB (UMR) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 125540kB Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.518903] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.518909] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.518914] 3642749 total pagecache pages Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.518919] 184 pages in swap cache Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.518922] Swap cache stats: add 654, delete 470, find 20/29 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.518926] Free swap = 3143168kB Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.518930] Total swap = 3145724kB Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.518933] 6283048 pages RAM Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.518936] 0 pages HighMem/MovableOnly Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.518939] 33129 pages reserved Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.518942] 0 pages hwpoisoned Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.525989] kworker/8:0: page allocation failure: order:5, mode:0x206050 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.526004] CPU: 8 PID: 13987 Comm: kworker/8:0 Tainted: G E 3.18.0-rc6-ceph-00024-g72ca172 #1 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.526011] Hardware name: Supermicro X9DRH-7TF/7F/iTF/iF/X9DRH-7TF/7F/iTF/iF, BIOS 3.0b 04/28/2014 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.526031] Workqueue: ceph-msgr con_work [libceph] Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.526037] 0000000000000005 ffff8800171cf5f8 ffffffff81733b38 ffff88067fc8e2d8 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.526045] 0000000000206050 ffff8800171cf688 ffffffff81167f8e 0000000000000000 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.526053] 0000000400000000 ffff88067fff9f48 0000001fffffffe0 0000000000000010 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.526062] Call Trace: Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.526072] [<ffffffff81733b38>] dump_stack+0x4e/0x68 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.526081] [<ffffffff81167f8e>] warn_alloc_failed+0xee/0x140 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.526088] [<ffffffff8116bc72>] __alloc_pages_nodemask+0x962/0xba0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.526096] [<ffffffff8173d03b>] ? _raw_spin_unlock+0x2b/0x40 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.526104] [<ffffffff8116bef9>] alloc_kmem_pages_node+0x49/0x60 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.526110] [<ffffffff8173095d>] kmalloc_large_node+0x28/0x56 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.526120] [<ffffffff811bd09c>] __kmalloc_node_track_caller+0x1ec/0x230 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.526130] [<ffffffff816162d7>] ? __alloc_skb+0x87/0x2a0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.526137] [<ffffffff81615e95>] __kmalloc_reserve.isra.52+0x95/0xa0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.526143] [<ffffffff816162d7>] __alloc_skb+0x87/0x2a0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.526151] [<ffffffff810a0000>] ? print_cfs_rq+0x5a0/0x1430 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.526159] [<ffffffff816748f1>] sk_stream_alloc_skb+0x41/0x130 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.526165] [<ffffffff81675364>] tcp_sendmsg+0x3b4/0xd10 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.526172] [<ffffffff816a14b5>] ? inet_sendmsg+0x5/0x110 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.526178] [<ffffffff816a157f>] inet_sendmsg+0xcf/0x110 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.526183] [<ffffffff816a14b5>] ? inet_sendmsg+0x5/0x110 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.526189] [<ffffffff8160d67f>] sock_sendmsg+0xaf/0xc0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.526198] [<ffffffffa051f1a5>] ? con_work+0x55/0x2cc0 [libceph] Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.526204] [<ffffffff8160d6d0>] kernel_sendmsg+0x40/0x60 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.526214] [<ffffffffa051d22b>] ceph_tcp_sendmsg+0x4b/0x60 [libceph] Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.526223] [<ffffffffa051fd91>] con_work+0xc41/0x2cc0 [libceph] Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.526232] [<ffffffff810801af>] ? finish_task_switch+0x3f/0x120 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.526238] [<ffffffff810801af>] ? finish_task_switch+0x3f/0x120 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.526245] [<ffffffff81073f83>] ? process_one_work+0x143/0x530 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.526250] [<ffffffff81073f83>] ? process_one_work+0x143/0x530 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.526256] [<ffffffff81073ffc>] process_one_work+0x1bc/0x530 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.526261] [<ffffffff81073f83>] ? process_one_work+0x143/0x530 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.526267] [<ffffffff8107475c>] worker_thread+0x11c/0x4b0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.526273] [<ffffffff81074640>] ? rescuer_thread+0x290/0x290 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.526279] [<ffffffff8107a664>] kthread+0xe4/0x100 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.526285] [<ffffffff8107a580>] ? flush_kthread_worker+0xf0/0xf0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.526292] [<ffffffff8173d66c>] ret_from_fork+0x7c/0xb0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.526298] [<ffffffff8107a580>] ? flush_kthread_worker+0xf0/0xf0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.526303] Mem-Info: Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.526306] Node 0 DMA per-cpu: Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.526311] CPU 0: hi: 0, btch: 1 usd: 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.526315] CPU 1: hi: 0, btch: 1 usd: 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.526319] CPU 2: hi: 0, btch: 1 usd: 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.526323] CPU 3: hi: 0, btch: 1 usd: 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.526327] CPU 4: hi: 0, btch: 1 usd: 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.526331] CPU 5: hi: 0, btch: 1 usd: 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.526335] CPU 6: hi: 0, btch: 1 usd: 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.526339] CPU 7: hi: 0, btch: 1 usd: 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.526343] CPU 8: hi: 0, btch: 1 usd: 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.526347] CPU 9: hi: 0, btch: 1 usd: 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.526351] CPU 10: hi: 0, btch: 1 usd: 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.526355] CPU 11: hi: 0, btch: 1 usd: 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.526359] Node 0 DMA32 per-cpu: Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.526363] CPU 0: hi: 186, btch: 31 usd: 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.526367] CPU 1: hi: 186, btch: 31 usd: 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.526371] CPU 2: hi: 186, btch: 31 usd: 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.526375] CPU 3: hi: 186, btch: 31 usd: 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.526379] CPU 4: hi: 186, btch: 31 usd: 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.526383] CPU 5: hi: 186, btch: 31 usd: 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.526387] CPU 6: hi: 186, btch: 31 usd: 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.526391] CPU 7: hi: 186, btch: 31 usd: 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.526395] CPU 8: hi: 186, btch: 31 usd: 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.526399] CPU 9: hi: 186, btch: 31 usd: 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.526403] CPU 10: hi: 186, btch: 31 usd: 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.526407] CPU 11: hi: 186, btch: 31 usd: 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.526411] Node 0 Normal per-cpu: Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.526415] CPU 0: hi: 186, btch: 31 usd: 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.526419] CPU 1: hi: 186, btch: 31 usd: 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.526423] CPU 2: hi: 186, btch: 31 usd: 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.526427] CPU 3: hi: 186, btch: 31 usd: 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.526431] CPU 4: hi: 186, btch: 31 usd: 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.526435] CPU 5: hi: 186, btch: 31 usd: 185 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.526439] CPU 6: hi: 186, btch: 31 usd: 20 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.526443] CPU 7: hi: 186, btch: 31 usd: 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.526447] CPU 8: hi: 186, btch: 31 usd: 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.526451] CPU 9: hi: 186, btch: 31 usd: 152 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.526455] CPU 10: hi: 186, btch: 31 usd: 1 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.526459] CPU 11: hi: 186, btch: 31 usd: 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.526463] Node 1 Normal per-cpu: Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.526467] CPU 0: hi: 186, btch: 31 usd: 4 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.526471] CPU 1: hi: 186, btch: 31 usd: 94 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.526475] CPU 2: hi: 186, btch: 31 usd: 57 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.526480] CPU 3: hi: 186, btch: 31 usd: 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.526484] CPU 4: hi: 186, btch: 31 usd: 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.526488] CPU 5: hi: 186, btch: 31 usd: 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.526492] CPU 6: hi: 186, btch: 31 usd: 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.526496] CPU 7: hi: 186, btch: 31 usd: 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.526500] CPU 8: hi: 186, btch: 31 usd: 31 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.526504] CPU 9: hi: 186, btch: 31 usd: 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.526508] CPU 10: hi: 186, btch: 31 usd: 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.526512] CPU 11: hi: 186, btch: 31 usd: 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.526519] active_anon:1484229 inactive_anon:368455 isolated_anon:0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.526519] active_file:1797005 inactive_file:1843789 isolated_file:0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.526519] unevictable:0 dirty:235862 writeback:179283 unstable:0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.526519] free:85670 slab_reclaimable:179365 slab_unreclaimable:67585 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.526519] mapped:9742 shmem:362 pagetables:8251 bounce:0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.526519] free_cma:0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.526537] Node 0 DMA free:96kB min:56kB low:68kB high:84kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15956kB managed:15872kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.526560] lowmem_reserve[]: 0 1907 15985 15985 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.526568] Node 0 DMA32 free:77064kB min:7140kB low:8924kB high:10708kB active_anon:120388kB inactive_anon:181600kB active_file:551744kB inactive_file:668500kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:2047564kB managed:1955580kB mlocked:0kB dirty:36196kB writeback:19828kB mapped:2208kB shmem:76kB slab_reclaimable:255044kB slab_unreclaimable:51120kB kernel_stack:24272kB pagetables:2240kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.526593] lowmem_reserve[]: 0 0 14078 14078 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.526600] Node 0 Normal free:140356kB min:52716kB low:65892kB high:79072kB active_anon:2725892kB inactive_anon:657836kB active_file:4994212kB inactive_file:5063904kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:14680064kB managed:14416320kB mlocked:0kB dirty:574616kB writeback:355076kB mapped:16640kB shmem:1012kB slab_reclaimable:262920kB slab_unreclaimable:94612kB kernel_stack:23104kB pagetables:14112kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.526626] lowmem_reserve[]: 0 0 0 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.526633] Node 1 Normal free:124544kB min:30192kB low:37740kB high:45288kB active_anon:3090636kB inactive_anon:634384kB active_file:1642064kB inactive_file:1642752kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:8388608kB managed:8256092kB mlocked:0kB dirty:332636kB writeback:342228kB mapped:20120kB shmem:360kB slab_reclaimable:199496kB slab_unreclaimable:124592kB kernel_stack:10704kB pagetables:16652kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.526658] lowmem_reserve[]: 0 0 0 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.526664] Node 0 DMA: 0*4kB 0*8kB 2*16kB (U) 0*32kB 1*64kB (U) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 96kB Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.526684] Node 0 DMA32: 18885*4kB (UEM) 114*8kB (UM) 6*16kB (M) 21*32kB (UMR) 2*64kB (MR) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 77348kB Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.526707] Node 0 Normal: 27040*4kB (UEM) 1335*8kB (UM) 404*16kB (UM) 318*32kB (UM) 89*64kB (UM) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 141176kB Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.526730] Node 1 Normal: 22230*4kB (UEM) 2256*8kB (UEM) 675*16kB (UMR) 176*32kB (UMR) 32*64kB (UMR) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 125448kB Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.526754] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.526760] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.526765] 3642937 total pagecache pages Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.526769] 184 pages in swap cache Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.526773] Swap cache stats: add 654, delete 470, find 20/29 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.526777] Free swap = 3143168kB Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.526780] Total swap = 3145724kB Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.526784] 6283048 pages RAM Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.526787] 0 pages HighMem/MovableOnly Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.526790] 33129 pages reserved Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.526793] 0 pages hwpoisoned Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.542065] kworker/8:0: page allocation failure: order:5, mode:0x206050 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.542079] CPU: 8 PID: 13987 Comm: kworker/8:0 Tainted: G E 3.18.0-rc6-ceph-00024-g72ca172 #1 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.542086] Hardware name: Supermicro X9DRH-7TF/7F/iTF/iF/X9DRH-7TF/7F/iTF/iF, BIOS 3.0b 04/28/2014 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.542107] Workqueue: ceph-msgr con_work [libceph] Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.542113] 0000000000000005 ffff8800171cf478 ffffffff81733b38 ffff88067fc8e2d8 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.542122] 0000000000206050 ffff8800171cf508 ffffffff81167f8e 0000000000000000 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.542131] 0000000400000000 ffff88067fff9f48 0000001fffffffe0 0000000000000010 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.542139] Call Trace: Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.542150] [<ffffffff81733b38>] dump_stack+0x4e/0x68 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.542160] [<ffffffff81167f8e>] warn_alloc_failed+0xee/0x140 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.542167] [<ffffffff8116bc72>] __alloc_pages_nodemask+0x962/0xba0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.542176] [<ffffffff810ca408>] ? lock_timer_base.isra.43+0x38/0x70 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.542183] [<ffffffff8116bef9>] alloc_kmem_pages_node+0x49/0x60 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.542189] [<ffffffff8173095d>] kmalloc_large_node+0x28/0x56 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.542199] [<ffffffff811bd09c>] __kmalloc_node_track_caller+0x1ec/0x230 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.542210] [<ffffffff816162d7>] ? __alloc_skb+0x87/0x2a0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.542217] [<ffffffff81615e95>] __kmalloc_reserve.isra.52+0x95/0xa0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.542223] [<ffffffff816162d7>] __alloc_skb+0x87/0x2a0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.542231] [<ffffffff816748f1>] sk_stream_alloc_skb+0x41/0x130 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.542237] [<ffffffff81675364>] tcp_sendmsg+0x3b4/0xd10 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.542244] [<ffffffff816a14b5>] ? inet_sendmsg+0x5/0x110 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.542250] [<ffffffff816a157f>] inet_sendmsg+0xcf/0x110 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.542255] [<ffffffff816a14b5>] ? inet_sendmsg+0x5/0x110 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.542261] [<ffffffff8160d67f>] sock_sendmsg+0xaf/0xc0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.542270] [<ffffffff8173cf59>] ? _raw_spin_unlock_bh+0x39/0x40 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.542276] [<ffffffff81675083>] ? tcp_sendmsg+0xd3/0xd10 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.542281] [<ffffffff8160d6d0>] kernel_sendmsg+0x40/0x60 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.542287] [<ffffffff8160fe4b>] sock_no_sendpage+0x9b/0xb0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.542293] [<ffffffff81674a37>] tcp_sendpage+0x57/0x590 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.542298] [<ffffffff816a15c5>] ? inet_sendpage+0x5/0x130 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.542309] [<ffffffffa051f1a5>] ? con_work+0x55/0x2cc0 [libceph] Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.542315] [<ffffffff816a16a2>] inet_sendpage+0xe2/0x130 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.542320] [<ffffffff816a15c5>] ? inet_sendpage+0x5/0x130 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.542328] [<ffffffff8160a72b>] kernel_sendpage+0x1b/0x30 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.542338] [<ffffffffa051e9db>] ceph_tcp_sendpage+0x5b/0x110 [libceph] Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.542348] [<ffffffffa051fc48>] con_work+0xaf8/0x2cc0 [libceph] Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.542356] [<ffffffff810801af>] ? finish_task_switch+0x3f/0x120 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.542363] [<ffffffff81073f83>] ? process_one_work+0x143/0x530 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.542369] [<ffffffff81073f01>] ? process_one_work+0xc1/0x530 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.542374] [<ffffffff81073ffc>] process_one_work+0x1bc/0x530 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.542379] [<ffffffff81073f83>] ? process_one_work+0x143/0x530 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.542385] [<ffffffff8107475c>] worker_thread+0x11c/0x4b0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.542391] [<ffffffff81074640>] ? rescuer_thread+0x290/0x290 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.542397] [<ffffffff8107a664>] kthread+0xe4/0x100 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.542403] [<ffffffff8107a580>] ? flush_kthread_worker+0xf0/0xf0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.542410] [<ffffffff8173d66c>] ret_from_fork+0x7c/0xb0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.542416] [<ffffffff8107a580>] ? flush_kthread_worker+0xf0/0xf0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.542421] Mem-Info: Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.542424] Node 0 DMA per-cpu: Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.542429] CPU 0: hi: 0, btch: 1 usd: 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.542433] CPU 1: hi: 0, btch: 1 usd: 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.542437] CPU 2: hi: 0, btch: 1 usd: 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.542441] CPU 3: hi: 0, btch: 1 usd: 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.542445] CPU 4: hi: 0, btch: 1 usd: 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.542449] CPU 5: hi: 0, btch: 1 usd: 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.542453] CPU 6: hi: 0, btch: 1 usd: 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.542457] CPU 7: hi: 0, btch: 1 usd: 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.542461] CPU 8: hi: 0, btch: 1 usd: 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.542465] CPU 9: hi: 0, btch: 1 usd: 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.542469] CPU 10: hi: 0, btch: 1 usd: 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.542473] CPU 11: hi: 0, btch: 1 usd: 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.542477] Node 0 DMA32 per-cpu: Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.542482] CPU 0: hi: 186, btch: 31 usd: 82 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.542486] CPU 1: hi: 186, btch: 31 usd: 24 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.542490] CPU 2: hi: 186, btch: 31 usd: 31 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.542494] CPU 3: hi: 186, btch: 31 usd: 15 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.542498] CPU 4: hi: 186, btch: 31 usd: 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.542502] CPU 5: hi: 186, btch: 31 usd: 27 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.542506] CPU 6: hi: 186, btch: 31 usd: 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.542510] CPU 7: hi: 186, btch: 31 usd: 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.542514] CPU 8: hi: 186, btch: 31 usd: 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.542519] CPU 9: hi: 186, btch: 31 usd: 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.542523] CPU 10: hi: 186, btch: 31 usd: 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.542527] CPU 11: hi: 186, btch: 31 usd: 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.542530] Node 0 Normal per-cpu: Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.542534] CPU 0: hi: 186, btch: 31 usd: 155 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.542539] CPU 1: hi: 186, btch: 31 usd: 28 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.542543] CPU 2: hi: 186, btch: 31 usd: 43 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.542547] CPU 3: hi: 186, btch: 31 usd: 27 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.542551] CPU 4: hi: 186, btch: 31 usd: 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.542555] CPU 5: hi: 186, btch: 31 usd: 27 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.542559] CPU 6: hi: 186, btch: 31 usd: 22 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.542563] CPU 7: hi: 186, btch: 31 usd: 14 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.542567] CPU 8: hi: 186, btch: 31 usd: 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.542571] CPU 9: hi: 186, btch: 31 usd: 17 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.542575] CPU 10: hi: 186, btch: 31 usd: 7 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.542579] CPU 11: hi: 186, btch: 31 usd: 11 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.542583] Node 1 Normal per-cpu: Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.542587] CPU 0: hi: 186, btch: 31 usd: 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.542591] CPU 1: hi: 186, btch: 31 usd: 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.542595] CPU 2: hi: 186, btch: 31 usd: 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.542600] CPU 3: hi: 186, btch: 31 usd: 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.542604] CPU 4: hi: 186, btch: 31 usd: 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.542608] CPU 5: hi: 186, btch: 31 usd: 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.542612] CPU 6: hi: 186, btch: 31 usd: 1 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.542616] CPU 7: hi: 186, btch: 31 usd: 30 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.542620] CPU 8: hi: 186, btch: 31 usd: 31 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.542624] CPU 9: hi: 186, btch: 31 usd: 30 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.542628] CPU 10: hi: 186, btch: 31 usd: 30 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.542632] CPU 11: hi: 186, btch: 31 usd: 30 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.542638] active_anon:1484229 inactive_anon:368455 isolated_anon:0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.542638] active_file:1797005 inactive_file:1850344 isolated_file:0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.542638] unevictable:0 dirty:229898 writeback:191503 unstable:0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.542638] free:79116 slab_reclaimable:179365 slab_unreclaimable:67585 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.542638] mapped:9742 shmem:362 pagetables:8251 bounce:0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.542638] free_cma:0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.542658] Node 0 DMA free:96kB min:56kB low:68kB high:84kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15956kB managed:15872kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.542681] lowmem_reserve[]: 0 1907 15985 15985 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.542689] Node 0 DMA32 free:73656kB min:7140kB low:8924kB high:10708kB active_anon:120388kB inactive_anon:181600kB active_file:551744kB inactive_file:671348kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:2047564kB managed:1955580kB mlocked:0kB dirty:38660kB writeback:19828kB mapped:2208kB shmem:76kB slab_reclaimable:255044kB slab_unreclaimable:51120kB kernel_stack:24272kB pagetables:2240kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:648 all_unreclaimable? no Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.542715] lowmem_reserve[]: 0 0 14078 14078 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.542722] Node 0 Normal free:116424kB min:52716kB low:65892kB high:79072kB active_anon:2725892kB inactive_anon:657836kB active_file:4994212kB inactive_file:5087276kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:14680064kB managed:14416320kB mlocked:0kB dirty:575368kB writeback:376884kB mapped:16640kB shmem:1012kB slab_reclaimable:262920kB slab_unreclaimable:94612kB kernel_stack:23104kB pagetables:14112kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.542748] lowmem_reserve[]: 0 0 0 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.542754] Node 1 Normal free:125048kB min:30192kB low:37740kB high:45288kB active_anon:3090636kB inactive_anon:634384kB active_file:1642064kB inactive_file:1642752kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:8388608kB managed:8256092kB mlocked:0kB dirty:305564kB writeback:370052kB mapped:20120kB shmem:360kB slab_reclaimable:199496kB slab_unreclaimable:124592kB kernel_stack:10704kB pagetables:16652kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.542781] lowmem_reserve[]: 0 0 0 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.542787] Node 0 DMA: 0*4kB 0*8kB 2*16kB (U) 0*32kB 1*64kB (U) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 96kB Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.542807] Node 0 DMA32: 18288*4kB (U) 0*8kB 0*16kB 17*32kB (R) 1*64kB (R) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 73760kB Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.542828] Node 0 Normal: 25235*4kB (UEM) 890*8kB (UM) 173*16kB (UM) 102*32kB (UM) 15*64kB (UM) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 115052kB Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.542851] Node 1 Normal: 21907*4kB (UEM) 2347*8kB (UEM) 702*16kB (UMR) 172*32kB (UMR) 32*64kB (UMR) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 125188kB Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.542875] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.542881] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.542887] 3650001 total pagecache pages Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.542891] 184 pages in swap cache Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.542895] Swap cache stats: add 654, delete 470, find 20/29 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.542899] Free swap = 3143168kB Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.542902] Total swap = 3145724kB Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.542906] 6283048 pages RAM Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.542909] 0 pages HighMem/MovableOnly Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.542912] 33129 pages reserved Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.542915] 0 pages hwpoisoned Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.547837] kworker/6:4: page allocation failure: order:5, mode:0x206050 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.547858] CPU: 6 PID: 1443 Comm: kworker/6:4 Tainted: G E 3.18.0-rc6-ceph-00024-g72ca172 #1 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.547865] Hardware name: Supermicro X9DRH-7TF/7F/iTF/iF/X9DRH-7TF/7F/iTF/iF, BIOS 3.0b 04/28/2014 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.547888] Workqueue: ceph-msgr con_work [libceph] Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.547894] 0000000000000005 ffff88010fa5b478 ffffffff81733b38 ffff88067fc0e2d8 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.547903] 0000000000206050 ffff88010fa5b508 ffffffff81167f8e 0000000000000000 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.547911] 0000000400000000 ffff88067fff9f48 0000001fffffffe0 0000000000000010 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.547920] Call Trace: Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.547930] [<ffffffff81733b38>] dump_stack+0x4e/0x68 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.547940] [<ffffffff81167f8e>] warn_alloc_failed+0xee/0x140 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.547948] [<ffffffff8116bc72>] __alloc_pages_nodemask+0x962/0xba0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.547957] [<ffffffff810ca408>] ? lock_timer_base.isra.43+0x38/0x70 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.547982] [<ffffffff8116bef9>] alloc_kmem_pages_node+0x49/0x60 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.547994] [<ffffffff8173095d>] kmalloc_large_node+0x28/0x56 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.548005] [<ffffffff811bd09c>] __kmalloc_node_track_caller+0x1ec/0x230 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.548015] [<ffffffff816162d7>] ? __alloc_skb+0x87/0x2a0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.548021] [<ffffffff81615e95>] __kmalloc_reserve.isra.52+0x95/0xa0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.548035] [<ffffffff816162d7>] __alloc_skb+0x87/0x2a0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.548044] [<ffffffff816748f1>] sk_stream_alloc_skb+0x41/0x130 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.548050] [<ffffffff81675364>] tcp_sendmsg+0x3b4/0xd10 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.548056] [<ffffffff816a14b5>] ? inet_sendmsg+0x5/0x110 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.548065] [<ffffffff816a157f>] inet_sendmsg+0xcf/0x110 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.548072] [<ffffffff816a14b5>] ? inet_sendmsg+0x5/0x110 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.548080] [<ffffffff810a98fd>] ? trace_hardirqs_on+0xd/0x10 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.548089] [<ffffffff8160d67f>] sock_sendmsg+0xaf/0xc0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.548096] [<ffffffff81611b23>] ? release_sock+0x193/0x1d0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.548102] [<ffffffff816732a0>] ? tcp_recvmsg+0x610/0xca0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.548107] [<ffffffff816a13a5>] ? inet_recvmsg+0x5/0x110 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.548113] [<ffffffff8160d6d0>] kernel_sendmsg+0x40/0x60 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.548119] [<ffffffff8160fe4b>] sock_no_sendpage+0x9b/0xb0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.548125] [<ffffffff81674a37>] tcp_sendpage+0x57/0x590 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.548130] [<ffffffff816a15c5>] ? inet_sendpage+0x5/0x130 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.548140] [<ffffffffa051f1a5>] ? con_work+0x55/0x2cc0 [libceph] Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.548146] [<ffffffff816a16a2>] inet_sendpage+0xe2/0x130 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.548151] [<ffffffff816a15c5>] ? inet_sendpage+0x5/0x130 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.548158] [<ffffffff8160a72b>] kernel_sendpage+0x1b/0x30 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.548168] [<ffffffffa051e9db>] ceph_tcp_sendpage+0x5b/0x110 [libceph] Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.548178] [<ffffffffa051fc48>] con_work+0xaf8/0x2cc0 [libceph] Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.548186] [<ffffffff810801af>] ? finish_task_switch+0x3f/0x120 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.548192] [<ffffffff8107408a>] ? process_one_work+0x24a/0x530 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.548198] [<ffffffff81073f83>] ? process_one_work+0x143/0x530 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.548203] [<ffffffff81073f01>] ? process_one_work+0xc1/0x530 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.548209] [<ffffffff81073ffc>] process_one_work+0x1bc/0x530 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.548214] [<ffffffff81073f83>] ? process_one_work+0x143/0x530 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.548220] [<ffffffff8107475c>] worker_thread+0x11c/0x4b0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.548225] [<ffffffff81074640>] ? rescuer_thread+0x290/0x290 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.548232] [<ffffffff8107a664>] kthread+0xe4/0x100 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.548238] [<ffffffff8107a580>] ? flush_kthread_worker+0xf0/0xf0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.548246] [<ffffffff8173d66c>] ret_from_fork+0x7c/0xb0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.548252] [<ffffffff8107a580>] ? flush_kthread_worker+0xf0/0xf0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.548257] Mem-Info: Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.548261] Node 0 DMA per-cpu: Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.548266] CPU 0: hi: 0, btch: 1 usd: 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.548270] CPU 1: hi: 0, btch: 1 usd: 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.548274] CPU 2: hi: 0, btch: 1 usd: 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.548278] CPU 3: hi: 0, btch: 1 usd: 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.548282] CPU 4: hi: 0, btch: 1 usd: 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.548286] CPU 5: hi: 0, btch: 1 usd: 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.548290] CPU 6: hi: 0, btch: 1 usd: 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.548294] CPU 7: hi: 0, btch: 1 usd: 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.548298] CPU 8: hi: 0, btch: 1 usd: 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.548302] CPU 9: hi: 0, btch: 1 usd: 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.548306] CPU 10: hi: 0, btch: 1 usd: 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.548310] CPU 11: hi: 0, btch: 1 usd: 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.548314] Node 0 DMA32 per-cpu: Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.548319] CPU 0: hi: 186, btch: 31 usd: 163 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.548323] CPU 1: hi: 186, btch: 31 usd: 14 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.548327] CPU 2: hi: 186, btch: 31 usd: 163 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.548331] CPU 3: hi: 186, btch: 31 usd: 15 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.548335] CPU 4: hi: 186, btch: 31 usd: 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.548339] CPU 5: hi: 186, btch: 31 usd: 178 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.548343] CPU 6: hi: 186, btch: 31 usd: 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.548347] CPU 7: hi: 186, btch: 31 usd: 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.548351] CPU 8: hi: 186, btch: 31 usd: 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.548355] CPU 9: hi: 186, btch: 31 usd: 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.548359] CPU 10: hi: 186, btch: 31 usd: 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.548363] CPU 11: hi: 186, btch: 31 usd: 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.548366] Node 0 Normal per-cpu: Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.548370] CPU 0: hi: 186, btch: 31 usd: 155 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.548374] CPU 1: hi: 186, btch: 31 usd: 24 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.548378] CPU 2: hi: 186, btch: 31 usd: 167 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.548382] CPU 3: hi: 186, btch: 31 usd: 27 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.548386] CPU 4: hi: 186, btch: 31 usd: 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.548390] CPU 5: hi: 186, btch: 31 usd: 167 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.548394] CPU 6: hi: 186, btch: 31 usd: 22 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.548398] CPU 7: hi: 186, btch: 31 usd: 30 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.548402] CPU 8: hi: 186, btch: 31 usd: 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.548406] CPU 9: hi: 186, btch: 31 usd: 9 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.548410] CPU 10: hi: 186, btch: 31 usd: 30 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.548414] CPU 11: hi: 186, btch: 31 usd: 30 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.548418] Node 1 Normal per-cpu: Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.548422] CPU 0: hi: 186, btch: 31 usd: 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.548426] CPU 1: hi: 186, btch: 31 usd: 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.548431] CPU 2: hi: 186, btch: 31 usd: 164 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.548434] CPU 3: hi: 186, btch: 31 usd: 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.548438] CPU 4: hi: 186, btch: 31 usd: 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.548442] CPU 5: hi: 186, btch: 31 usd: 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.548446] CPU 6: hi: 186, btch: 31 usd: 17 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.548450] CPU 7: hi: 186, btch: 31 usd: 30 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.548454] CPU 8: hi: 186, btch: 31 usd: 31 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.548458] CPU 9: hi: 186, btch: 31 usd: 30 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.548462] CPU 10: hi: 186, btch: 31 usd: 30 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.548466] CPU 11: hi: 186, btch: 31 usd: 34 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.548474] active_anon:1484229 inactive_anon:368455 isolated_anon:0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.548474] active_file:1796943 inactive_file:1850544 isolated_file:0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.548474] unevictable:0 dirty:229948 writeback:193283 unstable:0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.548474] free:77899 slab_reclaimable:179365 slab_unreclaimable:67658 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.548474] mapped:9748 shmem:362 pagetables:8251 bounce:0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.548474] free_cma:0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.548493] Node 0 DMA free:96kB min:56kB low:68kB high:84kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15956kB managed:15872kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.548517] lowmem_reserve[]: 0 1907 15985 15985 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.548525] Node 0 DMA32 free:73152kB min:7140kB low:8924kB high:10708kB active_anon:120388kB inactive_anon:181600kB active_file:551744kB inactive_file:670704kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:2047564kB managed:1955580kB mlocked:0kB dirty:39512kB writeback:20096kB mapped:2208kB shmem:76kB slab_reclaimable:255044kB slab_unreclaimable:51120kB kernel_stack:24272kB pagetables:2240kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.548551] lowmem_reserve[]: 0 0 14078 14078 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.548557] Node 0 Normal free:114304kB min:52716kB low:65892kB high:79072kB active_anon:2725888kB inactive_anon:657836kB active_file:4994216kB inactive_file:5088648kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:14680064kB managed:14416320kB mlocked:0kB dirty:579064kB writeback:378620kB mapped:16640kB shmem:1012kB slab_reclaimable:262920kB slab_unreclaimable:94836kB kernel_stack:23104kB pagetables:14112kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:128 all_unreclaimable? no Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.548584] lowmem_reserve[]: 0 0 0 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.548590] Node 1 Normal free:124044kB min:30192kB low:37740kB high:45288kB active_anon:3090640kB inactive_anon:634384kB active_file:1641812kB inactive_file:1642824kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:8388608kB managed:8256092kB mlocked:0kB dirty:300240kB writeback:375168kB mapped:20144kB shmem:360kB slab_reclaimable:199496kB slab_unreclaimable:124660kB kernel_stack:10704kB pagetables:16652kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.548616] lowmem_reserve[]: 0 0 0 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.548622] Node 0 DMA: 0*4kB 0*8kB 2*16kB (U) 0*32kB 1*64kB (U) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 96kB Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.548642] Node 0 DMA32: 18177*4kB (UM) 9*8kB (U) 0*16kB 17*32kB (R) 1*64kB (R) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 73388kB Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.548664] Node 0 Normal: 25963*4kB (UEM) 743*8kB (UM) 135*16kB (UM) 62*32kB (UM) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 113940kB Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.548685] Node 1 Normal: 21541*4kB (UEM) 2381*8kB (UEM) 719*16kB (UMR) 174*32kB (UMR) 32*64kB (UMR) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 124332kB Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.548710] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.548716] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.548721] 3649641 total pagecache pages Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.548726] 184 pages in swap cache Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.548730] Swap cache stats: add 654, delete 470, find 20/29 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.548734] Free swap = 3143168kB Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.548737] Total swap = 3145724kB Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.548741] 6283048 pages RAM Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.548744] 0 pages HighMem/MovableOnly Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.548747] 33129 pages reserved Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.548750] 0 pages hwpoisoned Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.552165] kworker/6:4: page allocation failure: order:5, mode:0x206050 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.552174] CPU: 6 PID: 1443 Comm: kworker/6:4 Tainted: G E 3.18.0-rc6-ceph-00024-g72ca172 #1 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.552183] Hardware name: Supermicro X9DRH-7TF/7F/iTF/iF/X9DRH-7TF/7F/iTF/iF, BIOS 3.0b 04/28/2014 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.552197] Workqueue: ceph-msgr con_work [libceph] Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.552201] 0000000000000005 ffff88010fa5b478 ffffffff81733b38 ffff88067fc0e2d8 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.552209] 0000000000206050 ffff88010fa5b508 ffffffff81167f8e 0000000000000000 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.552217] 0000000400000000 ffff88067fff9f48 0000001fffffffe0 0000000000000010 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.552225] Call Trace: Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.552231] [<ffffffff81733b38>] dump_stack+0x4e/0x68 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.552238] [<ffffffff81167f8e>] warn_alloc_failed+0xee/0x140 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.552244] [<ffffffff8116bc72>] __alloc_pages_nodemask+0x962/0xba0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.552251] [<ffffffff810ca408>] ? lock_timer_base.isra.43+0x38/0x70 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.552258] [<ffffffff8116bef9>] alloc_kmem_pages_node+0x49/0x60 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.552264] [<ffffffff8173095d>] kmalloc_large_node+0x28/0x56 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.552271] [<ffffffff811bd09c>] __kmalloc_node_track_caller+0x1ec/0x230 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.552278] [<ffffffff816162d7>] ? __alloc_skb+0x87/0x2a0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.552284] [<ffffffff81615e95>] __kmalloc_reserve.isra.52+0x95/0xa0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.552290] [<ffffffff816162d7>] __alloc_skb+0x87/0x2a0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.552296] [<ffffffff816748f1>] sk_stream_alloc_skb+0x41/0x130 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.552302] [<ffffffff81675364>] tcp_sendmsg+0x3b4/0xd10 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.552307] [<ffffffff816a14b5>] ? inet_sendmsg+0x5/0x110 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.552313] [<ffffffff816a157f>] inet_sendmsg+0xcf/0x110 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.552318] [<ffffffff816a14b5>] ? inet_sendmsg+0x5/0x110 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.552324] [<ffffffff810a98fd>] ? trace_hardirqs_on+0xd/0x10 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.552330] [<ffffffff8160d67f>] sock_sendmsg+0xaf/0xc0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.552336] [<ffffffff81611b23>] ? release_sock+0x193/0x1d0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.552341] [<ffffffff816732a0>] ? tcp_recvmsg+0x610/0xca0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.552346] [<ffffffff816a13a5>] ? inet_recvmsg+0x5/0x110 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.552352] [<ffffffff8160d6d0>] kernel_sendmsg+0x40/0x60 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.552357] [<ffffffff8160fe4b>] sock_no_sendpage+0x9b/0xb0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.552363] [<ffffffff81674a37>] tcp_sendpage+0x57/0x590 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.552369] [<ffffffff816a15c5>] ? inet_sendpage+0x5/0x130 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.552374] [<ffffffff816a16a2>] inet_sendpage+0xe2/0x130 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.552379] [<ffffffff816a15c5>] ? inet_sendpage+0x5/0x130 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.552386] [<ffffffff8160a72b>] kernel_sendpage+0x1b/0x30 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.552396] [<ffffffffa051e9db>] ceph_tcp_sendpage+0x5b/0x110 [libceph] Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.552424] [<ffffffffa051fc48>] con_work+0xaf8/0x2cc0 [libceph] Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.552433] [<ffffffff81096163>] ? pick_next_task_fair+0xf3/0x560 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.552440] [<ffffffff810801af>] ? finish_task_switch+0x3f/0x120 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.552446] [<ffffffff81073f83>] ? process_one_work+0x143/0x530 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.552451] [<ffffffff81073f01>] ? process_one_work+0xc1/0x530 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.552456] [<ffffffff81073ffc>] process_one_work+0x1bc/0x530 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.552462] [<ffffffff81073f83>] ? process_one_work+0x143/0x530 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.552468] [<ffffffff8107475c>] worker_thread+0x11c/0x4b0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.552473] [<ffffffff81074640>] ? rescuer_thread+0x290/0x290 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.552479] [<ffffffff8107a664>] kthread+0xe4/0x100 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.552485] [<ffffffff8107a580>] ? flush_kthread_worker+0xf0/0xf0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.552492] [<ffffffff8173d66c>] ret_from_fork+0x7c/0xb0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.552498] [<ffffffff8107a580>] ? flush_kthread_worker+0xf0/0xf0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.552502] Mem-Info: Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.552505] Node 0 DMA per-cpu: Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.552509] CPU 0: hi: 0, btch: 1 usd: 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.552513] CPU 1: hi: 0, btch: 1 usd: 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.552517] CPU 2: hi: 0, btch: 1 usd: 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.552522] CPU 3: hi: 0, btch: 1 usd: 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.552526] CPU 4: hi: 0, btch: 1 usd: 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.552530] CPU 5: hi: 0, btch: 1 usd: 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.552533] CPU 6: hi: 0, btch: 1 usd: 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.552538] CPU 7: hi: 0, btch: 1 usd: 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.552542] CPU 8: hi: 0, btch: 1 usd: 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.552545] CPU 9: hi: 0, btch: 1 usd: 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.552549] CPU 10: hi: 0, btch: 1 usd: 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.552553] CPU 11: hi: 0, btch: 1 usd: 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.552557] Node 0 DMA32 per-cpu: Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.552562] CPU 0: hi: 186, btch: 31 usd: 155 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.552566] CPU 1: hi: 186, btch: 31 usd: 14 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.552570] CPU 2: hi: 186, btch: 31 usd: 181 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.552574] CPU 3: hi: 186, btch: 31 usd: 15 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.552578] CPU 4: hi: 186, btch: 31 usd: 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.552582] CPU 5: hi: 186, btch: 31 usd: 178 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.552586] CPU 6: hi: 186, btch: 31 usd: 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.552590] CPU 7: hi: 186, btch: 31 usd: 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.552594] CPU 8: hi: 186, btch: 31 usd: 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.552598] CPU 9: hi: 186, btch: 31 usd: 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.552602] CPU 10: hi: 186, btch: 31 usd: 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.552606] CPU 11: hi: 186, btch: 31 usd: 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.552609] Node 0 Normal per-cpu: Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.552613] CPU 0: hi: 186, btch: 31 usd: 169 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.552617] CPU 1: hi: 186, btch: 31 usd: 24 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.552622] CPU 2: hi: 186, btch: 31 usd: 167 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.552626] CPU 3: hi: 186, btch: 31 usd: 27 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.552630] CPU 4: hi: 186, btch: 31 usd: 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.552634] CPU 5: hi: 186, btch: 31 usd: 167 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.552638] CPU 6: hi: 186, btch: 31 usd: 22 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.552642] CPU 7: hi: 186, btch: 31 usd: 30 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.552646] CPU 8: hi: 186, btch: 31 usd: 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.552650] CPU 9: hi: 186, btch: 31 usd: 14 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.552654] CPU 10: hi: 186, btch: 31 usd: 30 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.552658] CPU 11: hi: 186, btch: 31 usd: 24 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.552662] Node 1 Normal per-cpu: Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.552666] CPU 0: hi: 186, btch: 31 usd: 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.552670] CPU 1: hi: 186, btch: 31 usd: 15 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.552674] CPU 2: hi: 186, btch: 31 usd: 184 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.552678] CPU 3: hi: 186, btch: 31 usd: 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.552682] CPU 4: hi: 186, btch: 31 usd: 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.552686] CPU 5: hi: 186, btch: 31 usd: 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.552691] CPU 6: hi: 186, btch: 31 usd: 21 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.552695] CPU 7: hi: 186, btch: 31 usd: 74 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.552699] CPU 8: hi: 186, btch: 31 usd: 31 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.552703] CPU 9: hi: 186, btch: 31 usd: 30 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.552707] CPU 10: hi: 186, btch: 31 usd: 60 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.552711] CPU 11: hi: 186, btch: 31 usd: 35 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.552718] active_anon:1484229 inactive_anon:368455 isolated_anon:0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.552718] active_file:1796943 inactive_file:1849010 isolated_file:0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.552718] unevictable:0 dirty:222692 writeback:200295 unstable:0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.552718] free:79323 slab_reclaimable:179365 slab_unreclaimable:67658 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.552718] mapped:9748 shmem:362 pagetables:8251 bounce:0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.552718] free_cma:0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.552737] Node 0 DMA free:96kB min:56kB low:68kB high:84kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15956kB managed:15872kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.552760] lowmem_reserve[]: 0 1907 15985 15985 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.552767] Node 0 DMA32 free:74368kB min:7140kB low:8924kB high:10708kB active_anon:120388kB inactive_anon:181600kB active_file:551744kB inactive_file:669424kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:2047564kB managed:1955580kB mlocked:0kB dirty:39288kB writeback:20320kB mapped:2208kB shmem:76kB slab_reclaimable:255044kB slab_unreclaimable:51120kB kernel_stack:24272kB pagetables:2240kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:160 all_unreclaimable? no Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.552793] lowmem_reserve[]: 0 0 14078 14078 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.552800] Node 0 Normal free:119344kB min:52716kB low:65892kB high:79072kB active_anon:2725888kB inactive_anon:657836kB active_file:4994216kB inactive_file:5083792kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:14680064kB managed:14416320kB mlocked:0kB dirty:568536kB writeback:389148kB mapped:16640kB shmem:1012kB slab_reclaimable:262920kB slab_unreclaimable:94836kB kernel_stack:23104kB pagetables:14112kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:128 all_unreclaimable? no Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.552826] lowmem_reserve[]: 0 0 0 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.552833] Node 1 Normal free:122980kB min:30192kB low:37740kB high:45288kB active_anon:3090640kB inactive_anon:634384kB active_file:1641812kB inactive_file:1642824kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:8388608kB managed:8256092kB mlocked:0kB dirty:282192kB writeback:392464kB mapped:20144kB shmem:360kB slab_reclaimable:199496kB slab_unreclaimable:124660kB kernel_stack:10704kB pagetables:16652kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.552859] lowmem_reserve[]: 0 0 0 0 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.552865] Node 0 DMA: 0*4kB 0*8kB 2*16kB (U) 0*32kB 1*64kB (U) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 96kB Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.552885] Node 0 DMA32: 18321*4kB (UM) 74*8kB (UM) 7*16kB (UM) 18*32kB (MR) 1*64kB (R) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 74628kB Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.552908] Node 0 Normal: 27071*4kB (UEM) 769*8kB (UM) 149*16kB (UM) 65*32kB (UM) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 118900kB Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.552929] Node 1 Normal: 20948*4kB (UEM) 2474*8kB (UEM) 757*16kB (UMR) 175*32kB (UMR) 32*64kB (UMR) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 123344kB Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.552954] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.552960] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.552966] 3648045 total pagecache pages Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.552969] 184 pages in swap cache Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.552973] Swap cache stats: add 654, delete 470, find 20/29 Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.552977] Free swap = 3143168kB Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.552980] Total swap = 3145724kB Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.552984] 6283048 pages RAM Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.552986] 0 pages HighMem/MovableOnly Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.552990] 33129 pages reserved Nov 30 21:22:06 arh-ibstorage1-ib kernel: [77680.552993] 0 pages hwpoisoned Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.805583] kworker/2:3: page allocation failure: order:5, mode:0x206050 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.805596] CPU: 2 PID: 1599 Comm: kworker/2:3 Tainted: G E 3.18.0-rc6-ceph-00024-g72ca172 #1 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.805603] Hardware name: Supermicro X9DRH-7TF/7F/iTF/iF/X9DRH-7TF/7F/iTF/iF, BIOS 3.0b 04/28/2014 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.805623] Workqueue: ceph-msgr con_work [libceph] Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.805629] 0000000000000005 ffff8801132c7478 ffffffff81733b38 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.805639] kworker/6:3: page allocation failure: order:5, mode:0x206050 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.805645] ffff88046fc8e2d8 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.805650] 0000000000206050 ffff8801132c7508 ffffffff81167f8e 0000000000000000 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.805658] 0000000400000000 ffff88047fffcf48 0000001fffffffe0 0000000000000010 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.805666] Call Trace: Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.805675] [<ffffffff81733b38>] dump_stack+0x4e/0x68 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.805683] [<ffffffff81167f8e>] warn_alloc_failed+0xee/0x140 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.805690] [<ffffffff8116bc72>] __alloc_pages_nodemask+0x962/0xba0 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.805698] [<ffffffff810ca408>] ? lock_timer_base.isra.43+0x38/0x70 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.805705] [<ffffffff8116bef9>] alloc_kmem_pages_node+0x49/0x60 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.805711] [<ffffffff8173095d>] kmalloc_large_node+0x28/0x56 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.805720] [<ffffffff811bd09c>] __kmalloc_node_track_caller+0x1ec/0x230 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.805728] [<ffffffff816162d7>] ? __alloc_skb+0x87/0x2a0 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.805735] [<ffffffff8160fcd3>] ? lock_sock_nested+0x43/0xc0 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.805741] [<ffffffff81615e95>] __kmalloc_reserve.isra.52+0x95/0xa0 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.805747] [<ffffffff816162d7>] __alloc_skb+0x87/0x2a0 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.805755] [<ffffffff816748f1>] sk_stream_alloc_skb+0x41/0x130 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.805760] [<ffffffff81675364>] tcp_sendmsg+0x3b4/0xd10 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.805767] [<ffffffff816a14b5>] ? inet_sendmsg+0x5/0x110 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.805772] [<ffffffff816a157f>] inet_sendmsg+0xcf/0x110 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.805777] [<ffffffff816a14b5>] ? inet_sendmsg+0x5/0x110 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.805782] [<ffffffff8160d67f>] sock_sendmsg+0xaf/0xc0 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.805789] [<ffffffff8173cf59>] ? _raw_spin_unlock_bh+0x39/0x40 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.805795] [<ffffffff81675083>] ? tcp_sendmsg+0xd3/0xd10 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.805801] [<ffffffff8160d6d0>] kernel_sendmsg+0x40/0x60 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.805806] [<ffffffff8160fe4b>] sock_no_sendpage+0x9b/0xb0 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.805812] [<ffffffff81674a37>] tcp_sendpage+0x57/0x590 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.805817] [<ffffffff816a15c5>] ? inet_sendpage+0x5/0x130 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.805823] [<ffffffff816a16a2>] inet_sendpage+0xe2/0x130 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.805828] [<ffffffff816a15c5>] ? inet_sendpage+0x5/0x130 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.805835] [<ffffffff8160a72b>] kernel_sendpage+0x1b/0x30 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.805845] [<ffffffffa051e9db>] ceph_tcp_sendpage+0x5b/0x110 [libceph] Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.805855] [<ffffffffa051fc48>] con_work+0xaf8/0x2cc0 [libceph] Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.805863] [<ffffffff810801af>] ? finish_task_switch+0x3f/0x120 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.805870] [<ffffffff8107408a>] ? process_one_work+0x24a/0x530 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.805875] [<ffffffff81073f83>] ? process_one_work+0x143/0x530 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.805880] [<ffffffff81073f01>] ? process_one_work+0xc1/0x530 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.805886] [<ffffffff81073ffc>] process_one_work+0x1bc/0x530 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.805891] [<ffffffff81073f83>] ? process_one_work+0x143/0x530 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.805897] [<ffffffff8107475c>] worker_thread+0x11c/0x4b0 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.805903] [<ffffffff81074640>] ? rescuer_thread+0x290/0x290 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.805909] [<ffffffff8107a664>] kthread+0xe4/0x100 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.805915] [<ffffffff8107a580>] ? flush_kthread_worker+0xf0/0xf0 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.805922] [<ffffffff8173d66c>] ret_from_fork+0x7c/0xb0 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.805928] [<ffffffff8107a580>] ? flush_kthread_worker+0xf0/0xf0 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.805932] Mem-Info: Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.805935] Node 0 DMA per-cpu: Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.805940] CPU 0: hi: 0, btch: 1 usd: 0 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.805944] CPU 1: hi: 0, btch: 1 usd: 0 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.805948] CPU 2: hi: 0, btch: 1 usd: 0 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.805952] CPU 3: hi: 0, btch: 1 usd: 0 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.805956] CPU 4: hi: 0, btch: 1 usd: 0 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.805960] CPU 5: hi: 0, btch: 1 usd: 0 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.805965] CPU: 6 PID: 11659 Comm: kworker/6:3 Tainted: G E 3.18.0-rc6-ceph-00024-g72ca172 #1 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.805968] Hardware name: Supermicro X9DRH-7TF/7F/iTF/iF/X9DRH-7TF/7F/iTF/iF, BIOS 3.0b 04/28/2014 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.805988] Workqueue: ceph-msgr con_work [libceph] Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.805996] 0000000000000005 ffff88011114b478 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.805996] CPU 6: hi: 0, btch: 1 usd: 0 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.806000] CPU 7: hi: 0, btch: 1 usd: 0 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.806001] CPU 8: hi: 0, btch: 1 usd: 0 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.806002] CPU 9: hi: 0, btch: 1 usd: 0 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.806003] CPU 10: hi: 0, btch: 1 usd: 0 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.806004] CPU 11: hi: 0, btch: 1 usd: 0 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.806007] Node 0 DMA32 per-cpu: Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.806008] CPU 0: hi: 186, btch: 31 usd: 0 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.806011] CPU 1: hi: 186, btch: 31 usd: 0 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.806011] CPU 2: hi: 186, btch: 31 usd: 0 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.806012] CPU 3: hi: 186, btch: 31 usd: 0 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.806013] CPU 4: hi: 186, btch: 31 usd: 0 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.806014] CPU 5: hi: 186, btch: 31 usd: 0 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.806017] CPU 6: hi: 186, btch: 31 usd: 0 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.806018] CPU 7: hi: 186, btch: 31 usd: 0 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.806019] CPU 8: hi: 186, btch: 31 usd: 0 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.806022] CPU 9: hi: 186, btch: 31 usd: 0 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.806022] CPU 10: hi: 186, btch: 31 usd: 0 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.806025] CPU 11: hi: 186, btch: 31 usd: 0 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.806027] Node 0 Normal per-cpu: Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.806029] CPU 0: hi: 186, btch: 31 usd: 0 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.806030] CPU 1: hi: 186, btch: 31 usd: 0 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.806032] CPU 2: hi: 186, btch: 31 usd: 0 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.806033] CPU 3: hi: 186, btch: 31 usd: 0 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.806034] CPU 4: hi: 186, btch: 31 usd: 0 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.806035] CPU 5: hi: 186, btch: 31 usd: 0 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.806038] CPU 6: hi: 186, btch: 31 usd: 0 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.806039] CPU 7: hi: 186, btch: 31 usd: 0 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.806040] CPU 8: hi: 186, btch: 31 usd: 0 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.806042] CPU 9: hi: 186, btch: 31 usd: 0 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.806043] CPU 10: hi: 186, btch: 31 usd: 0 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.806044] CPU 11: hi: 186, btch: 31 usd: 0 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.806047] Node 1 Normal per-cpu: Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.806048] CPU 0: hi: 186, btch: 31 usd: 0 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.806050] CPU 1: hi: 186, btch: 31 usd: 0 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.806051] CPU 2: hi: 186, btch: 31 usd: 0 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.806052] CPU 3: hi: 186, btch: 31 usd: 0 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.806053] CPU 4: hi: 186, btch: 31 usd: 0 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.806054] CPU 5: hi: 186, btch: 31 usd: 0 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.806056] CPU 6: hi: 186, btch: 31 usd: 0 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.806057] CPU 7: hi: 186, btch: 31 usd: 0 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.806058] CPU 8: hi: 186, btch: 31 usd: 0 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.806061] CPU 9: hi: 186, btch: 31 usd: 0 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.806062] CPU 10: hi: 186, btch: 31 usd: 45 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.806063] CPU 11: hi: 186, btch: 31 usd: 0 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.806068] active_anon:1512662 inactive_anon:368455 isolated_anon:0 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.806068] active_file:1790766 inactive_file:1830033 isolated_file:0 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.806068] unevictable:0 dirty:274149 writeback:127645 unstable:0 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.806068] free:85880 slab_reclaimable:179356 slab_unreclaimable:67367 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.806068] mapped:9507 shmem:362 pagetables:8258 bounce:0 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.806068] free_cma:0 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.806075] Node 0 DMA free:2784kB min:56kB low:68kB high:84kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15956kB managed:15872kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.806081] lowmem_reserve[]: 0 1907 15985 15985 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.806087] Node 0 DMA32 free:79612kB min:7140kB low:8924kB high:10708kB active_anon:125976kB inactive_anon:181592kB active_file:551752kB inactive_file:661684kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:2047564kB managed:1955580kB mlocked:0kB dirty:40904kB writeback:14188kB mapped:1952kB shmem:76kB slab_reclaimable:254916kB slab_unreclaimable:51008kB kernel_stack:24240kB pagetables:2240kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.806091] lowmem_reserve[]: 0 0 14078 14078 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.806096] Node 0 Normal free:146332kB min:52716kB low:65892kB high:79072kB active_anon:2769344kB inactive_anon:657864kB active_file:4987548kB inactive_file:5035884kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:14680064kB managed:14416320kB mlocked:0kB dirty:648164kB writeback:237176kB mapped:16260kB shmem:1012kB slab_reclaimable:263104kB slab_unreclaimable:92612kB kernel_stack:23104kB pagetables:14116kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.806100] lowmem_reserve[]: 0 0 0 0 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.806106] Node 1 Normal free:114792kB min:30192kB low:37740kB high:45288kB active_anon:3155328kB inactive_anon:634364kB active_file:1623764kB inactive_file:1622564kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:8388608kB managed:8256092kB mlocked:0kB dirty:407528kB writeback:259216kB mapped:19816kB shmem:360kB slab_reclaimable:199404kB slab_unreclaimable:125832kB kernel_stack:10736kB pagetables:16676kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.806109] lowmem_reserve[]: 0 0 0 0 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.806123] Node 0 DMA: 0*4kB 0*8kB 2*16kB (U) 0*32kB 1*64kB (U) 9*128kB (U) 2*256kB (U) 2*512kB (U) 0*1024kB 0*2048kB 0*4096kB = 2784kB Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.806136] Node 0 DMA32: 19775*4kB (U) 23*8kB (U) 0*16kB 17*32kB (R) 1*64kB (R) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 79892kB Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.806150] Node 0 Normal: 27707*4kB (UEM) 1850*8kB (UEM) 256*16kB (UM) 316*32kB (UM) 96*64kB (UM) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 145980kB Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.806165] Node 1 Normal: 25873*4kB (UEM) 1260*8kB (UEM) 127*16kB (UMR) 46*32kB (UMR) 6*64kB (MR) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 117460kB Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.806168] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.806169] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.806170] 3621652 total pagecache pages Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.806172] 184 pages in swap cache Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.806173] Swap cache stats: add 654, delete 470, find 20/29 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.806175] Free swap = 3143168kB Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.806176] Total swap = 3145724kB Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.806179] 6283048 pages RAM Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.806179] 0 pages HighMem/MovableOnly Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.806180] 33129 pages reserved Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.806180] 0 pages hwpoisoned Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.806402] ffffffff81733b38 0000000000002b76 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.806408] 0000000000206050 ffff88011114b508 ffffffff81167f8e 0000000000000000 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.806417] 0000000400000000 ffff88067fff9f48 0000001fffffffe0 0000000000000010 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.806425] Call Trace: Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.806436] [<ffffffff81733b38>] dump_stack+0x4e/0x68 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.806445] [<ffffffff81167f8e>] warn_alloc_failed+0xee/0x140 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.806452] [<ffffffff8116bc72>] __alloc_pages_nodemask+0x962/0xba0 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.806460] [<ffffffff810ca408>] ? lock_timer_base.isra.43+0x38/0x70 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.806467] [<ffffffff8116bef9>] alloc_kmem_pages_node+0x49/0x60 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.806474] [<ffffffff8173095d>] kmalloc_large_node+0x28/0x56 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.806482] [<ffffffff811bd09c>] __kmalloc_node_track_caller+0x1ec/0x230 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.806492] [<ffffffff816162d7>] ? __alloc_skb+0x87/0x2a0 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.806499] [<ffffffff8160fcd3>] ? lock_sock_nested+0x43/0xc0 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.806505] [<ffffffff81615e95>] __kmalloc_reserve.isra.52+0x95/0xa0 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.806511] [<ffffffff816162d7>] __alloc_skb+0x87/0x2a0 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.806519] [<ffffffff816748f1>] sk_stream_alloc_skb+0x41/0x130 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.806525] [<ffffffff81675364>] tcp_sendmsg+0x3b4/0xd10 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.806532] [<ffffffff816a14b5>] ? inet_sendmsg+0x5/0x110 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.806537] [<ffffffff816a157f>] inet_sendmsg+0xcf/0x110 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.806542] [<ffffffff816a14b5>] ? inet_sendmsg+0x5/0x110 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.806551] [<ffffffff810a98fd>] ? trace_hardirqs_on+0xd/0x10 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.806557] [<ffffffff8160d67f>] sock_sendmsg+0xaf/0xc0 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.806562] [<ffffffff81611b23>] ? release_sock+0x193/0x1d0 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.806571] [<ffffffff813b78be>] ? trace_hardirqs_on_thunk+0x3a/0x3f Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.806577] [<ffffffff8160d6d0>] kernel_sendmsg+0x40/0x60 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.806583] [<ffffffff8160fe4b>] sock_no_sendpage+0x9b/0xb0 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.806589] [<ffffffff81674a37>] tcp_sendpage+0x57/0x590 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.806594] [<ffffffff816a15c5>] ? inet_sendpage+0x5/0x130 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.806602] [<ffffffff8173e460>] ? retint_restore_args+0xe/0xe Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.806608] [<ffffffff816a16a2>] inet_sendpage+0xe2/0x130 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.806613] [<ffffffff816a15c5>] ? inet_sendpage+0x5/0x130 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.806620] [<ffffffff8160a72b>] kernel_sendpage+0x1b/0x30 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.806634] [<ffffffffa051e9db>] ceph_tcp_sendpage+0x5b/0x110 [libceph] Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.806644] [<ffffffffa051fc48>] con_work+0xaf8/0x2cc0 [libceph] Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.806653] [<ffffffff810801af>] ? finish_task_switch+0x3f/0x120 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.806659] [<ffffffff81073f83>] ? process_one_work+0x143/0x530 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.806665] [<ffffffff81073f01>] ? process_one_work+0xc1/0x530 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.806853] [<ffffffff81073ffc>] process_one_work+0x1bc/0x530 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.806860] [<ffffffff81073f83>] ? process_one_work+0x143/0x530 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.806866] [<ffffffff8107475c>] worker_thread+0x11c/0x4b0 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.806872] [<ffffffff81074640>] ? rescuer_thread+0x290/0x290 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.806891] [<ffffffff8107a664>] kthread+0xe4/0x100 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.806922] [<ffffffff8107a580>] ? flush_kthread_worker+0xf0/0xf0 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.806930] [<ffffffff8173d66c>] ret_from_fork+0x7c/0xb0 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.806936] [<ffffffff8107a580>] ? flush_kthread_worker+0xf0/0xf0 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.806940] Mem-Info: Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.806944] Node 0 DMA per-cpu: Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.806949] CPU 0: hi: 0, btch: 1 usd: 0 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.806953] CPU 1: hi: 0, btch: 1 usd: 0 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.806957] CPU 2: hi: 0, btch: 1 usd: 0 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.806974] CPU 3: hi: 0, btch: 1 usd: 0 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.806978] CPU 4: hi: 0, btch: 1 usd: 0 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.806987] CPU 5: hi: 0, btch: 1 usd: 0 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.806999] CPU 6: hi: 0, btch: 1 usd: 0 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.807005] CPU 7: hi: 0, btch: 1 usd: 0 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.807012] CPU 8: hi: 0, btch: 1 usd: 0 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.807018] CPU 9: hi: 0, btch: 1 usd: 0 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.807025] CPU 10: hi: 0, btch: 1 usd: 0 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.807031] CPU 11: hi: 0, btch: 1 usd: 0 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.807047] Node 0 DMA32 per-cpu: Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.807054] CPU 0: hi: 186, btch: 31 usd: 0 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.807059] CPU 1: hi: 186, btch: 31 usd: 0 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.807063] CPU 2: hi: 186, btch: 31 usd: 0 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.807067] CPU 3: hi: 186, btch: 31 usd: 0 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.807071] CPU 4: hi: 186, btch: 31 usd: 0 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.807075] CPU 5: hi: 186, btch: 31 usd: 0 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.807079] CPU 6: hi: 186, btch: 31 usd: 0 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.807083] CPU 7: hi: 186, btch: 31 usd: 0 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.807087] CPU 8: hi: 186, btch: 31 usd: 0 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.807091] CPU 9: hi: 186, btch: 31 usd: 0 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.807096] CPU 10: hi: 186, btch: 31 usd: 0 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.807100] CPU 11: hi: 186, btch: 31 usd: 0 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.807103] Node 0 Normal per-cpu: Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.807108] CPU 0: hi: 186, btch: 31 usd: 0 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.807112] CPU 1: hi: 186, btch: 31 usd: 0 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.807116] CPU 2: hi: 186, btch: 31 usd: 0 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.807120] CPU 3: hi: 186, btch: 31 usd: 0 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.807124] CPU 4: hi: 186, btch: 31 usd: 0 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.807128] CPU 5: hi: 186, btch: 31 usd: 0 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.807132] CPU 6: hi: 186, btch: 31 usd: 0 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.807136] CPU 7: hi: 186, btch: 31 usd: 0 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.807140] CPU 8: hi: 186, btch: 31 usd: 0 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.807143] CPU 9: hi: 186, btch: 31 usd: 0 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.807148] CPU 10: hi: 186, btch: 31 usd: 0 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.807151] CPU 11: hi: 186, btch: 31 usd: 0 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.807155] Node 1 Normal per-cpu: Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.807160] CPU 0: hi: 186, btch: 31 usd: 0 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.807164] CPU 1: hi: 186, btch: 31 usd: 0 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.807168] CPU 2: hi: 186, btch: 31 usd: 0 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.807172] CPU 3: hi: 186, btch: 31 usd: 0 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.807175] CPU 4: hi: 186, btch: 31 usd: 0 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.807180] CPU 5: hi: 186, btch: 31 usd: 15 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.807184] CPU 6: hi: 186, btch: 31 usd: 0 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.807188] CPU 7: hi: 186, btch: 31 usd: 0 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.807192] CPU 8: hi: 186, btch: 31 usd: 0 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.807196] CPU 9: hi: 186, btch: 31 usd: 0 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.807200] CPU 10: hi: 186, btch: 31 usd: 0 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.807265] CPU 11: hi: 186, btch: 31 usd: 0 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.807272] active_anon:1512662 inactive_anon:368455 isolated_anon:0 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.807272] active_file:1790766 inactive_file:1830033 isolated_file:0 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.807272] unevictable:0 dirty:274149 writeback:127645 unstable:0 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.807272] free:86674 slab_reclaimable:179356 slab_unreclaimable:67367 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.807272] mapped:9507 shmem:362 pagetables:8258 bounce:0 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.807272] free_cma:0 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.807292] Node 0 DMA free:5984kB min:56kB low:68kB high:84kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15956kB managed:15872kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.807315] lowmem_reserve[]: 0 1907 15985 15985 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.807337] Node 0 DMA32 free:79612kB min:7140kB low:8924kB high:10708kB active_anon:125976kB inactive_anon:181592kB active_file:551752kB inactive_file:661684kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:2047564kB managed:1955580kB mlocked:0kB dirty:40904kB writeback:14188kB mapped:1952kB shmem:76kB slab_reclaimable:254916kB slab_unreclaimable:51008kB kernel_stack:24240kB pagetables:2240kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.807368] lowmem_reserve[]: 0 0 14078 14078 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.807378] Node 0 Normal free:146332kB min:52716kB low:65892kB high:79072kB active_anon:2769344kB inactive_anon:657864kB active_file:4987548kB inactive_file:5035884kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:14680064kB managed:14416320kB mlocked:0kB dirty:648164kB writeback:237176kB mapped:16260kB shmem:1012kB slab_reclaimable:263104kB slab_unreclaimable:92612kB kernel_stack:23104kB pagetables:14116kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.807404] lowmem_reserve[]: 0 0 0 0 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.807411] Node 1 Normal free:114768kB min:30192kB low:37740kB high:45288kB active_anon:3155328kB inactive_anon:634364kB active_file:1623764kB inactive_file:1622564kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:8388608kB managed:8256092kB mlocked:0kB dirty:407528kB writeback:259216kB mapped:19816kB shmem:360kB slab_reclaimable:199404kB slab_unreclaimable:125832kB kernel_stack:10736kB pagetables:16676kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.807437] lowmem_reserve[]: 0 0 0 0 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.807443] Node 0 DMA: 0*4kB 0*8kB 2*16kB (U) 0*32kB 1*64kB (U) 16*128kB (UR) 5*256kB (UR) 3*512kB (U) 1*1024kB (U) 0*2048kB 0*4096kB = 5984kB Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.807510] Node 0 DMA32: 19775*4kB (U) 23*8kB (U) 0*16kB 17*32kB (R) 1*64kB (R) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 79892kB Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.807542] Node 0 Normal: 27707*4kB (UEM) 1850*8kB (UEM) 256*16kB (UM) 316*32kB (UM) 96*64kB (UM) 1*128kB (R) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 146108kB Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.807589] Node 1 Normal: 25775*4kB (UEM) 1266*8kB (UEM) 127*16kB (UMR) 46*32kB (UMR) 6*64kB (MR) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 117116kB Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.807614] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.807622] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.807630] 3621652 total pagecache pages Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.807639] 184 pages in swap cache Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.807646] Swap cache stats: add 654, delete 470, find 20/29 Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.807654] Free swap = 3143168kB Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.807658] Total swap = 3145724kB Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.807662] 6283048 pages RAM Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.807665] 0 pages HighMem/MovableOnly Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.807668] 33129 pages reserved Nov 30 21:22:11 arh-ibstorage1-ib kernel: [77685.807671] 0 pages hwpoisoned Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.725731] kworker/7:3: page allocation failure: order:5, mode:0x206050 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.725748] CPU: 7 PID: 858 Comm: kworker/7:3 Tainted: G E 3.18.0-rc6-ceph-00024-g72ca172 #1 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.725755] Hardware name: Supermicro X9DRH-7TF/7F/iTF/iF/X9DRH-7TF/7F/iTF/iF, BIOS 3.0b 04/28/2014 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.725777] Workqueue: ceph-msgr con_work [libceph] Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.725784] 0000000000000005 ffff880114377478 ffffffff81733b38 ffff88067fc4e2d8 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.725792] 0000000000206050 ffff880114377508 ffffffff81167f8e 0000000000000000 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.725800] 0000000400000000 ffff88067fff9f48 0000001fffffffe0 0000000000000010 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.725809] Call Trace: Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.725820] [<ffffffff81733b38>] dump_stack+0x4e/0x68 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.725830] [<ffffffff81167f8e>] warn_alloc_failed+0xee/0x140 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.725837] [<ffffffff8116bc72>] __alloc_pages_nodemask+0x962/0xba0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.725846] [<ffffffff810ca408>] ? lock_timer_base.isra.43+0x38/0x70 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.725853] [<ffffffff8116bef9>] alloc_kmem_pages_node+0x49/0x60 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.725860] [<ffffffff8173095d>] kmalloc_large_node+0x28/0x56 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.725869] [<ffffffff811bd09c>] __kmalloc_node_track_caller+0x1ec/0x230 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.725880] [<ffffffff816162d7>] ? __alloc_skb+0x87/0x2a0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.725887] [<ffffffff8160fcd3>] ? lock_sock_nested+0x43/0xc0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.725893] [<ffffffff81615e95>] __kmalloc_reserve.isra.52+0x95/0xa0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.725900] [<ffffffff816162d7>] __alloc_skb+0x87/0x2a0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.725908] [<ffffffff816748f1>] sk_stream_alloc_skb+0x41/0x130 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.725914] [<ffffffff81675364>] tcp_sendmsg+0x3b4/0xd10 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.725921] [<ffffffff816a14b5>] ? inet_sendmsg+0x5/0x110 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.725926] [<ffffffff816a157f>] inet_sendmsg+0xcf/0x110 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.725931] [<ffffffff816a14b5>] ? inet_sendmsg+0x5/0x110 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.725937] [<ffffffff8160d67f>] sock_sendmsg+0xaf/0xc0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.725945] [<ffffffff8173cf59>] ? _raw_spin_unlock_bh+0x39/0x40 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.725950] [<ffffffff81675083>] ? tcp_sendmsg+0xd3/0xd10 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.725956] [<ffffffff8160d6d0>] kernel_sendmsg+0x40/0x60 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.725962] [<ffffffff8160fe4b>] sock_no_sendpage+0x9b/0xb0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.725968] [<ffffffff81674a37>] tcp_sendpage+0x57/0x590 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.725973] [<ffffffff816a15c5>] ? inet_sendpage+0x5/0x130 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.725983] [<ffffffffa051f1a5>] ? con_work+0x55/0x2cc0 [libceph] Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.725989] [<ffffffff816a16a2>] inet_sendpage+0xe2/0x130 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.725994] [<ffffffff816a15c5>] ? inet_sendpage+0x5/0x130 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.726001] [<ffffffff8160a72b>] kernel_sendpage+0x1b/0x30 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.726011] [<ffffffffa051e9db>] ceph_tcp_sendpage+0x5b/0x110 [libceph] Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.726021] [<ffffffffa051fc48>] con_work+0xaf8/0x2cc0 [libceph] Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.726030] [<ffffffff810801af>] ? finish_task_switch+0x3f/0x120 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.726038] [<ffffffff81073f83>] ? process_one_work+0x143/0x530 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.726044] [<ffffffff81073f01>] ? process_one_work+0xc1/0x530 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.726049] [<ffffffff81073ffc>] process_one_work+0x1bc/0x530 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.726054] [<ffffffff81073f83>] ? process_one_work+0x143/0x530 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.726060] [<ffffffff8107475c>] worker_thread+0x11c/0x4b0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.726066] [<ffffffff81074640>] ? rescuer_thread+0x290/0x290 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.726072] [<ffffffff8107a664>] kthread+0xe4/0x100 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.726079] [<ffffffff8107a580>] ? flush_kthread_worker+0xf0/0xf0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.726085] [<ffffffff8173d66c>] ret_from_fork+0x7c/0xb0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.726091] [<ffffffff8107a580>] ? flush_kthread_worker+0xf0/0xf0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.726096] Mem-Info: Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.726100] Node 0 DMA per-cpu: Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.726105] CPU 0: hi: 0, btch: 1 usd: 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.726109] CPU 1: hi: 0, btch: 1 usd: 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.726114] CPU 2: hi: 0, btch: 1 usd: 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.726118] CPU 3: hi: 0, btch: 1 usd: 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.726122] CPU 4: hi: 0, btch: 1 usd: 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.726126] CPU 5: hi: 0, btch: 1 usd: 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.726130] CPU 6: hi: 0, btch: 1 usd: 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.726134] CPU 7: hi: 0, btch: 1 usd: 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.726138] CPU 8: hi: 0, btch: 1 usd: 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.726142] CPU 9: hi: 0, btch: 1 usd: 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.726146] CPU 10: hi: 0, btch: 1 usd: 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.726150] CPU 11: hi: 0, btch: 1 usd: 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.726153] Node 0 DMA32 per-cpu: Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.726158] CPU 0: hi: 186, btch: 31 usd: 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.726162] CPU 1: hi: 186, btch: 31 usd: 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.726166] CPU 2: hi: 186, btch: 31 usd: 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.726170] CPU 3: hi: 186, btch: 31 usd: 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.726174] CPU 4: hi: 186, btch: 31 usd: 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.726178] CPU 5: hi: 186, btch: 31 usd: 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.726182] CPU 6: hi: 186, btch: 31 usd: 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.726186] CPU 7: hi: 186, btch: 31 usd: 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.726190] CPU 8: hi: 186, btch: 31 usd: 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.726194] CPU 9: hi: 186, btch: 31 usd: 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.726198] CPU 10: hi: 186, btch: 31 usd: 5 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.726202] CPU 11: hi: 186, btch: 31 usd: 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.726206] Node 0 Normal per-cpu: Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.726210] CPU 0: hi: 186, btch: 31 usd: 1 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.726214] CPU 1: hi: 186, btch: 31 usd: 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.726218] CPU 2: hi: 186, btch: 31 usd: 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.726222] CPU 3: hi: 186, btch: 31 usd: 46 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.726226] CPU 4: hi: 186, btch: 31 usd: 35 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.726230] CPU 5: hi: 186, btch: 31 usd: 30 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.726234] CPU 6: hi: 186, btch: 31 usd: 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.726238] CPU 7: hi: 186, btch: 31 usd: 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.726242] CPU 8: hi: 186, btch: 31 usd: 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.726246] CPU 9: hi: 186, btch: 31 usd: 30 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.726250] CPU 10: hi: 186, btch: 31 usd: 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.726254] CPU 11: hi: 186, btch: 31 usd: 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.726258] Node 1 Normal per-cpu: Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.726263] CPU 0: hi: 186, btch: 31 usd: 15 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.726267] CPU 1: hi: 186, btch: 31 usd: 15 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.726270] CPU 2: hi: 186, btch: 31 usd: 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.726275] CPU 3: hi: 186, btch: 31 usd: 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.726278] CPU 4: hi: 186, btch: 31 usd: 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.726282] CPU 5: hi: 186, btch: 31 usd: 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.726287] CPU 6: hi: 186, btch: 31 usd: 17 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.726291] CPU 7: hi: 186, btch: 31 usd: 31 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.726295] CPU 8: hi: 186, btch: 31 usd: 15 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.726299] CPU 9: hi: 186, btch: 31 usd: 53 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.726303] CPU 10: hi: 186, btch: 31 usd: 54 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.726307] CPU 11: hi: 186, btch: 31 usd: 31 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.726314] active_anon:1522659 inactive_anon:369255 isolated_anon:0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.726314] active_file:1779147 inactive_file:1832055 isolated_file:0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.726314] unevictable:0 dirty:211879 writeback:234513 unstable:0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.726314] free:84679 slab_reclaimable:183241 slab_unreclaimable:67551 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.726314] mapped:9518 shmem:362 pagetables:8258 bounce:0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.726314] free_cma:0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.726334] Node 0 DMA free:96kB min:56kB low:68kB high:84kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15956kB managed:15872kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.726357] lowmem_reserve[]: 0 1907 15985 15985 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.726365] Node 0 DMA32 free:77916kB min:7140kB low:8924kB high:10708kB active_anon:128116kB inactive_anon:181580kB active_file:551756kB inactive_file:659984kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:2047564kB managed:1955580kB mlocked:0kB dirty:48948kB writeback:43944kB mapped:1952kB shmem:76kB slab_reclaimable:255548kB slab_unreclaimable:50908kB kernel_stack:24240kB pagetables:2240kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.726391] lowmem_reserve[]: 0 0 14078 14078 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.726398] Node 0 Normal free:176924kB min:52716kB low:65892kB high:79072kB active_anon:2784344kB inactive_anon:657864kB active_file:4966504kB inactive_file:5009300kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:14680064kB managed:14416320kB mlocked:0kB dirty:527704kB writeback:507300kB mapped:15708kB shmem:1012kB slab_reclaimable:269160kB slab_unreclaimable:94340kB kernel_stack:23120kB pagetables:14116kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.726425] lowmem_reserve[]: 0 0 0 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.726432] Node 1 Normal free:83780kB min:30192kB low:37740kB high:45288kB active_anon:3178176kB inactive_anon:637576kB active_file:1598328kB inactive_file:1658936kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:8388608kB managed:8256092kB mlocked:0kB dirty:270864kB writeback:386808kB mapped:20412kB shmem:360kB slab_reclaimable:208256kB slab_unreclaimable:124940kB kernel_stack:10736kB pagetables:16676kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.726458] lowmem_reserve[]: 0 0 0 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.726464] Node 0 DMA: 0*4kB 0*8kB 2*16kB (U) 0*32kB 1*64kB (U) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 96kB Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.726485] Node 0 DMA32: 17983*4kB (UEM) 304*8kB (UM) 37*16kB (UM) 45*32kB (UMR) 18*64kB (UR) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 77548kB Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.726507] Node 0 Normal: 33734*4kB (UM) 3289*8kB (UM) 473*16kB (UM) 87*32kB (UM) 63*64kB (UM) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 175632kB Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.726530] Node 1 Normal: 20947*4kB (UEM) 114*8kB (UM) 1*16kB (R) 2*32kB (R) 2*64kB (R) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 84908kB Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.726553] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.726559] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.726565] 3612394 total pagecache pages Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.726569] 184 pages in swap cache Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.726573] Swap cache stats: add 654, delete 470, find 20/29 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.726577] Free swap = 3143168kB Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.726580] Total swap = 3145724kB Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.726584] 6283048 pages RAM Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.726587] 0 pages HighMem/MovableOnly Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.726590] 33129 pages reserved Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.726593] 0 pages hwpoisoned Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.738200] kworker/7:3: page allocation failure: order:5, mode:0x206050 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.738208] kworker/6:1: page allocation failure: order:5, mode:0x206050 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.738212] CPU: 6 PID: 13037 Comm: kworker/6:1 Tainted: G E 3.18.0-rc6-ceph-00024-g72ca172 #1 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.738214] Hardware name: Supermicro X9DRH-7TF/7F/iTF/iF/X9DRH-7TF/7F/iTF/iF, BIOS 3.0b 04/28/2014 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.738239] Workqueue: ceph-msgr con_work [libceph] Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.738244] 0000000000000005 ffff8801112ff478 ffffffff81733b38 000000000000313d Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.738246] 0000000000206050 ffff8801112ff508 ffffffff81167f8e 0000000000000000 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.738249] 0000000400000000 ffff88067fff9f48 0000001fffffffe0 0000000000000010 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.738250] Call Trace: Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.738261] [<ffffffff81733b38>] dump_stack+0x4e/0x68 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.738272] [<ffffffff81167f8e>] warn_alloc_failed+0xee/0x140 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.738275] [<ffffffff8116bc72>] __alloc_pages_nodemask+0x962/0xba0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.738280] [<ffffffff810ca408>] ? lock_timer_base.isra.43+0x38/0x70 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.738284] [<ffffffff8116bef9>] alloc_kmem_pages_node+0x49/0x60 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.738286] [<ffffffff8173095d>] kmalloc_large_node+0x28/0x56 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.738293] [<ffffffff811bd09c>] __kmalloc_node_track_caller+0x1ec/0x230 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.738300] [<ffffffff816162d7>] ? __alloc_skb+0x87/0x2a0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.738303] [<ffffffff81615e95>] __kmalloc_reserve.isra.52+0x95/0xa0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.738306] [<ffffffff816162d7>] __alloc_skb+0x87/0x2a0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.738311] [<ffffffff816748f1>] sk_stream_alloc_skb+0x41/0x130 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.738313] [<ffffffff81675364>] tcp_sendmsg+0x3b4/0xd10 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.738316] [<ffffffff816a14b5>] ? inet_sendmsg+0x5/0x110 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.738319] [<ffffffff816a157f>] inet_sendmsg+0xcf/0x110 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.738320] [<ffffffff816a14b5>] ? inet_sendmsg+0x5/0x110 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.738326] [<ffffffff810a98fd>] ? trace_hardirqs_on+0xd/0x10 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.738329] [<ffffffff8160d67f>] sock_sendmsg+0xaf/0xc0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.738333] [<ffffffff81611b23>] ? release_sock+0x193/0x1d0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.738335] [<ffffffff816732a0>] ? tcp_recvmsg+0x610/0xca0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.738337] [<ffffffff816a13a5>] ? inet_recvmsg+0x5/0x110 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.738340] [<ffffffff8160d6d0>] kernel_sendmsg+0x40/0x60 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.738342] [<ffffffff8160fe4b>] sock_no_sendpage+0x9b/0xb0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.738344] [<ffffffff81674a37>] tcp_sendpage+0x57/0x590 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.738346] [<ffffffff816a15c5>] ? inet_sendpage+0x5/0x130 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.738348] [<ffffffff816a16a2>] inet_sendpage+0xe2/0x130 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.738350] [<ffffffff816a15c5>] ? inet_sendpage+0x5/0x130 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.738354] [<ffffffff8160a72b>] kernel_sendpage+0x1b/0x30 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.738362] [<ffffffffa051e9db>] ceph_tcp_sendpage+0x5b/0x110 [libceph] Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.738368] [<ffffffffa051fc48>] con_work+0xaf8/0x2cc0 [libceph] Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.738373] [<ffffffff81096163>] ? pick_next_task_fair+0xf3/0x560 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.738378] [<ffffffff810801af>] ? finish_task_switch+0x3f/0x120 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.738381] [<ffffffff81073f83>] ? process_one_work+0x143/0x530 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.738383] [<ffffffff81073f01>] ? process_one_work+0xc1/0x530 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.738385] [<ffffffff81073ffc>] process_one_work+0x1bc/0x530 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.738387] [<ffffffff81073f83>] ? process_one_work+0x143/0x530 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.738389] [<ffffffff8107475c>] worker_thread+0x11c/0x4b0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.738391] [<ffffffff81074640>] ? rescuer_thread+0x290/0x290 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.738394] [<ffffffff8107a664>] kthread+0xe4/0x100 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.738397] [<ffffffff8107a580>] ? flush_kthread_worker+0xf0/0xf0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.738402] [<ffffffff8173d66c>] ret_from_fork+0x7c/0xb0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.738405] [<ffffffff8107a580>] ? flush_kthread_worker+0xf0/0xf0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.738407] Mem-Info: Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.738409] Node 0 DMA per-cpu: Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.738411] CPU 0: hi: 0, btch: 1 usd: 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.738412] CPU 1: hi: 0, btch: 1 usd: 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.738413] CPU 2: hi: 0, btch: 1 usd: 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.738414] CPU 3: hi: 0, btch: 1 usd: 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.738415] CPU 4: hi: 0, btch: 1 usd: 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.738416] CPU 5: hi: 0, btch: 1 usd: 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.738418] CPU 6: hi: 0, btch: 1 usd: 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.738419] CPU 7: hi: 0, btch: 1 usd: 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.738420] CPU 8: hi: 0, btch: 1 usd: 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.738421] CPU 9: hi: 0, btch: 1 usd: 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.738422] CPU 10: hi: 0, btch: 1 usd: 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.738423] CPU 11: hi: 0, btch: 1 usd: 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.738425] Node 0 DMA32 per-cpu: Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.738426] CPU 0: hi: 186, btch: 31 usd: 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.738427] CPU 1: hi: 186, btch: 31 usd: 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.738428] CPU 2: hi: 186, btch: 31 usd: 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.738430] CPU 3: hi: 186, btch: 31 usd: 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.738431] CPU 4: hi: 186, btch: 31 usd: 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.738432] CPU 5: hi: 186, btch: 31 usd: 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.738433] CPU 6: hi: 186, btch: 31 usd: 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.738434] CPU 7: hi: 186, btch: 31 usd: 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.738435] CPU 8: hi: 186, btch: 31 usd: 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.738436] CPU 9: hi: 186, btch: 31 usd: 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.738437] CPU 10: hi: 186, btch: 31 usd: 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.738438] CPU 11: hi: 186, btch: 31 usd: 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.738439] Node 0 Normal per-cpu: Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.738441] CPU 0: hi: 186, btch: 31 usd: 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.738442] CPU 1: hi: 186, btch: 31 usd: 32 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.738443] CPU 2: hi: 186, btch: 31 usd: 13 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.738444] CPU 3: hi: 186, btch: 31 usd: 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.738445] CPU 4: hi: 186, btch: 31 usd: 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.738446] CPU 5: hi: 186, btch: 31 usd: 30 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.738448] CPU 6: hi: 186, btch: 31 usd: 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.738449] CPU 7: hi: 186, btch: 31 usd: 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.738450] CPU 8: hi: 186, btch: 31 usd: 25 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.738451] CPU 9: hi: 186, btch: 31 usd: 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.738452] CPU 10: hi: 186, btch: 31 usd: 28 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.738453] CPU 11: hi: 186, btch: 31 usd: 27 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.738455] Node 1 Normal per-cpu: Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.738456] CPU 0: hi: 186, btch: 31 usd: 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.738457] CPU 1: hi: 186, btch: 31 usd: 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.738458] CPU 2: hi: 186, btch: 31 usd: 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.738459] CPU 3: hi: 186, btch: 31 usd: 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.738460] CPU 4: hi: 186, btch: 31 usd: 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.738462] CPU 5: hi: 186, btch: 31 usd: 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.738463] CPU 6: hi: 186, btch: 31 usd: 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.738464] CPU 7: hi: 186, btch: 31 usd: 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.738465] CPU 8: hi: 186, btch: 31 usd: 30 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.738466] CPU 9: hi: 186, btch: 31 usd: 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.738467] CPU 10: hi: 186, btch: 31 usd: 30 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.738468] CPU 11: hi: 186, btch: 31 usd: 30 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.738472] active_anon:1522659 inactive_anon:369255 isolated_anon:0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.738472] active_file:1779147 inactive_file:1838236 isolated_file:0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.738472] unevictable:0 dirty:207163 writeback:242745 unstable:0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.738472] free:78107 slab_reclaimable:183241 slab_unreclaimable:67591 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.738472] mapped:9518 shmem:362 pagetables:8258 bounce:0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.738472] free_cma:0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.738478] Node 0 DMA free:96kB min:56kB low:68kB high:84kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15956kB managed:15872kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.738481] lowmem_reserve[]: 0 1907 15985 15985 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.738487] Node 0 DMA32 free:76148kB min:7140kB low:8924kB high:10708kB active_anon:128116kB inactive_anon:181580kB active_file:551756kB inactive_file:661048kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:2047564kB managed:1955580kB mlocked:0kB dirty:47380kB writeback:45288kB mapped:1952kB shmem:76kB slab_reclaimable:255548kB slab_unreclaimable:51068kB kernel_stack:24240kB pagetables:2240kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.738490] lowmem_reserve[]: 0 0 14078 14078 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.738496] Node 0 Normal free:156324kB min:52716kB low:65892kB high:79072kB active_anon:2784344kB inactive_anon:657864kB active_file:4966504kB inactive_file:5029460kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:14680064kB managed:14416320kB mlocked:0kB dirty:511912kB writeback:535876kB mapped:15708kB shmem:1012kB slab_reclaimable:269160kB slab_unreclaimable:94340kB kernel_stack:23120kB pagetables:14116kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.738498] lowmem_reserve[]: 0 0 0 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.738504] Node 1 Normal free:79860kB min:30192kB low:37740kB high:45288kB active_anon:3178176kB inactive_anon:637576kB active_file:1598328kB inactive_file:1662436kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:8388608kB managed:8256092kB mlocked:0kB dirty:269360kB writeback:389816kB mapped:20412kB shmem:360kB slab_reclaimable:208256kB slab_unreclaimable:124940kB kernel_stack:10736kB pagetables:16676kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.738506] lowmem_reserve[]: 0 0 0 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.738517] Node 0 DMA: 0*4kB 0*8kB 2*16kB (U) 0*32kB 1*64kB (U) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 96kB Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.738528] Node 0 DMA32: 17766*4kB (UEM) 263*8kB (UEM) 36*16kB (UEM) 44*32kB (UEMR) 15*64kB (UER) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 76112kB Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.738539] Node 0 Normal: 28518*4kB (UM) 3267*8kB (UEM) 474*16kB (UEM) 92*32kB (UM) 61*64kB (UM) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 154640kB Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.738549] Node 1 Normal: 19901*4kB (UEM) 110*8kB (UEM) 1*16kB (R) 2*32kB (R) 2*64kB (R) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 80692kB Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.738552] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.738553] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.738553] 3618746 total pagecache pages Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.738555] 184 pages in swap cache Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.738556] Swap cache stats: add 654, delete 470, find 20/29 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.738558] Free swap = 3143168kB Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.738558] Total swap = 3145724kB Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.738560] 6283048 pages RAM Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.738561] 0 pages HighMem/MovableOnly Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.738561] 33129 pages reserved Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.738562] 0 pages hwpoisoned Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.738809] CPU: 7 PID: 858 Comm: kworker/7:3 Tainted: G E 3.18.0-rc6-ceph-00024-g72ca172 #1 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.738815] Hardware name: Supermicro X9DRH-7TF/7F/iTF/iF/X9DRH-7TF/7F/iTF/iF, BIOS 3.0b 04/28/2014 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.738829] Workqueue: ceph-msgr con_work [libceph] Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.738834] 0000000000000005 ffff880114377478 ffffffff81733b38 ffff88067fc4e2d8 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.738842] 0000000000206050 ffff880114377508 ffffffff81167f8e 0000000000000000 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.738851] 0000000400000000 ffff88067fff9f48 0000001fffffffe0 0000000000000010 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.738859] Call Trace: Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.738866] [<ffffffff81733b38>] dump_stack+0x4e/0x68 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.738873] [<ffffffff81167f8e>] warn_alloc_failed+0xee/0x140 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.738880] [<ffffffff8116bc72>] __alloc_pages_nodemask+0x962/0xba0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.738887] [<ffffffff8173d03b>] ? _raw_spin_unlock+0x2b/0x40 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.738893] [<ffffffff810ca408>] ? lock_timer_base.isra.43+0x38/0x70 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.738900] [<ffffffff8116bef9>] alloc_kmem_pages_node+0x49/0x60 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.738907] [<ffffffff8173095d>] kmalloc_large_node+0x28/0x56 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.738914] [<ffffffff811bd09c>] __kmalloc_node_track_caller+0x1ec/0x230 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.738921] [<ffffffff816162d7>] ? __alloc_skb+0x87/0x2a0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.738927] [<ffffffff8160fcd3>] ? lock_sock_nested+0x43/0xc0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.738934] [<ffffffff81615e95>] __kmalloc_reserve.isra.52+0x95/0xa0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.738940] [<ffffffff816162d7>] __alloc_skb+0x87/0x2a0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.738946] [<ffffffff816748f1>] sk_stream_alloc_skb+0x41/0x130 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.738952] [<ffffffff81675364>] tcp_sendmsg+0x3b4/0xd10 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.738958] [<ffffffff816a14b5>] ? inet_sendmsg+0x5/0x110 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.738963] [<ffffffff816a157f>] inet_sendmsg+0xcf/0x110 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.738968] [<ffffffff816a14b5>] ? inet_sendmsg+0x5/0x110 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.738974] [<ffffffff8160d67f>] sock_sendmsg+0xaf/0xc0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.738980] [<ffffffff8173cf59>] ? _raw_spin_unlock_bh+0x39/0x40 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.738985] [<ffffffff81675083>] ? tcp_sendmsg+0xd3/0xd10 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.738991] [<ffffffff8160d6d0>] kernel_sendmsg+0x40/0x60 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.738996] [<ffffffff8160fe4b>] sock_no_sendpage+0x9b/0xb0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.739002] [<ffffffff81674a37>] tcp_sendpage+0x57/0x590 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.739007] [<ffffffff816a15c5>] ? inet_sendpage+0x5/0x130 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.739017] [<ffffffffa051f1a5>] ? con_work+0x55/0x2cc0 [libceph] Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.739023] [<ffffffff816a16a2>] inet_sendpage+0xe2/0x130 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.739028] [<ffffffff816a15c5>] ? inet_sendpage+0x5/0x130 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.739034] [<ffffffff8160a72b>] kernel_sendpage+0x1b/0x30 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.739044] [<ffffffffa051e9db>] ceph_tcp_sendpage+0x5b/0x110 [libceph] Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.739054] [<ffffffffa051fc48>] con_work+0xaf8/0x2cc0 [libceph] Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.739060] [<ffffffff810801af>] ? finish_task_switch+0x3f/0x120 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.739067] [<ffffffff81073f83>] ? process_one_work+0x143/0x530 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.739072] [<ffffffff81073f01>] ? process_one_work+0xc1/0x530 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.739077] [<ffffffff81073ffc>] process_one_work+0x1bc/0x530 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.739083] [<ffffffff81073f83>] ? process_one_work+0x143/0x530 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.739089] [<ffffffff8107475c>] worker_thread+0x11c/0x4b0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.739094] [<ffffffff81074640>] ? rescuer_thread+0x290/0x290 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.739100] [<ffffffff8107a664>] kthread+0xe4/0x100 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.739106] [<ffffffff8107a580>] ? flush_kthread_worker+0xf0/0xf0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.739113] [<ffffffff8173d66c>] ret_from_fork+0x7c/0xb0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.739119] [<ffffffff8107a580>] ? flush_kthread_worker+0xf0/0xf0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.739123] Mem-Info: Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.739126] Node 0 DMA per-cpu: Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.739130] CPU 0: hi: 0, btch: 1 usd: 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.739134] CPU 1: hi: 0, btch: 1 usd: 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.739138] CPU 2: hi: 0, btch: 1 usd: 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.739142] CPU 3: hi: 0, btch: 1 usd: 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.739146] CPU 4: hi: 0, btch: 1 usd: 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.739150] CPU 5: hi: 0, btch: 1 usd: 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.739154] CPU 6: hi: 0, btch: 1 usd: 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.739158] CPU 7: hi: 0, btch: 1 usd: 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.739162] CPU 8: hi: 0, btch: 1 usd: 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.739166] CPU 9: hi: 0, btch: 1 usd: 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.739170] CPU 10: hi: 0, btch: 1 usd: 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.739174] CPU 11: hi: 0, btch: 1 usd: 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.739178] Node 0 DMA32 per-cpu: Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.739182] CPU 0: hi: 186, btch: 31 usd: 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.739186] CPU 1: hi: 186, btch: 31 usd: 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.739190] CPU 2: hi: 186, btch: 31 usd: 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.739194] CPU 3: hi: 186, btch: 31 usd: 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.739198] CPU 4: hi: 186, btch: 31 usd: 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.739202] CPU 5: hi: 186, btch: 31 usd: 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.739206] CPU 6: hi: 186, btch: 31 usd: 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.739210] CPU 7: hi: 186, btch: 31 usd: 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.739214] CPU 8: hi: 186, btch: 31 usd: 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.739218] CPU 9: hi: 186, btch: 31 usd: 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.739222] CPU 10: hi: 186, btch: 31 usd: 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.739226] CPU 11: hi: 186, btch: 31 usd: 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.739230] Node 0 Normal per-cpu: Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.739234] CPU 0: hi: 186, btch: 31 usd: 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.739238] CPU 1: hi: 186, btch: 31 usd: 53 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.739242] CPU 2: hi: 186, btch: 31 usd: 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.739246] CPU 3: hi: 186, btch: 31 usd: 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.739250] CPU 4: hi: 186, btch: 31 usd: 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.739254] CPU 5: hi: 186, btch: 31 usd: 30 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.739258] CPU 6: hi: 186, btch: 31 usd: 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.739262] CPU 7: hi: 186, btch: 31 usd: 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.739266] CPU 8: hi: 186, btch: 31 usd: 27 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.739270] CPU 9: hi: 186, btch: 31 usd: 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.739274] CPU 10: hi: 186, btch: 31 usd: 29 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.739278] CPU 11: hi: 186, btch: 31 usd: 30 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.739282] Node 1 Normal per-cpu: Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.739287] CPU 0: hi: 186, btch: 31 usd: 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.739291] CPU 1: hi: 186, btch: 31 usd: 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.739295] CPU 2: hi: 186, btch: 31 usd: 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.739299] CPU 3: hi: 186, btch: 31 usd: 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.739303] CPU 4: hi: 186, btch: 31 usd: 101 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.739307] CPU 5: hi: 186, btch: 31 usd: 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.739311] CPU 6: hi: 186, btch: 31 usd: 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.739315] CPU 7: hi: 186, btch: 31 usd: 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.739319] CPU 8: hi: 186, btch: 31 usd: 42 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.739323] CPU 9: hi: 186, btch: 31 usd: 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.739327] CPU 10: hi: 186, btch: 31 usd: 37 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.739331] CPU 11: hi: 186, btch: 31 usd: 39 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.739338] active_anon:1522659 inactive_anon:369255 isolated_anon:0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.739338] active_file:1779147 inactive_file:1839118 isolated_file:0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.739338] unevictable:0 dirty:207915 writeback:242745 unstable:0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.739338] free:78545 slab_reclaimable:183241 slab_unreclaimable:67591 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.739338] mapped:9518 shmem:362 pagetables:8258 bounce:0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.739338] free_cma:0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.739356] Node 0 DMA free:2784kB min:56kB low:68kB high:84kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15956kB managed:15872kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.739380] lowmem_reserve[]: 0 1907 15985 15985 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.739387] Node 0 DMA32 free:76404kB min:7140kB low:8924kB high:10708kB active_anon:128116kB inactive_anon:181580kB active_file:551756kB inactive_file:661048kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:2047564kB managed:1955580kB mlocked:0kB dirty:47380kB writeback:45288kB mapped:1952kB shmem:76kB slab_reclaimable:255548kB slab_unreclaimable:51068kB kernel_stack:24240kB pagetables:2240kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.739413] lowmem_reserve[]: 0 0 14078 14078 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.739420] Node 0 Normal free:154064kB min:52716kB low:65892kB high:79072kB active_anon:2784344kB inactive_anon:657864kB active_file:4966504kB inactive_file:5032484kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:14680064kB managed:14416320kB mlocked:0kB dirty:514920kB writeback:535876kB mapped:15708kB shmem:1012kB slab_reclaimable:269160kB slab_unreclaimable:94340kB kernel_stack:23120kB pagetables:14116kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.739446] lowmem_reserve[]: 0 0 0 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.739453] Node 1 Normal free:80928kB min:30192kB low:37740kB high:45288kB active_anon:3178176kB inactive_anon:637576kB active_file:1598328kB inactive_file:1663444kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:8388608kB managed:8256092kB mlocked:0kB dirty:270112kB writeback:389816kB mapped:20412kB shmem:360kB slab_reclaimable:208256kB slab_unreclaimable:124940kB kernel_stack:10736kB pagetables:16676kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.739479] lowmem_reserve[]: 0 0 0 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.739486] Node 0 DMA: 0*4kB 0*8kB 2*16kB (U) 0*32kB 1*64kB (U) 17*128kB (U) 0*256kB 1*512kB (U) 0*1024kB 0*2048kB 0*4096kB = 2784kB Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.739508] Node 0 DMA32: 17766*4kB (UEM) 263*8kB (UEM) 36*16kB (UEM) 44*32kB (UEMR) 15*64kB (UER) 2*128kB (UR) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 76368kB Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.739533] Node 0 Normal: 27887*4kB (UM) 3263*8kB (UEM) 476*16kB (UEM) 93*32kB (UEM) 62*64kB (UEM) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 152212kB Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.739557] Node 1 Normal: 20009*4kB (UEM) 107*8kB (UM) 1*16kB (R) 2*32kB (R) 2*64kB (R) 8*128kB (R) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 82124kB Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.739581] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.739587] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.739592] 3619498 total pagecache pages Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.739596] 184 pages in swap cache Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.739600] Swap cache stats: add 654, delete 470, find 20/29 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.739604] Free swap = 3143168kB Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.739607] Total swap = 3145724kB Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.739611] 6283048 pages RAM Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.739614] 0 pages HighMem/MovableOnly Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.739617] 33129 pages reserved Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.739620] 0 pages hwpoisoned Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.756226] kworker/7:3: page allocation failure: order:5, mode:0x206050 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.756241] CPU: 7 PID: 858 Comm: kworker/7:3 Tainted: G E 3.18.0-rc6-ceph-00024-g72ca172 #1 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.756248] Hardware name: Supermicro X9DRH-7TF/7F/iTF/iF/X9DRH-7TF/7F/iTF/iF, BIOS 3.0b 04/28/2014 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.756268] Workqueue: ceph-msgr con_work [libceph] Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.756275] 0000000000000005 ffff880114377478 ffffffff81733b38 ffff88067fc4e2d8 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.756283] 0000000000206050 ffff880114377508 ffffffff81167f8e 0000000000000000 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.756291] 0000000400000000 ffff88067fff9f48 0000001fffffffe0 0000000000000010 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.756299] Call Trace: Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.756310] [<ffffffff81733b38>] dump_stack+0x4e/0x68 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.756319] [<ffffffff81167f8e>] warn_alloc_failed+0xee/0x140 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.756326] [<ffffffff8116bc72>] __alloc_pages_nodemask+0x962/0xba0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.756335] [<ffffffff810ca408>] ? lock_timer_base.isra.43+0x38/0x70 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.756342] [<ffffffff8116bef9>] alloc_kmem_pages_node+0x49/0x60 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.756348] [<ffffffff8173095d>] kmalloc_large_node+0x28/0x56 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.756358] [<ffffffff811bd09c>] __kmalloc_node_track_caller+0x1ec/0x230 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.756369] [<ffffffff816162d7>] ? __alloc_skb+0x87/0x2a0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.756375] [<ffffffff81615e95>] __kmalloc_reserve.isra.52+0x95/0xa0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.756381] [<ffffffff816162d7>] __alloc_skb+0x87/0x2a0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.756389] [<ffffffff816748f1>] sk_stream_alloc_skb+0x41/0x130 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.756395] [<ffffffff81675364>] tcp_sendmsg+0x3b4/0xd10 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.756402] [<ffffffff816a14b5>] ? inet_sendmsg+0x5/0x110 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.756408] [<ffffffff816a157f>] inet_sendmsg+0xcf/0x110 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.756413] [<ffffffff816a14b5>] ? inet_sendmsg+0x5/0x110 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.756419] [<ffffffff8160d67f>] sock_sendmsg+0xaf/0xc0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.756427] [<ffffffff8173cf59>] ? _raw_spin_unlock_bh+0x39/0x40 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.756433] [<ffffffff81675083>] ? tcp_sendmsg+0xd3/0xd10 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.756438] [<ffffffff8160d6d0>] kernel_sendmsg+0x40/0x60 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.756444] [<ffffffff8160fe4b>] sock_no_sendpage+0x9b/0xb0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.756450] [<ffffffff81674a37>] tcp_sendpage+0x57/0x590 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.756455] [<ffffffff816a15c5>] ? inet_sendpage+0x5/0x130 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.756465] [<ffffffffa051f1a5>] ? con_work+0x55/0x2cc0 [libceph] Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.756471] [<ffffffff816a16a2>] inet_sendpage+0xe2/0x130 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.756476] [<ffffffff816a15c5>] ? inet_sendpage+0x5/0x130 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.756483] [<ffffffff8160a72b>] kernel_sendpage+0x1b/0x30 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.756493] [<ffffffffa051e9db>] ceph_tcp_sendpage+0x5b/0x110 [libceph] Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.756503] [<ffffffffa051fc48>] con_work+0xaf8/0x2cc0 [libceph] Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.756512] [<ffffffff810801af>] ? finish_task_switch+0x3f/0x120 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.756520] [<ffffffff81073f83>] ? process_one_work+0x143/0x530 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.756525] [<ffffffff81073f01>] ? process_one_work+0xc1/0x530 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.756531] [<ffffffff81073ffc>] process_one_work+0x1bc/0x530 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.756536] [<ffffffff81073f83>] ? process_one_work+0x143/0x530 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.756542] [<ffffffff8107475c>] worker_thread+0x11c/0x4b0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.756547] [<ffffffff81074640>] ? rescuer_thread+0x290/0x290 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.756553] [<ffffffff8107a664>] kthread+0xe4/0x100 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.756560] [<ffffffff8107a580>] ? flush_kthread_worker+0xf0/0xf0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.756567] [<ffffffff8173d66c>] ret_from_fork+0x7c/0xb0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.756572] [<ffffffff8107a580>] ? flush_kthread_worker+0xf0/0xf0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.756577] Mem-Info: Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.756581] Node 0 DMA per-cpu: Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.756586] CPU 0: hi: 0, btch: 1 usd: 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.756590] CPU 1: hi: 0, btch: 1 usd: 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.756594] CPU 2: hi: 0, btch: 1 usd: 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.756598] CPU 3: hi: 0, btch: 1 usd: 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.756602] CPU 4: hi: 0, btch: 1 usd: 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.756606] CPU 5: hi: 0, btch: 1 usd: 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.756610] CPU 6: hi: 0, btch: 1 usd: 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.756614] CPU 7: hi: 0, btch: 1 usd: 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.756618] CPU 8: hi: 0, btch: 1 usd: 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.756622] CPU 9: hi: 0, btch: 1 usd: 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.756626] CPU 10: hi: 0, btch: 1 usd: 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.756630] CPU 11: hi: 0, btch: 1 usd: 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.756634] Node 0 DMA32 per-cpu: Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.756639] CPU 0: hi: 186, btch: 31 usd: 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.756643] CPU 1: hi: 186, btch: 31 usd: 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.756647] CPU 2: hi: 186, btch: 31 usd: 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.756651] CPU 3: hi: 186, btch: 31 usd: 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.756655] CPU 4: hi: 186, btch: 31 usd: 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.756659] CPU 5: hi: 186, btch: 31 usd: 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.756663] CPU 6: hi: 186, btch: 31 usd: 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.756667] CPU 7: hi: 186, btch: 31 usd: 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.756671] CPU 8: hi: 186, btch: 31 usd: 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.756676] CPU 9: hi: 186, btch: 31 usd: 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.756680] CPU 10: hi: 186, btch: 31 usd: 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.756684] CPU 11: hi: 186, btch: 31 usd: 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.756687] Node 0 Normal per-cpu: Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.756692] CPU 0: hi: 186, btch: 31 usd: 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.756696] CPU 1: hi: 186, btch: 31 usd: 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.756700] CPU 2: hi: 186, btch: 31 usd: 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.756704] CPU 3: hi: 186, btch: 31 usd: 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.756707] CPU 4: hi: 186, btch: 31 usd: 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.756712] CPU 5: hi: 186, btch: 31 usd: 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.756716] CPU 6: hi: 186, btch: 31 usd: 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.756719] CPU 7: hi: 186, btch: 31 usd: 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.756723] CPU 8: hi: 186, btch: 31 usd: 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.756727] CPU 9: hi: 186, btch: 31 usd: 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.756731] CPU 10: hi: 186, btch: 31 usd: 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.756735] CPU 11: hi: 186, btch: 31 usd: 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.756739] Node 1 Normal per-cpu: Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.756743] CPU 0: hi: 186, btch: 31 usd: 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.756747] CPU 1: hi: 186, btch: 31 usd: 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.756751] CPU 2: hi: 186, btch: 31 usd: 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.756755] CPU 3: hi: 186, btch: 31 usd: 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.756759] CPU 4: hi: 186, btch: 31 usd: 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.756763] CPU 5: hi: 186, btch: 31 usd: 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.756767] CPU 6: hi: 186, btch: 31 usd: 31 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.756771] CPU 7: hi: 186, btch: 31 usd: 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.756775] CPU 8: hi: 186, btch: 31 usd: 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.756779] CPU 9: hi: 186, btch: 31 usd: 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.756783] CPU 10: hi: 186, btch: 31 usd: 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.756787] CPU 11: hi: 186, btch: 31 usd: 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.756794] active_anon:1522659 inactive_anon:369255 isolated_anon:0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.756794] active_file:1779147 inactive_file:1840061 isolated_file:0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.756794] unevictable:0 dirty:200495 writeback:250917 unstable:0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.756794] free:76772 slab_reclaimable:183241 slab_unreclaimable:67591 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.756794] mapped:9518 shmem:362 pagetables:8258 bounce:0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.756794] free_cma:0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.756813] Node 0 DMA free:96kB min:56kB low:68kB high:84kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15956kB managed:15872kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.756837] lowmem_reserve[]: 0 1907 15985 15985 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.756844] Node 0 DMA32 free:76148kB min:7140kB low:8924kB high:10708kB active_anon:128116kB inactive_anon:181580kB active_file:551756kB inactive_file:661048kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:2047564kB managed:1955580kB mlocked:0kB dirty:44020kB writeback:48648kB mapped:1952kB shmem:76kB slab_reclaimable:255548kB slab_unreclaimable:51068kB kernel_stack:24240kB pagetables:2240kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.756870] lowmem_reserve[]: 0 0 14078 14078 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.756877] Node 0 Normal free:146836kB min:52716kB low:65892kB high:79072kB active_anon:2784344kB inactive_anon:657864kB active_file:4966504kB inactive_file:5039036kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:14680064kB managed:14416320kB mlocked:0kB dirty:504392kB writeback:549412kB mapped:15708kB shmem:1012kB slab_reclaimable:269160kB slab_unreclaimable:94340kB kernel_stack:23120kB pagetables:14116kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.756903] lowmem_reserve[]: 0 0 0 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.756910] Node 1 Normal free:84008kB min:30192kB low:37740kB high:45288kB active_anon:3178176kB inactive_anon:637576kB active_file:1598328kB inactive_file:1660160kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:8388608kB managed:8256092kB mlocked:0kB dirty:252816kB writeback:407112kB mapped:20412kB shmem:360kB slab_reclaimable:208256kB slab_unreclaimable:124940kB kernel_stack:10736kB pagetables:16676kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.756935] lowmem_reserve[]: 0 0 0 0 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.756942] Node 0 DMA: 0*4kB 0*8kB 2*16kB (U) 0*32kB 1*64kB (U) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 96kB Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.756962] Node 0 DMA32: 17766*4kB (UEM) 263*8kB (UEM) 36*16kB (UEM) 43*32kB (UEMR) 15*64kB (UER) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 76080kB Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.756985] Node 0 Normal: 26128*4kB (UM) 3183*8kB (UM) 478*16kB (UM) 105*32kB (UEM) 71*64kB (UM) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 145528kB Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.757008] Node 1 Normal: 21101*4kB (UEM) 139*8kB (UEM) 1*16kB (R) 2*32kB (R) 2*64kB (R) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 85724kB Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.757032] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.757038] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.757044] 3620438 total pagecache pages Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.757048] 184 pages in swap cache Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.757052] Swap cache stats: add 654, delete 470, find 20/29 Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.757056] Free swap = 3143168kB Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.757059] Total swap = 3145724kB Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.757062] 6283048 pages RAM Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.757065] 0 pages HighMem/MovableOnly Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.757069] 33129 pages reserved Nov 30 21:22:20 arh-ibstorage1-ib kernel: [77694.757071] 0 pages hwpoisoned
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
