Sorry about that. Here it is
> Please check starting with the following line: Nov 29 12:07:38
> arh-ibstorage1-ib kernel: [ 3831.906510]. This is when I've started the
> concurrent 8 dds.
Andrei
> Please check starting with the following line: Nov 29 12:07:38
> arh-ibstorage1-ib kernel: [ 3831.906510]. This is when I've started the
> concurrent 8 dds.
Andrei
From: "Ilya Dryomov" <ilya.dryomov@xxxxxxxxxxx>
To: "Andrei Mikhailovsky" <andrei@xxxxxxxxxx>
Cc: "ceph-users" <ceph-users@xxxxxxxxxxxxxx>
Sent: Saturday, 29 November, 2014 12:49:37 PM
Subject: Re: Giant + nfs over cephfs hang tasks
On Sat, Nov 29, 2014 at 3:22 PM, Andrei Mikhailovsky <andrei@xxxxxxxxxx> wrote:
> Ilya,
>
> I think i spoke too soon in my last message. I've not given it more load
> (running 8 concurrent dds with bs=4M) and about a minute or so after
> starting i've seen problems in dmesg output. I am attaching kern.log file
> for you reference.
>
> Please check starting with the following line: Nov 29 12:07:38
> arh-ibstorage1-ib kernel: [ 3831.906510]. This is when I've started the
> concurrent 8 dds.
>
> The command that caused this is:
>
> time dd if=/dev/zero of=4G00 bs=4M count=5K oflag=direct & time dd
> if=/dev/zero of=4G11 bs=4M count=5K oflag=direct &time dd if=/dev/zero
> of=4G22 bs=4M count=5K oflag=direct &time dd if=/dev/zero of=4G33 bs=4M
> count=5K oflag=direct & time dd if=/dev/zero of=4G44 bs=4M count=5K
> oflag=direct & time dd if=/dev/zero of=4G55 bs=4M count=5K oflag=direct
> &time dd if=/dev/zero of=4G66 bs=4M count=5K oflag=direct &time dd
> if=/dev/zero of=4G77 bs=4M count=5K oflag=direct &
>
> I've ran the same test about 10 times but with only 4 concurrent dds and
> that didn't cause the issue.
>
> Should I try the 3.18 kernel again to see if 8dds produce similar output?
Missing attachment.
Thanks,
Ilya
Nov 29 11:04:04 arh-ibstorage1-ib kernel: [ 17.344547] bond0: link status definitely up for interface eth1, 1000 Mbps full duplex
Nov 29 11:04:04 arh-ibstorage1-ib kernel: [ 17.768195] init: plymouth-splash main process (4464) terminated with status 1
Nov 29 11:06:00 arh-ibstorage1-ib kernel: [ 134.584720] NFSD: Unable to end grace period: -110
Nov 29 11:28:49 arh-ibstorage1-ib kernel: [ 1503.458621] libceph: client50784718 fsid 51e9f641-372e-44ec-92a4-b9fe55cbf9fe
Nov 29 11:28:49 arh-ibstorage1-ib kernel: [ 1503.460322] libceph: mon0 192.168.168.200:6789 session established
Nov 29 11:29:16 arh-ibstorage1-ib kernel: [ 1529.920621] nfsd: last server has exited, flushing export cache
Nov 29 11:29:17 arh-ibstorage1-ib kernel: [ 1531.215693] NFSD: the nfsdcld client tracking upcall will be removed in 3.10. Please transition to using nfsdcltrack.
Nov 29 11:29:17 arh-ibstorage1-ib kernel: [ 1531.215768] NFSD: starting 90-second grace period (net ffffffff81cea7c0)
Nov 29 11:31:17 arh-ibstorage1-ib kernel: [ 1651.634845] NFSD: Unable to end grace period: -110
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.858729]
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.858736] =========================================================
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.858741] [ INFO: possible irq lock inversion dependency detected ]
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.858747] 3.17.4-ceph-00638-g0f25ebb #1 Tainted: G E
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.858751] ---------------------------------------------------------
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.858756] kswapd0/86 just changed the state of lock:
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.858760] (&xfs_dir_ilock_class){++++-+}, at: [<ffffffffa06448aa>] xfs_ilock+0x10a/0x150 [xfs]
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.858798] but this lock took another, RECLAIM_FS-unsafe lock in the past:
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.858802] (&mm->mmap_sem){++++++}
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.858802]
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.858802] and interrupts could create inverse lock ordering between them.
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.858802]
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.858810]
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.858810] other info that might help us debug this:
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.858815] Possible interrupt unsafe locking scenario:
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.858815]
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.858820] CPU0 CPU1
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.858823] ---- ----
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.858826] lock(&mm->mmap_sem);
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.858831] local_irq_disable();
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.858835] lock(&xfs_dir_ilock_class);
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.858840] lock(&mm->mmap_sem);
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.858846] <Interrupt>
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.858848] lock(&xfs_dir_ilock_class);
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.858853]
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.858853] *** DEADLOCK ***
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.858853]
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.858859] 3 locks held by kswapd0/86:
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.858862] #0: (shrinker_rwsem){++++..}, at: [<ffffffff8117209f>] shrink_slab+0x3f/0x140
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.858875] #1: (&type->s_umount_key#27){++++.+}, at: [<ffffffff811d4454>] grab_super_passive+0x44/0x90
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.858887] #2: (&pag->pag_ici_reclaim_lock){+.+...}, at: [<ffffffffa063ca64>] xfs_reclaim_inodes_ag+0xb4/0x400 [xfs]
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.858913]
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.858913] the shortest dependencies between 2nd lock and 1st lock:
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.858927] -> (&mm->mmap_sem){++++++} ops: 112343007 {
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.858935] HARDIRQ-ON-W at:
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.858938] [<ffffffff810a4cc9>] __lock_acquire+0x629/0x1c90
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.858947] [<ffffffff810a6937>] lock_acquire+0x87/0x120
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.858954] [<ffffffff81731619>] down_write+0x49/0x80
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.858962] [<ffffffff811d84a3>] do_execve_common.isra.26+0x283/0x6d0
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.858970] [<ffffffff811d8908>] do_execve+0x18/0x20
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.858976] [<ffffffff8106e44a>] ____call_usermodehelper+0x10a/0x140
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.858984] [<ffffffff81733e2c>] ret_from_fork+0x7c/0xb0
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.858991] HARDIRQ-ON-R at:
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.858994] [<ffffffff810a49e3>] __lock_acquire+0x343/0x1c90
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.859001] [<ffffffff810a6937>] lock_acquire+0x87/0x120
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.859008] [<ffffffff8118a323>] might_fault+0x93/0xc0
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.859015] [<ffffffff813b0fa0>] __clear_user+0x20/0x70
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.859023] [<ffffffff813b101e>] clear_user+0x2e/0x40
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.859029] [<ffffffff8122559d>] padzero+0x2d/0x40
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.859036] [<ffffffff81226044>] load_elf_binary+0x884/0x1940
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.859043] [<ffffffff811d722c>] search_binary_handler+0x9c/0x1a0
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.859050] [<ffffffff811d8742>] do_execve_common.isra.26+0x522/0x6d0
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.859057] [<ffffffff811d8908>] do_execve+0x18/0x20
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.859063] [<ffffffff8106e44a>] ____call_usermodehelper+0x10a/0x140
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.859070] [<ffffffff81733e2c>] ret_from_fork+0x7c/0xb0
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.859077] SOFTIRQ-ON-W at:
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.859080] [<ffffffff810a4a14>] __lock_acquire+0x374/0x1c90
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.859087] [<ffffffff810a6937>] lock_acquire+0x87/0x120
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.859094] [<ffffffff81731619>] down_write+0x49/0x80
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.859100] [<ffffffff811d84a3>] do_execve_common.isra.26+0x283/0x6d0
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.859108] [<ffffffff811d8908>] do_execve+0x18/0x20
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.859114] [<ffffffff8106e44a>] ____call_usermodehelper+0x10a/0x140
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.859121] [<ffffffff81733e2c>] ret_from_fork+0x7c/0xb0
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.859128] SOFTIRQ-ON-R at:
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.859131] [<ffffffff810a4a14>] __lock_acquire+0x374/0x1c90
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.859138] [<ffffffff810a6937>] lock_acquire+0x87/0x120
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.859144] [<ffffffff8118a323>] might_fault+0x93/0xc0
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.859152] [<ffffffff813b0fa0>] __clear_user+0x20/0x70
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.859158] [<ffffffff813b101e>] clear_user+0x2e/0x40
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.859165] [<ffffffff8122559d>] padzero+0x2d/0x40
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.859170] [<ffffffff81226044>] load_elf_binary+0x884/0x1940
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.859177] [<ffffffff811d722c>] search_binary_handler+0x9c/0x1a0
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.859184] [<ffffffff811d8742>] do_execve_common.isra.26+0x522/0x6d0
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.859191] [<ffffffff811d8908>] do_execve+0x18/0x20
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.859197] [<ffffffff8106e44a>] ____call_usermodehelper+0x10a/0x140
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.859204] [<ffffffff81733e2c>] ret_from_fork+0x7c/0xb0
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.859210] RECLAIM_FS-ON-W at:
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.859214] [<ffffffff810a718a>] mark_held_locks+0x6a/0x90
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.859221] [<ffffffff810a78a7>] lockdep_trace_alloc+0x67/0xd0
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.859228] [<ffffffff811687a4>] __alloc_pages_nodemask+0x84/0xb70
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.859238] [<ffffffff811aadeb>] alloc_pages_current+0x10b/0x1f0
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.859246] [<ffffffff8118d229>] __pmd_alloc+0x39/0x160
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.859253] [<ffffffff8119716e>] move_page_tables+0x62e/0x6b0
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.859260] [<ffffffff811d6a0b>] shift_arg_pages+0xcb/0x1a0
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.859267] [<ffffffff811d6cbb>] setup_arg_pages+0x1db/0x200
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.859274] [<ffffffff81225c03>] load_elf_binary+0x443/0x1940
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.859280] [<ffffffff811d722c>] search_binary_handler+0x9c/0x1a0
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.859288] [<ffffffff811d8742>] do_execve_common.isra.26+0x522/0x6d0
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.859295] [<ffffffff811d8908>] do_execve+0x18/0x20
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.859301] [<ffffffff8106e44a>] ____call_usermodehelper+0x10a/0x140
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.859308] [<ffffffff81733e2c>] ret_from_fork+0x7c/0xb0
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.859315] RECLAIM_FS-ON-R at:
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.859318] [<ffffffff810a718a>] mark_held_locks+0x6a/0x90
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.859325] [<ffffffff810a78a7>] lockdep_trace_alloc+0x67/0xd0
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.859332] [<ffffffff811687a4>] __alloc_pages_nodemask+0x84/0xb70
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.859339] [<ffffffff811aadeb>] alloc_pages_current+0x10b/0x1f0
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.859346] [<ffffffff8116377e>] __get_free_pages+0xe/0x40
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.859354] [<ffffffff811637c6>] get_zeroed_page+0x16/0x20
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.859361] [<ffffffff8118ca89>] __pud_alloc+0x29/0xb0
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.859368] [<ffffffff8118e373>] handle_mm_fault+0xec3/0x10b0
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.859375] [<ffffffff81046eb2>] __do_page_fault+0x1c2/0x580
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.859382] [<ffffffff810473ac>] do_page_fault+0xc/0x10
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.859388] [<ffffffff81735d02>] page_fault+0x22/0x30
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.859395] [<ffffffff813b101e>] clear_user+0x2e/0x40
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.859402] [<ffffffff8122559d>] padzero+0x2d/0x40
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.859408] [<ffffffff81226044>] load_elf_binary+0x884/0x1940
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.859414] [<ffffffff811d722c>] search_binary_handler+0x9c/0x1a0
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.859421] [<ffffffff811d8742>] do_execve_common.isra.26+0x522/0x6d0
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.859428] [<ffffffff811d8908>] do_execve+0x18/0x20
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.859434] [<ffffffff8106e44a>] ____call_usermodehelper+0x10a/0x140
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.859442] [<ffffffff81733e2c>] ret_from_fork+0x7c/0xb0
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.859448] INITIAL USE at:
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.859451] [<ffffffff810a4ab1>] __lock_acquire+0x411/0x1c90
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.859458] [<ffffffff810a6937>] lock_acquire+0x87/0x120
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.859465] [<ffffffff81731619>] down_write+0x49/0x80
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.859471] [<ffffffff811d84a3>] do_execve_common.isra.26+0x283/0x6d0
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.859479] [<ffffffff811d8908>] do_execve+0x18/0x20
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.859485] [<ffffffff8106e44a>] ____call_usermodehelper+0x10a/0x140
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.859492] [<ffffffff81733e2c>] ret_from_fork+0x7c/0xb0
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.859498] }
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.859500] ... key at: [<ffffffff81ea43c8>] __key.49636+0x0/0x8
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.859508] ... acquired at:
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.859510] [<ffffffff810a6937>] lock_acquire+0x87/0x120
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.859516] [<ffffffff8118a323>] might_fault+0x93/0xc0
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.859522] [<ffffffff811e4834>] filldir+0xa4/0x120
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.859527] [<ffffffffa06360e8>] xfs_dir2_leaf_getdents+0x2d8/0x370 [xfs]
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.859547] [<ffffffffa0636326>] xfs_readdir+0x1a6/0x1b0 [xfs]
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.859565] [<ffffffffa0637d1b>] xfs_file_readdir+0x2b/0x30 [xfs]
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.859584] [<ffffffff811e496e>] iterate_dir+0xbe/0x150
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.859589] [<ffffffff811e4b1c>] SyS_getdents+0x8c/0x110
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.859594] [<ffffffff81733ed6>] system_call_fastpath+0x1a/0x1f
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.859600]
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.859602] -> (&xfs_dir_ilock_class){++++-+} ops: 302420 {
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.859609] HARDIRQ-ON-W at:
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.859613] [<ffffffff810a4cc9>] __lock_acquire+0x629/0x1c90
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.859620] [<ffffffff810a6937>] lock_acquire+0x87/0x120
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.859626] [<ffffffff810a0eaf>] down_write_nested+0x4f/0x80
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.859633] [<ffffffffa06448aa>] xfs_ilock+0x10a/0x150 [xfs]
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.859653] [<ffffffffa0647a1d>] xfs_inactive_ifree+0x8d/0x230 [xfs]
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.859674] [<ffffffffa0647c79>] xfs_inactive+0xb9/0x170 [xfs]
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.859693] [<ffffffffa064daa1>] xfs_fs_evict_inode+0x91/0x100 [xfs]
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.859714] [<ffffffff811ed3b8>] evict+0xb8/0x190
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.859721] [<ffffffff811edcf5>] iput+0x105/0x190
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.859728] [<ffffffff811e8258>] d_delete+0x168/0x1a0
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.859734] [<ffffffff811dee81>] vfs_rmdir+0x101/0x120
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.859740] [<ffffffff811df06b>] do_rmdir+0x1cb/0x1e0
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.859747] [<ffffffff811e2436>] SyS_rmdir+0x16/0x20
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.859752] [<ffffffff81733ed6>] system_call_fastpath+0x1a/0x1f
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.859759] HARDIRQ-ON-R at:
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.859763] [<ffffffff810a49e3>] __lock_acquire+0x343/0x1c90
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.859769] [<ffffffff810a6937>] lock_acquire+0x87/0x120
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.859776] [<ffffffff810a0e47>] down_read_nested+0x57/0x70
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.859782] [<ffffffffa06448c2>] xfs_ilock+0x122/0x150 [xfs]
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.859802] [<ffffffffa0644975>] xfs_ilock_data_map_shared+0x35/0x40 [xfs]
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.859821] [<ffffffffa0645adb>] xfs_lookup+0x4b/0x150 [xfs]
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.859840] [<ffffffffa064255b>] xfs_vn_lookup+0x5b/0xa0 [xfs]
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.859858] [<ffffffff811da8bd>] lookup_real+0x1d/0x60
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.859865] [<ffffffff811e0c88>] do_last+0x7f8/0xf50
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.859871] [<ffffffff811e14a4>] path_openat+0xc4/0x660
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.859878] [<ffffffff811e1f83>] do_filp_open+0x43/0xa0
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.859884] [<ffffffff811cfca2>] do_sys_open+0x132/0x220
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.859890] [<ffffffff811cfdae>] SyS_open+0x1e/0x20
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.859896] [<ffffffff81733ed6>] system_call_fastpath+0x1a/0x1f
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.859903] SOFTIRQ-ON-W at:
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.859907] [<ffffffff810a4a14>] __lock_acquire+0x374/0x1c90
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.859914] [<ffffffff810a6937>] lock_acquire+0x87/0x120
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.859920] [<ffffffff810a0eaf>] down_write_nested+0x4f/0x80
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.859926] [<ffffffffa06448aa>] xfs_ilock+0x10a/0x150 [xfs]
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.859944] [<ffffffffa0647a1d>] xfs_inactive_ifree+0x8d/0x230 [xfs]
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.859963] [<ffffffffa0647c79>] xfs_inactive+0xb9/0x170 [xfs]
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.859982] [<ffffffffa064daa1>] xfs_fs_evict_inode+0x91/0x100 [xfs]
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.860002] [<ffffffff811ed3b8>] evict+0xb8/0x190
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.860009] [<ffffffff811edcf5>] iput+0x105/0x190
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.860015] [<ffffffff811e8258>] d_delete+0x168/0x1a0
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.860021] [<ffffffff811dee81>] vfs_rmdir+0x101/0x120
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.860027] [<ffffffff811df06b>] do_rmdir+0x1cb/0x1e0
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.860034] [<ffffffff811e2436>] SyS_rmdir+0x16/0x20
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.860040] [<ffffffff81733ed6>] system_call_fastpath+0x1a/0x1f
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.860046] SOFTIRQ-ON-R at:
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.860050] [<ffffffff810a4a14>] __lock_acquire+0x374/0x1c90
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.860056] [<ffffffff810a6937>] lock_acquire+0x87/0x120
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.860062] [<ffffffff810a0e47>] down_read_nested+0x57/0x70
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.860069] [<ffffffffa06448c2>] xfs_ilock+0x122/0x150 [xfs]
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.860087] [<ffffffffa0644975>] xfs_ilock_data_map_shared+0x35/0x40 [xfs]
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.860106] [<ffffffffa0645adb>] xfs_lookup+0x4b/0x150 [xfs]
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.860125] [<ffffffffa064255b>] xfs_vn_lookup+0x5b/0xa0 [xfs]
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.860143] [<ffffffff811da8bd>] lookup_real+0x1d/0x60
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.860150] [<ffffffff811e0c88>] do_last+0x7f8/0xf50
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.860156] [<ffffffff811e14a4>] path_openat+0xc4/0x660
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.860163] [<ffffffff811e1f83>] do_filp_open+0x43/0xa0
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.860168] [<ffffffff811cfca2>] do_sys_open+0x132/0x220
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.860175] [<ffffffff811cfdae>] SyS_open+0x1e/0x20
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.860181] [<ffffffff81733ed6>] system_call_fastpath+0x1a/0x1f
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.860187] IN-RECLAIM_FS-W at:
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.860191] [<ffffffff810a4a70>] __lock_acquire+0x3d0/0x1c90
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.860198] [<ffffffff810a6937>] lock_acquire+0x87/0x120
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.860204] [<ffffffff810a0eaf>] down_write_nested+0x4f/0x80
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.860211] [<ffffffffa06448aa>] xfs_ilock+0x10a/0x150 [xfs]
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.860229] [<ffffffffa063c79c>] xfs_reclaim_inode+0x12c/0x340 [xfs]
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.860248] [<ffffffffa063cc37>] xfs_reclaim_inodes_ag+0x287/0x400 [xfs]
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.860268] [<ffffffffa063daa3>] xfs_reclaim_inodes_nr+0x33/0x40 [xfs]
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.860286] [<ffffffffa064d235>] xfs_fs_free_cached_objects+0x15/0x20 [xfs]
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.860308] [<ffffffff811d4618>] super_cache_scan+0x178/0x180
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.860314] [<ffffffff811703fe>] shrink_slab_node+0x14e/0x2f0
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.860321] [<ffffffff81172160>] shrink_slab+0x100/0x140
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.860328] [<ffffffff81174e16>] kswapd_shrink_zone+0x116/0x1a0
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.860334] [<ffffffff81175cdb>] kswapd+0x4ab/0x980
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.860340] [<ffffffff81078a34>] kthread+0xe4/0x100
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.860347] [<ffffffff81733e2c>] ret_from_fork+0x7c/0xb0
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.860353] RECLAIM_FS-ON-R at:
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.860357] [<ffffffff810a718a>] mark_held_locks+0x6a/0x90
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.860364] [<ffffffff810a78a7>] lockdep_trace_alloc+0x67/0xd0
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.860371] [<ffffffff811687a4>] __alloc_pages_nodemask+0x84/0xb70
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.860378] [<ffffffff811ad3d2>] alloc_pages_vma+0xf2/0x1c0
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.860386] [<ffffffff8118e1ce>] handle_mm_fault+0xd1e/0x10b0
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.860393] [<ffffffff81046eb2>] __do_page_fault+0x1c2/0x580
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.860399] [<ffffffff810473ac>] do_page_fault+0xc/0x10
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.860406] [<ffffffff81735d02>] page_fault+0x22/0x30
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.860412] [<ffffffffa06360e8>] xfs_dir2_leaf_getdents+0x2d8/0x370 [xfs]
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.860432] [<ffffffffa0636326>] xfs_readdir+0x1a6/0x1b0 [xfs]
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.860451] [<ffffffffa0637d1b>] xfs_file_readdir+0x2b/0x30 [xfs]
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.860469] [<ffffffff811e496e>] iterate_dir+0xbe/0x150
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.860476] [<ffffffff811e4b1c>] SyS_getdents+0x8c/0x110
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.860482] [<ffffffff81733ed6>] system_call_fastpath+0x1a/0x1f
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.860489] INITIAL USE at:
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.860492] [<ffffffff810a4ab1>] __lock_acquire+0x411/0x1c90
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.860499] [<ffffffff810a6937>] lock_acquire+0x87/0x120
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.860505] [<ffffffff810a0e47>] down_read_nested+0x57/0x70
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.860511] [<ffffffffa06448c2>] xfs_ilock+0x122/0x150 [xfs]
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.860529] [<ffffffffa0644975>] xfs_ilock_data_map_shared+0x35/0x40 [xfs]
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.860548] [<ffffffffa0645adb>] xfs_lookup+0x4b/0x150 [xfs]
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.860566] [<ffffffffa064255b>] xfs_vn_lookup+0x5b/0xa0 [xfs]
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.860584] [<ffffffff811da8bd>] lookup_real+0x1d/0x60
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.860591] [<ffffffff811e0c88>] do_last+0x7f8/0xf50
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.860597] [<ffffffff811e14a4>] path_openat+0xc4/0x660
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.860604] [<ffffffff811e1f83>] do_filp_open+0x43/0xa0
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.860609] [<ffffffff811cfca2>] do_sys_open+0x132/0x220
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.860615] [<ffffffff811cfdae>] SyS_open+0x1e/0x20
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.860621] [<ffffffff81733ed6>] system_call_fastpath+0x1a/0x1f
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.860628] }
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.860630] ... key at: [<ffffffffa069fc10>] xfs_dir_ilock_class+0x0/0xfffffffffffd13f0 [xfs]
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.860649] ... acquired at:
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.860652] [<ffffffff810a3b3b>] check_usage_forwards+0x10b/0x150
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.860658] [<ffffffff810a454b>] mark_lock+0x18b/0x2e0
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.860663] [<ffffffff810a4a70>] __lock_acquire+0x3d0/0x1c90
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.860669] [<ffffffff810a6937>] lock_acquire+0x87/0x120
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.860674] [<ffffffff810a0eaf>] down_write_nested+0x4f/0x80
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.860680] [<ffffffffa06448aa>] xfs_ilock+0x10a/0x150 [xfs]
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.860696] [<ffffffffa063c79c>] xfs_reclaim_inode+0x12c/0x340 [xfs]
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.860714] [<ffffffffa063cc37>] xfs_reclaim_inodes_ag+0x287/0x400 [xfs]
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.860731] [<ffffffffa063daa3>] xfs_reclaim_inodes_nr+0x33/0x40 [xfs]
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.860748] [<ffffffffa064d235>] xfs_fs_free_cached_objects+0x15/0x20 [xfs]
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.860769] [<ffffffff811d4618>] super_cache_scan+0x178/0x180
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.860774] [<ffffffff811703fe>] shrink_slab_node+0x14e/0x2f0
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.860779] [<ffffffff81172160>] shrink_slab+0x100/0x140
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.860785] [<ffffffff81174e16>] kswapd_shrink_zone+0x116/0x1a0
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.860790] [<ffffffff81175cdb>] kswapd+0x4ab/0x980
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.860795] [<ffffffff81078a34>] kthread+0xe4/0x100
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.860801] [<ffffffff81733e2c>] ret_from_fork+0x7c/0xb0
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.860806]
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.860808]
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.860808] stack backtrace:
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.860814] CPU: 1 PID: 86 Comm: kswapd0 Tainted: G E 3.17.4-ceph-00638-g0f25ebb #1
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.860820] Hardware name: Supermicro X9DRH-7TF/7F/iTF/iF/X9DRH-7TF/7F/iTF/iF, BIOS 3.0b 04/28/2014
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.860827] ffffffff8211e7e0 ffff880467017608 ffffffff8172a4cf 0000000000000007
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.860835] ffff880467017670 ffff880467017658 ffffffff810a3a28 ffffffff8211e7e0
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.860842] ffffffff81a79f14 ffff880467017674 0000000000000000 ffff8804677a2a80
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.860850] Call Trace:
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.860856] [<ffffffff8172a4cf>] dump_stack+0x4e/0x68
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.860862] [<ffffffff810a3a28>] print_irq_inversion_bug.part.41+0x1e8/0x1f0
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.860868] [<ffffffff810a3b3b>] check_usage_forwards+0x10b/0x150
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.860874] [<ffffffff810a454b>] mark_lock+0x18b/0x2e0
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.860880] [<ffffffff810a3a30>] ? print_irq_inversion_bug.part.41+0x1f0/0x1f0
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.860889] [<ffffffff811c4f45>] ? __mem_cgroup_threshold+0x5/0x1d0
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.860895] [<ffffffff810a4a70>] __lock_acquire+0x3d0/0x1c90
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.860901] [<ffffffff81166eb5>] ? free_pcppages_bulk+0x35/0x470
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.860907] [<ffffffff810a4ab1>] ? __lock_acquire+0x411/0x1c90
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.860924] [<ffffffffa06448aa>] ? xfs_ilock+0x10a/0x150 [xfs]
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.860930] [<ffffffff810a6937>] lock_acquire+0x87/0x120
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.860947] [<ffffffffa06448aa>] ? xfs_ilock+0x10a/0x150 [xfs]
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.860953] [<ffffffff810a0eaf>] down_write_nested+0x4f/0x80
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.860969] [<ffffffffa06448aa>] ? xfs_ilock+0x10a/0x150 [xfs]
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.860986] [<ffffffffa06448aa>] xfs_ilock+0x10a/0x150 [xfs]
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.861003] [<ffffffffa063c79c>] ? xfs_reclaim_inode+0x12c/0x340 [xfs]
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.861020] [<ffffffffa063c79c>] xfs_reclaim_inode+0x12c/0x340 [xfs]
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.861037] [<ffffffffa063cc37>] xfs_reclaim_inodes_ag+0x287/0x400 [xfs]
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.861054] [<ffffffffa063ca90>] ? xfs_reclaim_inodes_ag+0xe0/0x400 [xfs]
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.861073] [<ffffffffa063daa3>] xfs_reclaim_inodes_nr+0x33/0x40 [xfs]
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.861093] [<ffffffffa064d235>] xfs_fs_free_cached_objects+0x15/0x20 [xfs]
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.861099] [<ffffffff811d4618>] super_cache_scan+0x178/0x180
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.861104] [<ffffffff811703fe>] shrink_slab_node+0x14e/0x2f0
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.861110] [<ffffffff81172160>] shrink_slab+0x100/0x140
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.861116] [<ffffffff81174e16>] kswapd_shrink_zone+0x116/0x1a0
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.861122] [<ffffffff81175cdb>] kswapd+0x4ab/0x980
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.861128] [<ffffffff81175830>] ? mem_cgroup_shrink_node_zone+0x190/0x190
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.861134] [<ffffffff81078a34>] kthread+0xe4/0x100
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.861140] [<ffffffff81078950>] ? flush_kthread_worker+0xf0/0xf0
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.861145] [<ffffffff81733e2c>] ret_from_fork+0x7c/0xb0
Nov 29 11:32:09 arh-ibstorage1-ib kernel: [ 1702.861151] [<ffffffff81078950>] ? flush_kthread_worker+0xf0/0xf0
Nov 29 12:05:21 arh-ibstorage1-ib kernel: [ 3695.191632] perf interrupt took too long (2522 > 2500), lowering kernel.perf_event_max_sample_rate to 50000
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.906510] kworker/6:2: page allocation failure: order:5, mode:0x206050
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.906517] kworker/9:1: page allocation failure: order:5, mode:0x206050
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.906521] CPU: 9 PID: 8512 Comm: kworker/9:1 Tainted: G E 3.17.4-ceph-00638-g0f25ebb #1
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.906522] Hardware name: Supermicro X9DRH-7TF/7F/iTF/iF/X9DRH-7TF/7F/iTF/iF, BIOS 3.0b 04/28/2014
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.906537] Workqueue: ceph-msgr con_work [libceph]
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.906540] 0000000000000005 ffff8801929e3478 ffffffff8172a4cf 00000000000018b8
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.906543] 0000000000206050 ffff8801929e3508 ffffffff8116483e 0000000000000000
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.906545] 0000000400000000 ffff88067fff9f48 0000001fffffffe0 0000000000000010
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.906546] Call Trace:
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.906553] [<ffffffff8172a4cf>] dump_stack+0x4e/0x68
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.906560] [<ffffffff8116483e>] warn_alloc_failed+0xee/0x140
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.906564] [<ffffffff8116908b>] __alloc_pages_nodemask+0x96b/0xb70
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.906567] [<ffffffff811692d9>] alloc_kmem_pages_node+0x49/0x60
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.906571] [<ffffffff8172735f>] kmalloc_large_node+0x28/0x56
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.906577] [<ffffffff811b8fbc>] __kmalloc_node_track_caller+0x1cc/0x210
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.906582] [<ffffffff8160c1a7>] ? __alloc_skb+0x87/0x2a0
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.906585] [<ffffffff8160bfd5>] __kmalloc_reserve.isra.56+0x95/0xa0
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.906587] [<ffffffff8160c1a7>] __alloc_skb+0x87/0x2a0
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.906593] [<ffffffff81668901>] sk_stream_alloc_skb+0x41/0x130
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.906596] [<ffffffff8166945c>] tcp_sendmsg+0x3ac/0xdc0
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.906600] [<ffffffff81697325>] ? inet_sendmsg+0x5/0x110
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.906602] [<ffffffff816973ef>] inet_sendmsg+0xcf/0x110
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.906604] [<ffffffff81697325>] ? inet_sendmsg+0x5/0x110
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.906609] [<ffffffff8160364f>] sock_sendmsg+0xaf/0xc0
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.906614] [<ffffffff817336e9>] ? _raw_spin_unlock_bh+0x39/0x40
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.906616] [<ffffffff81669181>] ? tcp_sendmsg+0xd1/0xdc0
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.906618] [<ffffffff816036a0>] kernel_sendmsg+0x40/0x60
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.906621] [<ffffffff81605e7b>] sock_no_sendpage+0x9b/0xb0
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.906625] [<ffffffff81668a47>] tcp_sendpage+0x57/0x680
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.906627] [<ffffffff81697435>] ? inet_sendpage+0x5/0x130
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.906634] [<ffffffffa0512175>] ? con_work+0x55/0x2c10 [libceph]
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.906637] [<ffffffff81697512>] inet_sendpage+0xe2/0x130
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.906639] [<ffffffff81697435>] ? inet_sendpage+0x5/0x130
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.906642] [<ffffffff816006fb>] kernel_sendpage+0x1b/0x30
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.906648] [<ffffffffa05119ab>] ceph_tcp_sendpage+0x5b/0x110 [libceph]
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.906655] [<ffffffffa0512c28>] con_work+0xb08/0x2c10 [libceph]
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.906659] [<ffffffff8107e46f>] ? finish_task_switch+0x3f/0x120
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.906665] [<ffffffff810724cc>] ? process_one_work+0x21c/0x500
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.906669] [<ffffffff810723f3>] ? process_one_work+0x143/0x500
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.906672] [<ffffffff81072301>] ? process_one_work+0x51/0x500
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.906675] [<ffffffff81072455>] process_one_work+0x1a5/0x500
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.906677] [<ffffffff810723f3>] ? process_one_work+0x143/0x500
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.906681] [<ffffffff81072b9c>] worker_thread+0x11c/0x4b0
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.906684] [<ffffffff81072a80>] ? rescuer_thread+0x290/0x290
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.906688] [<ffffffff81078a34>] kthread+0xe4/0x100
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.906691] [<ffffffff81078950>] ? flush_kthread_worker+0xf0/0xf0
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.906693] [<ffffffff81733e2c>] ret_from_fork+0x7c/0xb0
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.906695] [<ffffffff81078950>] ? flush_kthread_worker+0xf0/0xf0
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.906697] Mem-Info:
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.906699] Node 0 DMA per-cpu:
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.906700] CPU 0: hi: 0, btch: 1 usd: 0
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.906702] CPU 1: hi: 0, btch: 1 usd: 0
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.906702] CPU 2: hi: 0, btch: 1 usd: 0
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.906703] CPU 3: hi: 0, btch: 1 usd: 0
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.906705] CPU 4: hi: 0, btch: 1 usd: 0
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.906706] CPU 5: hi: 0, btch: 1 usd: 0
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.906707] CPU 6: hi: 0, btch: 1 usd: 0
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.906708] CPU 7: hi: 0, btch: 1 usd: 0
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.906709] CPU 8: hi: 0, btch: 1 usd: 0
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.906710] CPU 9: hi: 0, btch: 1 usd: 0
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.906711] CPU 10: hi: 0, btch: 1 usd: 0
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.906713] CPU 11: hi: 0, btch: 1 usd: 0
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.906714] Node 0 DMA32 per-cpu:
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.906715] CPU 0: hi: 186, btch: 31 usd: 0
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.906717] CPU 1: hi: 186, btch: 31 usd: 0
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.906718] CPU 2: hi: 186, btch: 31 usd: 0
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.906718] CPU 3: hi: 186, btch: 31 usd: 0
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.906720] CPU 4: hi: 186, btch: 31 usd: 0
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.906721] CPU 5: hi: 186, btch: 31 usd: 0
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.906722] CPU 6: hi: 186, btch: 31 usd: 0
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.906723] CPU 7: hi: 186, btch: 31 usd: 0
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.906724] CPU 8: hi: 186, btch: 31 usd: 0
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.906725] CPU 9: hi: 186, btch: 31 usd: 0
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.906727] CPU 10: hi: 186, btch: 31 usd: 0
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.906728] CPU 11: hi: 186, btch: 31 usd: 0
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.906729] Node 0 Normal per-cpu:
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.906730] CPU 0: hi: 186, btch: 31 usd: 0
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.906731] CPU 1: hi: 186, btch: 31 usd: 0
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.906732] CPU 2: hi: 186, btch: 31 usd: 6
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.906734] CPU 3: hi: 186, btch: 31 usd: 10
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.906735] CPU 4: hi: 186, btch: 31 usd: 0
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.906736] CPU 5: hi: 186, btch: 31 usd: 0
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.906738] CPU 6: hi: 186, btch: 31 usd: 0
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.906738] CPU 7: hi: 186, btch: 31 usd: 0
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.906739] CPU 8: hi: 186, btch: 31 usd: 0
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.906740] CPU 9: hi: 186, btch: 31 usd: 0
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.906742] CPU 10: hi: 186, btch: 31 usd: 0
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.906743] CPU 11: hi: 186, btch: 31 usd: 0
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.906744] Node 1 Normal per-cpu:
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.906745] CPU 0: hi: 186, btch: 31 usd: 0
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.906746] CPU 1: hi: 186, btch: 31 usd: 0
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.906747] CPU 2: hi: 186, btch: 31 usd: 0
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.906749] CPU 3: hi: 186, btch: 31 usd: 0
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.906750] CPU 4: hi: 186, btch: 31 usd: 0
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.906751] CPU 5: hi: 186, btch: 31 usd: 0
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.906752] CPU 6: hi: 186, btch: 31 usd: 0
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.906753] CPU 7: hi: 186, btch: 31 usd: 0
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.906754] CPU 8: hi: 186, btch: 31 usd: 0
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.906755] CPU 9: hi: 186, btch: 31 usd: 0
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.906757] CPU 10: hi: 186, btch: 31 usd: 0
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.906758] CPU 11: hi: 186, btch: 31 usd: 0
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.906762] active_anon:2198590 inactive_anon:394 isolated_anon:0
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.906762] active_file:203581 inactive_file:3280086 isolated_file:270
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.906762] unevictable:0 dirty:207734 writeback:359677 unstable:0
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.906762] free:76692 slab_reclaimable:123446 slab_unreclaimable:52123
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.906762] mapped:17606 shmem:438 pagetables:8715 bounce:0
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.906762] free_cma:0
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.906768] Node 0 DMA free:112kB min:56kB low:68kB high:84kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15956kB managed:15872kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.906772] lowmem_reserve[]: 0 1907 15985 15985
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.906778] Node 0 DMA32 free:76380kB min:7140kB low:8924kB high:10708kB active_anon:139408kB inactive_anon:36kB active_file:18688kB inactive_file:1615060kB unevictable:0kB isolated(anon):0kB isolated(file):516kB present:2047564kB managed:1955656kB mlocked:0kB dirty:76384kB writeback:137124kB mapped:44kB shmem:36kB slab_reclaimable:61204kB slab_unreclaimable:14472kB kernel_stack:5872kB pagetables:200kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.906780] lowmem_reserve[]: 0 0 14078 14078
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.906786] Node 0 Normal free:145704kB min:52716kB low:65892kB high:79072kB active_anon:4086128kB inactive_anon:1344kB active_file:471508kB inactive_file:9139644kB unevictable:0kB isolated(anon):0kB isolated(file):180kB present:14680064kB managed:14416344kB mlocked:0kB dirty:559224kB writeback:955156kB mapped:41996kB shmem:1456kB slab_reclaimable:228640kB slab_unreclaimable:83636kB kernel_stack:24000kB pagetables:16028kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.906788] lowmem_reserve[]: 0 0 0 0
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.906793] Node 1 Normal free:84572kB min:30192kB low:37740kB high:45288kB active_anon:4568824kB inactive_anon:196kB active_file:324128kB inactive_file:2365640kB unevictable:0kB isolated(anon):0kB isolated(file):384kB present:8388608kB managed:8256116kB mlocked:0kB dirty:195328kB writeback:346428kB mapped:28384kB shmem:260kB slab_reclaimable:203940kB slab_unreclaimable:110368kB kernel_stack:30592kB pagetables:18632kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.906795] lowmem_reserve[]: 0 0 0 0
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.906806] Node 0 DMA: 0*4kB 0*8kB 1*16kB (U) 1*32kB (U) 1*64kB (U) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 112kB
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.906816] Node 0 DMA32: 682*4kB (UEM) 1165*8kB (UEM) 1711*16kB (UEM) 672*32kB (UEM) 228*64kB (UEM) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 75520kB
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.906827] Node 0 Normal: 15695*4kB (UEM) 6086*8kB (UEM) 1190*16kB (UEM) 265*32kB (UEM) 78*64kB (UEM) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 143980kB
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.906837] Node 1 Normal: 14707*4kB (UEM) 906*8kB (UEM) 653*16kB (UEM) 246*32kB (EMR) 8*64kB (UMR) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 84908kB
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.906839] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.906841] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.906842] 3484617 total pagecache pages
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.906843] 0 pages in swap cache
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.906845] Swap cache stats: add 0, delete 0, find 0/0
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.906846] Free swap = 0kB
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.906846] Total swap = 0kB
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.906848] 6283048 pages RAM
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.906849] 0 pages HighMem/MovableOnly
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.906849] 33123 pages reserved
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.906850] 0 pages hwpoisoned
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.907160] CPU: 6 PID: 23661 Comm: kworker/6:2 Tainted: G E 3.17.4-ceph-00638-g0f25ebb #1
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.907167] Hardware name: Supermicro X9DRH-7TF/7F/iTF/iF/X9DRH-7TF/7F/iTF/iF, BIOS 3.0b 04/28/2014
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.907181] Workqueue: ceph-msgr con_work [libceph]
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.907186] 0000000000000005 ffff88003942f478 ffffffff8172a4cf ffff88067fc0f258
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.907195] 0000000000206050 ffff88003942f508 ffffffff8116483e 0000000000000000
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.907202] 0000000400000000 ffff88067fff9f48 0000001fffffffe0 0000000000000010
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.907211] Call Trace:
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.907217] [<ffffffff8172a4cf>] dump_stack+0x4e/0x68
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.907224] [<ffffffff8116483e>] warn_alloc_failed+0xee/0x140
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.907231] [<ffffffff8116908b>] __alloc_pages_nodemask+0x96b/0xb70
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.907239] [<ffffffff8107e46f>] ? finish_task_switch+0x3f/0x120
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.907245] [<ffffffff811692d9>] alloc_kmem_pages_node+0x49/0x60
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.907252] [<ffffffff8172735f>] kmalloc_large_node+0x28/0x56
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.907259] [<ffffffff811b8fbc>] __kmalloc_node_track_caller+0x1cc/0x210
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.907266] [<ffffffff8160c1a7>] ? __alloc_skb+0x87/0x2a0
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.907272] [<ffffffff81605d63>] ? lock_sock_nested+0x43/0xc0
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.907278] [<ffffffff8160bfd5>] __kmalloc_reserve.isra.56+0x95/0xa0
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.907283] [<ffffffff8160c1a7>] __alloc_skb+0x87/0x2a0
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.907292] [<ffffffff810a0000>] ? proc_sched_show_task+0x13e0/0x1720
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.907299] [<ffffffff81668901>] sk_stream_alloc_skb+0x41/0x130
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.907305] [<ffffffff8166945c>] tcp_sendmsg+0x3ac/0xdc0
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.907311] [<ffffffff81697325>] ? inet_sendmsg+0x5/0x110
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.907317] [<ffffffff816973ef>] inet_sendmsg+0xcf/0x110
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.907322] [<ffffffff81697325>] ? inet_sendmsg+0x5/0x110
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.907329] [<ffffffff810a738d>] ? trace_hardirqs_on+0xd/0x10
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.907448] [<ffffffff8160364f>] sock_sendmsg+0xaf/0xc0
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.907460] [<ffffffff81607e73>] ? release_sock+0x193/0x1f0
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.907467] [<ffffffff8166aea0>] ? tcp_recvmsg+0x620/0xcb0
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.907472] [<ffffffff81697215>] ? inet_recvmsg+0x5/0x110
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.907479] [<ffffffff816036a0>] kernel_sendmsg+0x40/0x60
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.907485] [<ffffffff81605e7b>] sock_no_sendpage+0x9b/0xb0
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.907515] [<ffffffff81668a47>] tcp_sendpage+0x57/0x680
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.907533] [<ffffffff81697435>] ? inet_sendpage+0x5/0x130
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.907564] [<ffffffffa0513ba6>] ? con_work+0x1a86/0x2c10 [libceph]
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.907592] [<ffffffff81697512>] inet_sendpage+0xe2/0x130
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.907620] [<ffffffff81697435>] ? inet_sendpage+0x5/0x130
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.907640] [<ffffffff816006fb>] kernel_sendpage+0x1b/0x30
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.907653] [<ffffffffa05119ab>] ceph_tcp_sendpage+0x5b/0x110 [libceph]
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.907736] [<ffffffffa0512c28>] con_work+0xb08/0x2c10 [libceph]
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.907743] [<ffffffff81733720>] ? _raw_spin_unlock_irq+0x30/0x40
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.907810] [<ffffffff810724cc>] ? process_one_work+0x21c/0x500
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.907816] [<ffffffff810724cc>] ? process_one_work+0x21c/0x500
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.907823] [<ffffffff81072301>] ? process_one_work+0x51/0x500
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.907832] [<ffffffff81072455>] process_one_work+0x1a5/0x500
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.907838] [<ffffffff810723f3>] ? process_one_work+0x143/0x500
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.907848] [<ffffffff81072b9c>] worker_thread+0x11c/0x4b0
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.907871] [<ffffffff81072a80>] ? rescuer_thread+0x290/0x290
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.907892] [<ffffffff81078a34>] kthread+0xe4/0x100
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.907899] [<ffffffff81078950>] ? flush_kthread_worker+0xf0/0xf0
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.907924] [<ffffffff81733e2c>] ret_from_fork+0x7c/0xb0
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.907943] [<ffffffff81078950>] ? flush_kthread_worker+0xf0/0xf0
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.907959] Mem-Info:
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.907972] Node 0 DMA per-cpu:
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.907984] CPU 0: hi: 0, btch: 1 usd: 0
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.907992] CPU 1: hi: 0, btch: 1 usd: 0
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.908005] CPU 2: hi: 0, btch: 1 usd: 0
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.908016] CPU 3: hi: 0, btch: 1 usd: 0
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.908025] CPU 4: hi: 0, btch: 1 usd: 0
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.908029] CPU 5: hi: 0, btch: 1 usd: 0
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.908033] CPU 6: hi: 0, btch: 1 usd: 0
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.908037] CPU 7: hi: 0, btch: 1 usd: 0
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.908041] CPU 8: hi: 0, btch: 1 usd: 0
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.908045] CPU 9: hi: 0, btch: 1 usd: 0
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.908049] CPU 10: hi: 0, btch: 1 usd: 0
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.908053] CPU 11: hi: 0, btch: 1 usd: 0
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.908057] Node 0 DMA32 per-cpu:
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.908061] CPU 0: hi: 186, btch: 31 usd: 0
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.908065] CPU 1: hi: 186, btch: 31 usd: 0
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.908069] CPU 2: hi: 186, btch: 31 usd: 0
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.908073] CPU 3: hi: 186, btch: 31 usd: 0
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.908077] CPU 4: hi: 186, btch: 31 usd: 0
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.908081] CPU 5: hi: 186, btch: 31 usd: 0
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.908085] CPU 6: hi: 186, btch: 31 usd: 0
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.908089] CPU 7: hi: 186, btch: 31 usd: 0
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.908093] CPU 8: hi: 186, btch: 31 usd: 0
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.908097] CPU 9: hi: 186, btch: 31 usd: 0
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.908101] CPU 10: hi: 186, btch: 31 usd: 0
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.908105] CPU 11: hi: 186, btch: 31 usd: 0
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.908108] Node 0 Normal per-cpu:
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.908113] CPU 0: hi: 186, btch: 31 usd: 0
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.908117] CPU 1: hi: 186, btch: 31 usd: 0
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.908121] CPU 2: hi: 186, btch: 31 usd: 20
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.908125] CPU 3: hi: 186, btch: 31 usd: 21
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.908129] CPU 4: hi: 186, btch: 31 usd: 172
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.908133] CPU 5: hi: 186, btch: 31 usd: 0
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.908137] CPU 6: hi: 186, btch: 31 usd: 0
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.908141] CPU 7: hi: 186, btch: 31 usd: 0
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.908145] CPU 8: hi: 186, btch: 31 usd: 0
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.908149] CPU 9: hi: 186, btch: 31 usd: 0
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.908153] CPU 10: hi: 186, btch: 31 usd: 0
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.908157] CPU 11: hi: 186, btch: 31 usd: 0
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.908168] Node 1 Normal per-cpu:
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.908186] CPU 0: hi: 186, btch: 31 usd: 0
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.908198] CPU 1: hi: 186, btch: 31 usd: 0
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.908210] CPU 2: hi: 186, btch: 31 usd: 0
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.908227] CPU 3: hi: 186, btch: 31 usd: 0
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.908247] CPU 4: hi: 186, btch: 31 usd: 0
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.908252] CPU 5: hi: 186, btch: 31 usd: 0
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.908256] CPU 6: hi: 186, btch: 31 usd: 0
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.908260] CPU 7: hi: 186, btch: 31 usd: 22
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.908264] CPU 8: hi: 186, btch: 31 usd: 0
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.908268] CPU 9: hi: 186, btch: 31 usd: 0
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.908272] CPU 10: hi: 186, btch: 31 usd: 0
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.908276] CPU 11: hi: 186, btch: 31 usd: 0
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.908283] active_anon:2198590 inactive_anon:394 isolated_anon:0
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.908283] active_file:203581 inactive_file:3280182 isolated_file:270
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.908283] unevictable:0 dirty:206794 writeback:361369 unstable:0
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.908283] free:77042 slab_reclaimable:123446 slab_unreclaimable:52123
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.908283] mapped:17606 shmem:438 pagetables:8715 bounce:0
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.908283] free_cma:0
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.908301] Node 0 DMA free:1776kB min:56kB low:68kB high:84kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15956kB managed:15872kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.908328] lowmem_reserve[]: 0 1907 15985 15985
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.908381] Node 0 DMA32 free:76124kB min:7140kB low:8924kB high:10708kB active_anon:139408kB inactive_anon:36kB active_file:18688kB inactive_file:1615060kB unevictable:0kB isolated(anon):0kB isolated(file):516kB present:2047564kB managed:1955656kB mlocked:0kB dirty:76384kB writeback:137124kB mapped:44kB shmem:36kB slab_reclaimable:61204kB slab_unreclaimable:14472kB kernel_stack:5872kB pagetables:200kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.908412] lowmem_reserve[]: 0 0 14078 14078
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.908420] Node 0 Normal free:146628kB min:52716kB low:65892kB high:79072kB active_anon:4086128kB inactive_anon:1344kB active_file:471508kB inactive_file:9139020kB unevictable:0kB isolated(anon):0kB isolated(file):180kB present:14680064kB managed:14416344kB mlocked:0kB dirty:556216kB writeback:959668kB mapped:41996kB shmem:1456kB slab_reclaimable:228640kB slab_unreclaimable:83636kB kernel_stack:24000kB pagetables:16028kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:144 all_unreclaimable? no
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.908446] lowmem_reserve[]: 0 0 0 0
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.908452] Node 1 Normal free:84144kB min:30192kB low:37740kB high:45288kB active_anon:4568824kB inactive_anon:196kB active_file:324128kB inactive_file:2367152kB unevictable:0kB isolated(anon):0kB isolated(file):384kB present:8388608kB managed:8256116kB mlocked:0kB dirty:194576kB writeback:348684kB mapped:28384kB shmem:260kB slab_reclaimable:203940kB slab_unreclaimable:110368kB kernel_stack:30592kB pagetables:18632kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.908479] lowmem_reserve[]: 0 0 0 0
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.908526] Node 0 DMA: 0*4kB 0*8kB 1*16kB (U) 1*32kB (U) 1*64kB (U) 9*128kB (UR) 2*256kB (R) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1776kB
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.908628] Node 0 DMA32: 682*4kB (UEM) 1165*8kB (UEM) 1711*16kB (UEM) 672*32kB (UEM) 228*64kB (UEM) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 75520kB
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.908651] Node 0 Normal: 15694*4kB (UE) 5907*8kB (UEM) 1314*16kB (UEM) 266*32kB (UEM) 78*64kB (UEM) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 144560kB
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.908725] Node 1 Normal: 14344*4kB (UEM) 906*8kB (UEM) 653*16kB (UEM) 246*32kB (EMR) 8*64kB (UMR) 2*128kB (R) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 83712kB
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.908753] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.908760] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.908765] 3484993 total pagecache pages
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.908769] 0 pages in swap cache
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.908773] Swap cache stats: add 0, delete 0, find 0/0
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.908777] Free swap = 0kB
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.908780] Total swap = 0kB
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.908784] 6283048 pages RAM
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.908786] 0 pages HighMem/MovableOnly
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.908790] 33123 pages reserved
Nov 29 12:07:38 arh-ibstorage1-ib kernel: [ 3831.908792] 0 pages hwpoisoned
Nov 29 12:07:46 arh-ibstorage1-ib kernel: [ 3840.009571] kworker/1:2: page allocation failure: order:5, mode:0x206050
Nov 29 12:07:46 arh-ibstorage1-ib kernel: [ 3840.009584] CPU: 1 PID: 9531 Comm: kworker/1:2 Tainted: G E 3.17.4-ceph-00638-g0f25ebb #1
Nov 29 12:07:46 arh-ibstorage1-ib kernel: [ 3840.009591] Hardware name: Supermicro X9DRH-7TF/7F/iTF/iF/X9DRH-7TF/7F/iTF/iF, BIOS 3.0b 04/28/2014
Nov 29 12:07:46 arh-ibstorage1-ib kernel: [ 3840.009612] Workqueue: ceph-msgr con_work [libceph]
Nov 29 12:07:46 arh-ibstorage1-ib kernel: [ 3840.009618] 0000000000000005 ffff88046828b478 ffffffff8172a4cf ffff88046fc4f258
Nov 29 12:07:46 arh-ibstorage1-ib kernel: [ 3840.009626] 0000000000206050 ffff88046828b508 ffffffff8116483e 0000000000000000
Nov 29 12:07:46 arh-ibstorage1-ib kernel: [ 3840.009633] 0000000400000000 ffff88047fffcf48 0000001fffffffe0 0000000000000010
Nov 29 12:07:46 arh-ibstorage1-ib kernel: [ 3840.009642] Call Trace:
Nov 29 12:07:46 arh-ibstorage1-ib kernel: [ 3840.009650] [<ffffffff8172a4cf>] dump_stack+0x4e/0x68
Nov 29 12:07:46 arh-ibstorage1-ib kernel: [ 3840.009658] [<ffffffff8116483e>] warn_alloc_failed+0xee/0x140
Nov 29 12:07:46 arh-ibstorage1-ib kernel: [ 3840.009665] [<ffffffff8116908b>] __alloc_pages_nodemask+0x96b/0xb70
Nov 29 12:07:46 arh-ibstorage1-ib kernel: [ 3840.009672] [<ffffffff817337cb>] ? _raw_spin_unlock+0x2b/0x40
Nov 29 12:07:46 arh-ibstorage1-ib kernel: [ 3840.009679] [<ffffffff811692d9>] alloc_kmem_pages_node+0x49/0x60
Nov 29 12:07:46 arh-ibstorage1-ib kernel: [ 3840.009686] [<ffffffff8172735f>] kmalloc_large_node+0x28/0x56
Nov 29 12:07:46 arh-ibstorage1-ib kernel: [ 3840.009693] [<ffffffff811b8fbc>] __kmalloc_node_track_caller+0x1cc/0x210
Nov 29 12:07:46 arh-ibstorage1-ib kernel: [ 3840.009701] [<ffffffff8160c1a7>] ? __alloc_skb+0x87/0x2a0
Nov 29 12:07:46 arh-ibstorage1-ib kernel: [ 3840.009708] [<ffffffff81605d63>] ? lock_sock_nested+0x43/0xc0
Nov 29 12:07:46 arh-ibstorage1-ib kernel: [ 3840.009713] [<ffffffff8160bfd5>] __kmalloc_reserve.isra.56+0x95/0xa0
Nov 29 12:07:46 arh-ibstorage1-ib kernel: [ 3840.009719] [<ffffffff8160c1a7>] __alloc_skb+0x87/0x2a0
Nov 29 12:07:46 arh-ibstorage1-ib kernel: [ 3840.009727] [<ffffffff81668901>] sk_stream_alloc_skb+0x41/0x130
Nov 29 12:07:46 arh-ibstorage1-ib kernel: [ 3840.009733] [<ffffffff8166945c>] tcp_sendmsg+0x3ac/0xdc0
Nov 29 12:07:46 arh-ibstorage1-ib kernel: [ 3840.009740] [<ffffffff81697325>] ? inet_sendmsg+0x5/0x110
Nov 29 12:07:46 arh-ibstorage1-ib kernel: [ 3840.009746] [<ffffffff816973ef>] inet_sendmsg+0xcf/0x110
Nov 29 12:07:46 arh-ibstorage1-ib kernel: [ 3840.009751] [<ffffffff81697325>] ? inet_sendmsg+0x5/0x110
Nov 29 12:07:46 arh-ibstorage1-ib kernel: [ 3840.009758] [<ffffffff8160364f>] sock_sendmsg+0xaf/0xc0
Nov 29 12:07:46 arh-ibstorage1-ib kernel: [ 3840.009764] [<ffffffff817336e9>] ? _raw_spin_unlock_bh+0x39/0x40
Nov 29 12:07:46 arh-ibstorage1-ib kernel: [ 3840.009770] [<ffffffff81669181>] ? tcp_sendmsg+0xd1/0xdc0
Nov 29 12:07:46 arh-ibstorage1-ib kernel: [ 3840.009776] [<ffffffff816036a0>] kernel_sendmsg+0x40/0x60
Nov 29 12:07:46 arh-ibstorage1-ib kernel: [ 3840.009782] [<ffffffff81605e7b>] sock_no_sendpage+0x9b/0xb0
Nov 29 12:07:46 arh-ibstorage1-ib kernel: [ 3840.009788] [<ffffffff81668a47>] tcp_sendpage+0x57/0x680
Nov 29 12:07:46 arh-ibstorage1-ib kernel: [ 3840.009794] [<ffffffff81697435>] ? inet_sendpage+0x5/0x130
Nov 29 12:07:46 arh-ibstorage1-ib kernel: [ 3840.009800] [<ffffffff81697512>] inet_sendpage+0xe2/0x130
Nov 29 12:07:46 arh-ibstorage1-ib kernel: [ 3840.009805] [<ffffffff81697435>] ? inet_sendpage+0x5/0x130
Nov 29 12:07:46 arh-ibstorage1-ib kernel: [ 3840.009811] [<ffffffff816006fb>] kernel_sendpage+0x1b/0x30
Nov 29 12:07:46 arh-ibstorage1-ib kernel: [ 3840.009822] [<ffffffffa05119ab>] ceph_tcp_sendpage+0x5b/0x110 [libceph]
Nov 29 12:07:46 arh-ibstorage1-ib kernel: [ 3840.009832] [<ffffffffa0512c28>] con_work+0xb08/0x2c10 [libceph]
Nov 29 12:07:46 arh-ibstorage1-ib kernel: [ 3840.009839] [<ffffffff8107e46f>] ? finish_task_switch+0x3f/0x120
Nov 29 12:07:46 arh-ibstorage1-ib kernel: [ 3840.009847] [<ffffffff810724cc>] ? process_one_work+0x21c/0x500
Nov 29 12:07:46 arh-ibstorage1-ib kernel: [ 3840.009853] [<ffffffff810723f3>] ? process_one_work+0x143/0x500
Nov 29 12:07:46 arh-ibstorage1-ib kernel: [ 3840.009860] [<ffffffff81072301>] ? process_one_work+0x51/0x500
Nov 29 12:07:46 arh-ibstorage1-ib kernel: [ 3840.009866] [<ffffffff81072455>] process_one_work+0x1a5/0x500
Nov 29 12:07:46 arh-ibstorage1-ib kernel: [ 3840.009872] [<ffffffff810723f3>] ? process_one_work+0x143/0x500
Nov 29 12:07:46 arh-ibstorage1-ib kernel: [ 3840.009880] [<ffffffff81072b9c>] worker_thread+0x11c/0x4b0
Nov 29 12:07:46 arh-ibstorage1-ib kernel: [ 3840.009886] [<ffffffff81072a80>] ? rescuer_thread+0x290/0x290
Nov 29 12:07:46 arh-ibstorage1-ib kernel: [ 3840.009892] [<ffffffff81078a34>] kthread+0xe4/0x100
Nov 29 12:07:46 arh-ibstorage1-ib kernel: [ 3840.009898] [<ffffffff81078950>] ? flush_kthread_worker+0xf0/0xf0
Nov 29 12:07:46 arh-ibstorage1-ib kernel: [ 3840.009904] [<ffffffff81733e2c>] ret_from_fork+0x7c/0xb0
Nov 29 12:07:46 arh-ibstorage1-ib kernel: [ 3840.009909] [<ffffffff81078950>] ? flush_kthread_worker+0xf0/0xf0
Nov 29 12:07:46 arh-ibstorage1-ib kernel: [ 3840.009914] Mem-Info:
Nov 29 12:07:46 arh-ibstorage1-ib kernel: [ 3840.009918] Node 0 DMA per-cpu:
Nov 29 12:07:46 arh-ibstorage1-ib kernel: [ 3840.009922] CPU 0: hi: 0, btch: 1 usd: 0
Nov 29 12:07:46 arh-ibstorage1-ib kernel: [ 3840.009927] CPU 1: hi: 0, btch: 1 usd: 0
Nov 29 12:07:46 arh-ibstorage1-ib kernel: [ 3840.009930] CPU 2: hi: 0, btch: 1 usd: 0
Nov 29 12:07:46 arh-ibstorage1-ib kernel: [ 3840.009934] CPU 3: hi: 0, btch: 1 usd: 0
Nov 29 12:07:46 arh-ibstorage1-ib kernel: [ 3840.009939] CPU 4: hi: 0, btch: 1 usd: 0
Nov 29 12:07:46 arh-ibstorage1-ib kernel: [ 3840.009943] CPU 5: hi: 0, btch: 1 usd: 0
Nov 29 12:07:46 arh-ibstorage1-ib kernel: [ 3840.009947] CPU 6: hi: 0, btch: 1 usd: 0
Nov 29 12:07:46 arh-ibstorage1-ib kernel: [ 3840.009951] CPU 7: hi: 0, btch: 1 usd: 0
Nov 29 12:07:46 arh-ibstorage1-ib kernel: [ 3840.009955] CPU 8: hi: 0, btch: 1 usd: 0
Nov 29 12:07:46 arh-ibstorage1-ib kernel: [ 3840.009959] CPU 9: hi: 0, btch: 1 usd: 0
Nov 29 12:07:46 arh-ibstorage1-ib kernel: [ 3840.009963] CPU 10: hi: 0, btch: 1 usd: 0
Nov 29 12:07:46 arh-ibstorage1-ib kernel: [ 3840.009967] CPU 11: hi: 0, btch: 1 usd: 0
Nov 29 12:07:46 arh-ibstorage1-ib kernel: [ 3840.009971] Node 0 DMA32 per-cpu:
Nov 29 12:07:46 arh-ibstorage1-ib kernel: [ 3840.009975] CPU 0: hi: 186, btch: 31 usd: 0
Nov 29 12:07:46 arh-ibstorage1-ib kernel: [ 3840.009979] CPU 1: hi: 186, btch: 31 usd: 0
Nov 29 12:07:46 arh-ibstorage1-ib kernel: [ 3840.009983] CPU 2: hi: 186, btch: 31 usd: 0
Nov 29 12:07:46 arh-ibstorage1-ib kernel: [ 3840.009987] CPU 3: hi: 186, btch: 31 usd: 0
Nov 29 12:07:46 arh-ibstorage1-ib kernel: [ 3840.009991] CPU 4: hi: 186, btch: 31 usd: 0
Nov 29 12:07:46 arh-ibstorage1-ib kernel: [ 3840.009995] CPU 5: hi: 186, btch: 31 usd: 0
Nov 29 12:07:46 arh-ibstorage1-ib kernel: [ 3840.009999] CPU 6: hi: 186, btch: 31 usd: 0
Nov 29 12:07:46 arh-ibstorage1-ib kernel: [ 3840.010003] CPU 7: hi: 186, btch: 31 usd: 0
Nov 29 12:07:46 arh-ibstorage1-ib kernel: [ 3840.010008] CPU 8: hi: 186, btch: 31 usd: 0
Nov 29 12:07:46 arh-ibstorage1-ib kernel: [ 3840.010012] CPU 9: hi: 186, btch: 31 usd: 0
Nov 29 12:07:46 arh-ibstorage1-ib kernel: [ 3840.010016] CPU 10: hi: 186, btch: 31 usd: 0
Nov 29 12:07:46 arh-ibstorage1-ib kernel: [ 3840.010020] CPU 11: hi: 186, btch: 31 usd: 0
Nov 29 12:07:46 arh-ibstorage1-ib kernel: [ 3840.010023] Node 0 Normal per-cpu:
Nov 29 12:07:46 arh-ibstorage1-ib kernel: [ 3840.010028] CPU 0: hi: 186, btch: 31 usd: 0
Nov 29 12:07:46 arh-ibstorage1-ib kernel: [ 3840.010031] CPU 1: hi: 186, btch: 31 usd: 0
Nov 29 12:07:46 arh-ibstorage1-ib kernel: [ 3840.010035] CPU 2: hi: 186, btch: 31 usd: 0
Nov 29 12:07:46 arh-ibstorage1-ib kernel: [ 3840.010039] CPU 3: hi: 186, btch: 31 usd: 0
Nov 29 12:07:46 arh-ibstorage1-ib kernel: [ 3840.010043] CPU 4: hi: 186, btch: 31 usd: 0
Nov 29 12:07:46 arh-ibstorage1-ib kernel: [ 3840.010047] CPU 5: hi: 186, btch: 31 usd: 0
Nov 29 12:07:46 arh-ibstorage1-ib kernel: [ 3840.010051] CPU 6: hi: 186, btch: 31 usd: 0
Nov 29 12:07:46 arh-ibstorage1-ib kernel: [ 3840.010055] CPU 7: hi: 186, btch: 31 usd: 0
Nov 29 12:07:46 arh-ibstorage1-ib kernel: [ 3840.010059] CPU 8: hi: 186, btch: 31 usd: 0
Nov 29 12:07:46 arh-ibstorage1-ib kernel: [ 3840.010063] CPU 9: hi: 186, btch: 31 usd: 0
Nov 29 12:07:46 arh-ibstorage1-ib kernel: [ 3840.010067] CPU 10: hi: 186, btch: 31 usd: 0
Nov 29 12:07:46 arh-ibstorage1-ib kernel: [ 3840.010071] CPU 11: hi: 186, btch: 31 usd: 0
Nov 29 12:07:46 arh-ibstorage1-ib kernel: [ 3840.010075] Node 1 Normal per-cpu:
Nov 29 12:07:46 arh-ibstorage1-ib kernel: [ 3840.010080] CPU 0: hi: 186, btch: 31 usd: 0
Nov 29 12:07:46 arh-ibstorage1-ib kernel: [ 3840.010084] CPU 1: hi: 186, btch: 31 usd: 0
Nov 29 12:07:46 arh-ibstorage1-ib kernel: [ 3840.010088] CPU 2: hi: 186, btch: 31 usd: 0
Nov 29 12:07:46 arh-ibstorage1-ib kernel: [ 3840.010092] CPU 3: hi: 186, btch: 31 usd: 0
Nov 29 12:07:46 arh-ibstorage1-ib kernel: [ 3840.010096] CPU 4: hi: 186, btch: 31 usd: 0
Nov 29 12:07:46 arh-ibstorage1-ib kernel: [ 3840.010100] CPU 5: hi: 186, btch: 31 usd: 0
Nov 29 12:07:46 arh-ibstorage1-ib kernel: [ 3840.010104] CPU 6: hi: 186, btch: 31 usd: 0
Nov 29 12:07:46 arh-ibstorage1-ib kernel: [ 3840.010108] CPU 7: hi: 186, btch: 31 usd: 0
Nov 29 12:07:46 arh-ibstorage1-ib kernel: [ 3840.010112] CPU 8: hi: 186, btch: 31 usd: 0
Nov 29 12:07:46 arh-ibstorage1-ib kernel: [ 3840.010116] CPU 9: hi: 186, btch: 31 usd: 0
Nov 29 12:07:46 arh-ibstorage1-ib kernel: [ 3840.010120] CPU 10: hi: 186, btch: 31 usd: 0
Nov 29 12:07:46 arh-ibstorage1-ib kernel: [ 3840.010124] CPU 11: hi: 186, btch: 31 usd: 0
Nov 29 12:07:46 arh-ibstorage1-ib kernel: [ 3840.010131] active_anon:2201112 inactive_anon:394 isolated_anon:0
Nov 29 12:07:46 arh-ibstorage1-ib kernel: [ 3840.010131] active_file:198728 inactive_file:3296014 isolated_file:0
Nov 29 12:07:46 arh-ibstorage1-ib kernel: [ 3840.010131] unevictable:0 dirty:207093 writeback:379777 unstable:0
Nov 29 12:07:46 arh-ibstorage1-ib kernel: [ 3840.010131] free:78114 slab_reclaimable:120008 slab_unreclaimable:52144
Nov 29 12:07:46 arh-ibstorage1-ib kernel: [ 3840.010131] mapped:17419 shmem:438 pagetables:8770 bounce:0
Nov 29 12:07:46 arh-ibstorage1-ib kernel: [ 3840.010131] free_cma:0
Nov 29 12:07:46 arh-ibstorage1-ib kernel: [ 3840.010150] Node 0 DMA free:112kB min:56kB low:68kB high:84kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15956kB managed:15872kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Nov 29 12:07:46 arh-ibstorage1-ib kernel: [ 3840.010174] lowmem_reserve[]: 0 1907 15985 15985
Nov 29 12:07:46 arh-ibstorage1-ib kernel: [ 3840.010181] Node 0 DMA32 free:77880kB min:7140kB low:8924kB high:10708kB active_anon:135864kB inactive_anon:36kB active_file:17776kB inactive_file:1626924kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:2047564kB managed:1955656kB mlocked:0kB dirty:73468kB writeback:145428kB mapped:44kB shmem:36kB slab_reclaimable:58040kB slab_unreclaimable:14608kB kernel_stack:5840kB pagetables:200kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Nov 29 12:07:46 arh-ibstorage1-ib kernel: [ 3840.010207] lowmem_reserve[]: 0 0 14078 14078
Nov 29 12:07:46 arh-ibstorage1-ib kernel: [ 3840.010214] Node 0 Normal free:150968kB min:52716kB low:65892kB high:79072kB active_anon:4090324kB inactive_anon:1344kB active_file:455668kB inactive_file:9166408kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:14680064kB managed:14416344kB mlocked:0kB dirty:581244kB writeback:1009216kB mapped:43100kB shmem:1456kB slab_reclaimable:219444kB slab_unreclaimable:83632kB kernel_stack:24000kB pagetables:16056kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Nov 29 12:07:46 arh-ibstorage1-ib kernel: [ 3840.010240] lowmem_reserve[]: 0 0 0 0
Nov 29 12:07:46 arh-ibstorage1-ib kernel: [ 3840.010246] Node 1 Normal free:83496kB min:30192kB low:37740kB high:45288kB active_anon:4578260kB inactive_anon:196kB active_file:321468kB inactive_file:2390724kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:8388608kB managed:8256116kB mlocked:0kB dirty:173660kB writeback:364464kB mapped:26532kB shmem:260kB slab_reclaimable:202548kB slab_unreclaimable:110320kB kernel_stack:30592kB pagetables:18824kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Nov 29 12:07:46 arh-ibstorage1-ib kernel: [ 3840.010272] lowmem_reserve[]: 0 0 0 0
Nov 29 12:07:46 arh-ibstorage1-ib kernel: [ 3840.010278] Node 0 DMA: 0*4kB 0*8kB 1*16kB (U) 1*32kB (U) 1*64kB (U) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 112kB
Nov 29 12:07:46 arh-ibstorage1-ib kernel: [ 3840.010299] Node 0 DMA32: 1021*4kB (UEM) 725*8kB (UEM) 2341*16kB (UEM) 547*32kB (UEM) 200*64kB (UEM) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 77644kB
Nov 29 12:07:46 arh-ibstorage1-ib kernel: [ 3840.010322] Node 0 Normal: 19778*4kB (UEM) 8444*8kB (UEM) 405*16kB (UEM) 13*32kB (UM) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 153560kB
Nov 29 12:07:46 arh-ibstorage1-ib kernel: [ 3840.010343] Node 1 Normal: 12989*4kB (UEM) 1207*8kB (UEM) 1073*16kB (UEM) 110*32kB (UEMR) 24*64kB (UEMR) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 83836kB
Nov 29 12:07:46 arh-ibstorage1-ib kernel: [ 3840.010368] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Nov 29 12:07:46 arh-ibstorage1-ib kernel: [ 3840.010374] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Nov 29 12:07:46 arh-ibstorage1-ib kernel: [ 3840.010380] 3495432 total pagecache pages
Nov 29 12:07:46 arh-ibstorage1-ib kernel: [ 3840.010384] 0 pages in swap cache
Nov 29 12:07:46 arh-ibstorage1-ib kernel: [ 3840.010388] Swap cache stats: add 0, delete 0, find 0/0
Nov 29 12:07:46 arh-ibstorage1-ib kernel: [ 3840.010392] Free swap = 0kB
Nov 29 12:07:46 arh-ibstorage1-ib kernel: [ 3840.010395] Total swap = 0kB
Nov 29 12:07:46 arh-ibstorage1-ib kernel: [ 3840.010399] 6283048 pages RAM
Nov 29 12:07:46 arh-ibstorage1-ib kernel: [ 3840.010402] 0 pages HighMem/MovableOnly
Nov 29 12:07:46 arh-ibstorage1-ib kernel: [ 3840.010405] 33123 pages reserved
Nov 29 12:07:46 arh-ibstorage1-ib kernel: [ 3840.010408] 0 pages hwpoisoned
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.171025] kworker/6:3: page allocation failure: order:5, mode:0x206050
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.171041] CPU: 6 PID: 8866 Comm: kworker/6:3 Tainted: G E 3.17.4-ceph-00638-g0f25ebb #1
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.171048] Hardware name: Supermicro X9DRH-7TF/7F/iTF/iF/X9DRH-7TF/7F/iTF/iF, BIOS 3.0b 04/28/2014
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.171072] Workqueue: ceph-msgr con_work [libceph]
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.171079] 0000000000000005 ffff8804809eb478 ffffffff8172a4cf ffff88067fc0f258
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.171087] 0000000000206050 ffff8804809eb508 ffffffff8116483e 0000000000000000
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.171095] 0000000400000000 ffff88067fff9f48 0000001fffffffe0 0000000000000010
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.171103] Call Trace:
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.171113] [<ffffffff8172a4cf>] dump_stack+0x4e/0x68
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.171123] [<ffffffff8116483e>] warn_alloc_failed+0xee/0x140
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.171130] [<ffffffff8116908b>] __alloc_pages_nodemask+0x96b/0xb70
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.171139] [<ffffffff817337cb>] ? _raw_spin_unlock+0x2b/0x40
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.171146] [<ffffffff811692d9>] alloc_kmem_pages_node+0x49/0x60
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.171154] [<ffffffff8172735f>] kmalloc_large_node+0x28/0x56
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.171162] [<ffffffff811b8fbc>] __kmalloc_node_track_caller+0x1cc/0x210
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.171171] [<ffffffff8160c1a7>] ? __alloc_skb+0x87/0x2a0
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.171176] [<ffffffff8160bfd5>] __kmalloc_reserve.isra.56+0x95/0xa0
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.171182] [<ffffffff8160c1a7>] __alloc_skb+0x87/0x2a0
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.171192] [<ffffffff81668901>] sk_stream_alloc_skb+0x41/0x130
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.171198] [<ffffffff8166945c>] tcp_sendmsg+0x3ac/0xdc0
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.171206] [<ffffffff81697325>] ? inet_sendmsg+0x5/0x110
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.171212] [<ffffffff816973ef>] inet_sendmsg+0xcf/0x110
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.171217] [<ffffffff81697325>] ? inet_sendmsg+0x5/0x110
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.171225] [<ffffffff8160364f>] sock_sendmsg+0xaf/0xc0
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.171232] [<ffffffff817336e9>] ? _raw_spin_unlock_bh+0x39/0x40
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.171238] [<ffffffff81669181>] ? tcp_sendmsg+0xd1/0xdc0
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.171244] [<ffffffff816036a0>] kernel_sendmsg+0x40/0x60
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.171250] [<ffffffff81605e7b>] sock_no_sendpage+0x9b/0xb0
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.171256] [<ffffffff81668a47>] tcp_sendpage+0x57/0x680
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.171262] [<ffffffff81697435>] ? inet_sendpage+0x5/0x130
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.171268] [<ffffffff81697512>] inet_sendpage+0xe2/0x130
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.171273] [<ffffffff81697435>] ? inet_sendpage+0x5/0x130
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.171279] [<ffffffff816006fb>] kernel_sendpage+0x1b/0x30
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.171289] [<ffffffffa05119ab>] ceph_tcp_sendpage+0x5b/0x110 [libceph]
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.171299] [<ffffffffa0512c28>] con_work+0xb08/0x2c10 [libceph]
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.171310] [<ffffffff810724cc>] ? process_one_work+0x21c/0x500
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.171316] [<ffffffff810723f3>] ? process_one_work+0x143/0x500
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.171322] [<ffffffff81072301>] ? process_one_work+0x51/0x500
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.171328] [<ffffffff81072455>] process_one_work+0x1a5/0x500
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.171334] [<ffffffff810723f3>] ? process_one_work+0x143/0x500
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.171342] [<ffffffff81072b9c>] worker_thread+0x11c/0x4b0
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.171348] [<ffffffff81072a80>] ? rescuer_thread+0x290/0x290
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.171355] [<ffffffff81078a34>] kthread+0xe4/0x100
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.171361] [<ffffffff81078950>] ? flush_kthread_worker+0xf0/0xf0
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.171367] [<ffffffff81733e2c>] ret_from_fork+0x7c/0xb0
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.171373] [<ffffffff81078950>] ? flush_kthread_worker+0xf0/0xf0
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.171378] Mem-Info:
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.171381] Node 0 DMA per-cpu:
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.171386] CPU 0: hi: 0, btch: 1 usd: 0
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.171390] CPU 1: hi: 0, btch: 1 usd: 0
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.171394] CPU 2: hi: 0, btch: 1 usd: 0
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.171398] CPU 3: hi: 0, btch: 1 usd: 0
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.171402] CPU 4: hi: 0, btch: 1 usd: 0
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.171406] CPU 5: hi: 0, btch: 1 usd: 0
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.171429] CPU 6: hi: 0, btch: 1 usd: 0
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.171433] CPU 7: hi: 0, btch: 1 usd: 0
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.171437] CPU 8: hi: 0, btch: 1 usd: 0
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.171441] CPU 9: hi: 0, btch: 1 usd: 0
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.171446] CPU 10: hi: 0, btch: 1 usd: 0
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.171450] CPU 11: hi: 0, btch: 1 usd: 0
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.171453] Node 0 DMA32 per-cpu:
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.171458] CPU 0: hi: 186, btch: 31 usd: 0
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.171462] CPU 1: hi: 186, btch: 31 usd: 0
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.171466] CPU 2: hi: 186, btch: 31 usd: 0
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.171470] CPU 3: hi: 186, btch: 31 usd: 0
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.171474] CPU 4: hi: 186, btch: 31 usd: 0
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.171478] CPU 5: hi: 186, btch: 31 usd: 0
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.171482] CPU 6: hi: 186, btch: 31 usd: 0
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.171486] CPU 7: hi: 186, btch: 31 usd: 0
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.171490] CPU 8: hi: 186, btch: 31 usd: 0
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.171494] CPU 9: hi: 186, btch: 31 usd: 0
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.171498] CPU 10: hi: 186, btch: 31 usd: 0
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.171502] CPU 11: hi: 186, btch: 31 usd: 0
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.171506] Node 0 Normal per-cpu:
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.171510] CPU 0: hi: 186, btch: 31 usd: 0
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.171514] CPU 1: hi: 186, btch: 31 usd: 0
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.171518] CPU 2: hi: 186, btch: 31 usd: 0
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.171522] CPU 3: hi: 186, btch: 31 usd: 0
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.171526] CPU 4: hi: 186, btch: 31 usd: 0
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.171530] CPU 5: hi: 186, btch: 31 usd: 0
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.171534] CPU 6: hi: 186, btch: 31 usd: 0
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.171538] CPU 7: hi: 186, btch: 31 usd: 0
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.171542] CPU 8: hi: 186, btch: 31 usd: 0
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.171546] CPU 9: hi: 186, btch: 31 usd: 0
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.171550] CPU 10: hi: 186, btch: 31 usd: 0
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.171554] CPU 11: hi: 186, btch: 31 usd: 0
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.171558] Node 1 Normal per-cpu:
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.171562] CPU 0: hi: 186, btch: 31 usd: 0
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.171566] CPU 1: hi: 186, btch: 31 usd: 0
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.171570] CPU 2: hi: 186, btch: 31 usd: 0
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.171574] CPU 3: hi: 186, btch: 31 usd: 0
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.171578] CPU 4: hi: 186, btch: 31 usd: 0
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.171582] CPU 5: hi: 186, btch: 31 usd: 0
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.171586] CPU 6: hi: 186, btch: 31 usd: 0
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.171590] CPU 7: hi: 186, btch: 31 usd: 0
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.171594] CPU 8: hi: 186, btch: 31 usd: 0
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.171598] CPU 9: hi: 186, btch: 31 usd: 0
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.171603] CPU 10: hi: 186, btch: 31 usd: 0
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.171607] CPU 11: hi: 186, btch: 31 usd: 0
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.171614] active_anon:2201157 inactive_anon:394 isolated_anon:0
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.171614] active_file:194496 inactive_file:3304234 isolated_file:153
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.171614] unevictable:0 dirty:206859 writeback:389642 unstable:0
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.171614] free:79895 slab_reclaimable:119332 slab_unreclaimable:52451
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.171614] mapped:17140 shmem:438 pagetables:9307 bounce:0
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.171614] free_cma:0
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.171639] Node 0 DMA free:112kB min:56kB low:68kB high:84kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15956kB managed:15872kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.171663] lowmem_reserve[]: 0 1907 15985 15985
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.171671] Node 0 DMA32 free:78524kB min:7140kB low:8924kB high:10708kB active_anon:136576kB inactive_anon:36kB active_file:17884kB inactive_file:1624208kB unevictable:0kB isolated(anon):0kB isolated(file):612kB present:2047564kB managed:1955656kB mlocked:0kB dirty:81628kB writeback:162292kB mapped:44kB shmem:36kB slab_reclaimable:55920kB slab_unreclaimable:15128kB kernel_stack:6464kB pagetables:148kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.171696] lowmem_reserve[]: 0 0 14078 14078
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.171703] Node 0 Normal free:149308kB min:52716kB low:65892kB high:79072kB active_anon:4114352kB inactive_anon:1344kB active_file:440152kB inactive_file:9181356kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:14680064kB managed:14416344kB mlocked:0kB dirty:554456kB writeback:1035272kB mapped:41272kB shmem:1456kB slab_reclaimable:209096kB slab_unreclaimable:81688kB kernel_stack:23856kB pagetables:17640kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.171729] lowmem_reserve[]: 0 0 0 0
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.171736] Node 1 Normal free:91636kB min:30192kB low:37740kB high:45288kB active_anon:4553700kB inactive_anon:196kB active_file:319948kB inactive_file:2411372kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:8388608kB managed:8256116kB mlocked:0kB dirty:190600kB writeback:361756kB mapped:27244kB shmem:260kB slab_reclaimable:212312kB slab_unreclaimable:112972kB kernel_stack:30624kB pagetables:19440kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.171775] lowmem_reserve[]: 0 0 0 0
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.171782] Node 0 DMA: 0*4kB 0*8kB 1*16kB (U) 1*32kB (U) 1*64kB (U) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 112kB
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.171803] Node 0 DMA32: 941*4kB (UEM) 983*8kB (UEM) 1523*16kB (UEM) 677*32kB (UEM) 312*64kB (UEM) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 77628kB
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.171827] Node 0 Normal: 29583*4kB (UEM) 3237*8kB (UEM) 397*16kB (UEM) 6*32kB (M) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 150772kB
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.171849] Node 1 Normal: 14738*4kB (UEM) 1454*8kB (UEM) 580*16kB (UM) 209*32kB (UEMR) 77*64kB (UMR) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 91480kB
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.171884] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.171891] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.171896] 3499644 total pagecache pages
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.171901] 0 pages in swap cache
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.171905] Swap cache stats: add 0, delete 0, find 0/0
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.171908] Free swap = 0kB
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.171911] Total swap = 0kB
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.171915] 6283048 pages RAM
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.171918] 0 pages HighMem/MovableOnly
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.171921] 33123 pages reserved
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.171924] 0 pages hwpoisoned
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.171972] kworker/6:3: page allocation failure: order:5, mode:0x206050
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.171979] CPU: 6 PID: 8866 Comm: kworker/6:3 Tainted: G E 3.17.4-ceph-00638-g0f25ebb #1
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.171985] Hardware name: Supermicro X9DRH-7TF/7F/iTF/iF/X9DRH-7TF/7F/iTF/iF, BIOS 3.0b 04/28/2014
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.172019] Workqueue: ceph-msgr con_work [libceph]
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.172024] 0000000000000005 ffff8804809eb5f8 ffffffff8172a4cf 0000000000006f39
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.172032] 0000000000206050 ffff8804809eb688 ffffffff8116483e 0000000000000000
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.172039] 0000000400000000 ffff88067fff9f48 0000001fffffffe0 0000000000000010
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.172047] Call Trace:
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.172052] [<ffffffff8172a4cf>] dump_stack+0x4e/0x68
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.172059] [<ffffffff8116483e>] warn_alloc_failed+0xee/0x140
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.172065] [<ffffffff8116908b>] __alloc_pages_nodemask+0x96b/0xb70
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.172075] [<ffffffff8109a4c1>] ? finish_wait+0x41/0x80
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.172082] [<ffffffff811692d9>] alloc_kmem_pages_node+0x49/0x60
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.172088] [<ffffffff8172735f>] kmalloc_large_node+0x28/0x56
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.172094] [<ffffffff811b8fbc>] __kmalloc_node_track_caller+0x1cc/0x210
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.172100] [<ffffffff8160c1a7>] ? __alloc_skb+0x87/0x2a0
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.172106] [<ffffffff81605d63>] ? lock_sock_nested+0x43/0xc0
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.172112] [<ffffffff8160bfd5>] __kmalloc_reserve.isra.56+0x95/0xa0
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.172118] [<ffffffff8160c1a7>] __alloc_skb+0x87/0x2a0
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.172124] [<ffffffff810a0000>] ? proc_sched_show_task+0x13e0/0x1720
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.172140] [<ffffffff81668901>] sk_stream_alloc_skb+0x41/0x130
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.172146] [<ffffffff8166945c>] tcp_sendmsg+0x3ac/0xdc0
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.172152] [<ffffffff81697325>] ? inet_sendmsg+0x5/0x110
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.172158] [<ffffffff816973ef>] inet_sendmsg+0xcf/0x110
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.172163] [<ffffffff81697325>] ? inet_sendmsg+0x5/0x110
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.172169] [<ffffffff8160364f>] sock_sendmsg+0xaf/0xc0
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.172178] [<ffffffffa0512175>] ? con_work+0x55/0x2c10 [libceph]
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.172191] [<ffffffff816036a0>] kernel_sendmsg+0x40/0x60
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.172200] [<ffffffffa05101fb>] ceph_tcp_sendmsg+0x4b/0x60 [libceph]
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.172210] [<ffffffffa0512d71>] con_work+0xc51/0x2c10 [libceph]
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.172217] [<ffffffff810724cc>] ? process_one_work+0x21c/0x500
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.172223] [<ffffffff810723f3>] ? process_one_work+0x143/0x500
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.172229] [<ffffffff810723f3>] ? process_one_work+0x143/0x500
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.172270] [<ffffffff81072455>] process_one_work+0x1a5/0x500
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.172279] [<ffffffff810723f3>] ? process_one_work+0x143/0x500
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.172287] [<ffffffff81072b9c>] worker_thread+0x11c/0x4b0
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.172293] [<ffffffff81072a80>] ? rescuer_thread+0x290/0x290
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.172299] [<ffffffff81078a34>] kthread+0xe4/0x100
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.172305] [<ffffffff81078950>] ? flush_kthread_worker+0xf0/0xf0
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.172322] [<ffffffff81733e2c>] ret_from_fork+0x7c/0xb0
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.172328] [<ffffffff81078950>] ? flush_kthread_worker+0xf0/0xf0
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.172332] Mem-Info:
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.172335] Node 0 DMA per-cpu:
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.172339] CPU 0: hi: 0, btch: 1 usd: 0
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.172344] CPU 1: hi: 0, btch: 1 usd: 0
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.172348] CPU 2: hi: 0, btch: 1 usd: 0
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.172352] CPU 3: hi: 0, btch: 1 usd: 0
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.172356] CPU 4: hi: 0, btch: 1 usd: 0
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.172360] CPU 5: hi: 0, btch: 1 usd: 0
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.172364] CPU 6: hi: 0, btch: 1 usd: 0
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.172367] CPU 7: hi: 0, btch: 1 usd: 0
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.172371] CPU 8: hi: 0, btch: 1 usd: 0
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.172375] CPU 9: hi: 0, btch: 1 usd: 0
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.172379] CPU 10: hi: 0, btch: 1 usd: 0
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.172383] CPU 11: hi: 0, btch: 1 usd: 0
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.172387] Node 0 DMA32 per-cpu:
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.172391] CPU 0: hi: 186, btch: 31 usd: 0
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.172401] CPU 1: hi: 186, btch: 31 usd: 0
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.172412] CPU 2: hi: 186, btch: 31 usd: 0
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.172422] CPU 3: hi: 186, btch: 31 usd: 0
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.172437] CPU 4: hi: 186, btch: 31 usd: 0
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.172448] CPU 5: hi: 186, btch: 31 usd: 0
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.172464] CPU 6: hi: 186, btch: 31 usd: 0
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.172474] CPU 7: hi: 186, btch: 31 usd: 0
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.172484] CPU 8: hi: 186, btch: 31 usd: 0
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.172498] CPU 9: hi: 186, btch: 31 usd: 0
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.172510] CPU 10: hi: 186, btch: 31 usd: 0
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.172517] CPU 11: hi: 186, btch: 31 usd: 0
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.172521] Node 0 Normal per-cpu:
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.172525] CPU 0: hi: 186, btch: 31 usd: 0
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.172529] CPU 1: hi: 186, btch: 31 usd: 0
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.172533] CPU 2: hi: 186, btch: 31 usd: 0
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.172546] CPU 3: hi: 186, btch: 31 usd: 0
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.172550] CPU 4: hi: 186, btch: 31 usd: 0
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.172554] CPU 5: hi: 186, btch: 31 usd: 0
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.172558] CPU 6: hi: 186, btch: 31 usd: 0
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.172563] CPU 7: hi: 186, btch: 31 usd: 0
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.172567] CPU 8: hi: 186, btch: 31 usd: 0
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.172571] CPU 9: hi: 186, btch: 31 usd: 0
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.172575] CPU 10: hi: 186, btch: 31 usd: 0
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.172578] CPU 11: hi: 186, btch: 31 usd: 0
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.172582] Node 1 Normal per-cpu:
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.172587] CPU 0: hi: 186, btch: 31 usd: 0
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.172591] CPU 1: hi: 186, btch: 31 usd: 0
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.172595] CPU 2: hi: 186, btch: 31 usd: 0
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.172599] CPU 3: hi: 186, btch: 31 usd: 0
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.172603] CPU 4: hi: 186, btch: 31 usd: 0
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.172607] CPU 5: hi: 186, btch: 31 usd: 0
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.172610] CPU 6: hi: 186, btch: 31 usd: 0
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.172614] CPU 7: hi: 186, btch: 31 usd: 0
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.172618] CPU 8: hi: 186, btch: 31 usd: 0
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.172622] CPU 9: hi: 186, btch: 31 usd: 0
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.172626] CPU 10: hi: 186, btch: 31 usd: 0
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.172630] CPU 11: hi: 186, btch: 31 usd: 30
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.172637] active_anon:2201157 inactive_anon:394 isolated_anon:0
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.172637] active_file:194496 inactive_file:3304234 isolated_file:153
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.172637] unevictable:0 dirty:205827 writeback:390994 unstable:0
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.172637] free:79831 slab_reclaimable:119332 slab_unreclaimable:52451
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.172637] mapped:17140 shmem:438 pagetables:9307 bounce:0
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.172637] free_cma:0
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.172656] Node 0 DMA free:112kB min:56kB low:68kB high:84kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15956kB managed:15872kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.172687] lowmem_reserve[]: 0 1907 15985 15985
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.172733] Node 0 DMA32 free:78268kB min:7140kB low:8924kB high:10708kB active_anon:136576kB inactive_anon:36kB active_file:17884kB inactive_file:1624208kB unevictable:0kB isolated(anon):0kB isolated(file):612kB present:2047564kB managed:1955656kB mlocked:0kB dirty:80508kB writeback:163188kB mapped:44kB shmem:36kB slab_reclaimable:55920kB slab_unreclaimable:15128kB kernel_stack:6464kB pagetables:148kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.172764] lowmem_reserve[]: 0 0 14078 14078
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.172771] Node 0 Normal free:149308kB min:52716kB low:65892kB high:79072kB active_anon:4114352kB inactive_anon:1344kB active_file:440152kB inactive_file:9181356kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:14680064kB managed:14416344kB mlocked:0kB dirty:554456kB writeback:1036776kB mapped:41272kB shmem:1456kB slab_reclaimable:209096kB slab_unreclaimable:81688kB kernel_stack:23856kB pagetables:17640kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.172798] lowmem_reserve[]: 0 0 0 0
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.172804] Node 1 Normal free:91636kB min:30192kB low:37740kB high:45288kB active_anon:4553700kB inactive_anon:196kB active_file:319948kB inactive_file:2411372kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:8388608kB managed:8256116kB mlocked:0kB dirty:188344kB writeback:364012kB mapped:27244kB shmem:260kB slab_reclaimable:212312kB slab_unreclaimable:112972kB kernel_stack:30624kB pagetables:19440kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.172838] lowmem_reserve[]: 0 0 0 0
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.172883] Node 0 DMA: 0*4kB 0*8kB 1*16kB (U) 1*32kB (U) 1*64kB (U) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 112kB
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.172903] Node 0 DMA32: 941*4kB (UEM) 983*8kB (UEM) 1523*16kB (UEM) 677*32kB (UEM) 312*64kB (UEM) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 77628kB
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.172925] Node 0 Normal: 29531*4kB (UEM) 3237*8kB (UEM) 397*16kB (UEM) 6*32kB (M) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 150564kB
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.172947] Node 1 Normal: 14736*4kB (UEM) 1454*8kB (UEM) 579*16kB (UM) 209*32kB (UEMR) 77*64kB (UMR) 5*128kB (UR) 1*256kB (R) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 92352kB
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.173056] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.173067] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.173082] 3499644 total pagecache pages
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.173085] 0 pages in swap cache
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.173089] Swap cache stats: add 0, delete 0, find 0/0
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.173093] Free swap = 0kB
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.173096] Total swap = 0kB
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.173099] 6283048 pages RAM
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.173102] 0 pages HighMem/MovableOnly
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.173105] 33123 pages reserved
Nov 29 12:08:17 arh-ibstorage1-ib kernel: [ 3871.173108] 0 pages hwpoisoned
Nov 29 12:08:20 arh-ibstorage1-ib kernel: [ 3873.919333] kworker/6:3: page allocation failure: order:5, mode:0x206050
Nov 29 12:08:20 arh-ibstorage1-ib kernel: [ 3873.919353] CPU: 6 PID: 8866 Comm: kworker/6:3 Tainted: G E 3.17.4-ceph-00638-g0f25ebb #1
Nov 29 12:08:20 arh-ibstorage1-ib kernel: [ 3873.919360] Hardware name: Supermicro X9DRH-7TF/7F/iTF/iF/X9DRH-7TF/7F/iTF/iF, BIOS 3.0b 04/28/2014
Nov 29 12:08:20 arh-ibstorage1-ib kernel: [ 3873.919380] Workqueue: ceph-msgr con_work [libceph]
Nov 29 12:08:20 arh-ibstorage1-ib kernel: [ 3873.919386] 0000000000000005 ffff8804809eb478 ffffffff8172a4cf ffff88067fc0f258
Nov 29 12:08:20 arh-ibstorage1-ib kernel: [ 3873.919395] 0000000000206050 ffff8804809eb508 ffffffff8116483e 0000000000000000
Nov 29 12:08:20 arh-ibstorage1-ib kernel: [ 3873.919402] 0000000400000000 ffff88067fff9f48 0000001fffffffe0 0000000000000010
Nov 29 12:08:20 arh-ibstorage1-ib kernel: [ 3873.919411] Call Trace:
Nov 29 12:08:20 arh-ibstorage1-ib kernel: [ 3873.919421] [<ffffffff8172a4cf>] dump_stack+0x4e/0x68
Nov 29 12:08:20 arh-ibstorage1-ib kernel: [ 3873.919431] [<ffffffff8116483e>] warn_alloc_failed+0xee/0x140
Nov 29 12:08:20 arh-ibstorage1-ib kernel: [ 3873.919439] [<ffffffff8116908b>] __alloc_pages_nodemask+0x96b/0xb70
Nov 29 12:08:20 arh-ibstorage1-ib kernel: [ 3873.919448] [<ffffffff817337cb>] ? _raw_spin_unlock+0x2b/0x40
Nov 29 12:08:20 arh-ibstorage1-ib kernel: [ 3873.919455] [<ffffffff811692d9>] alloc_kmem_pages_node+0x49/0x60
Nov 29 12:08:20 arh-ibstorage1-ib kernel: [ 3873.919463] [<ffffffff8172735f>] kmalloc_large_node+0x28/0x56
Nov 29 12:08:20 arh-ibstorage1-ib kernel: [ 3873.919471] [<ffffffff811b8fbc>] __kmalloc_node_track_caller+0x1cc/0x210
Nov 29 12:08:20 arh-ibstorage1-ib kernel: [ 3873.919480] [<ffffffff8160c1a7>] ? __alloc_skb+0x87/0x2a0
Nov 29 12:08:20 arh-ibstorage1-ib kernel: [ 3873.919486] [<ffffffff8160bfd5>] __kmalloc_reserve.isra.56+0x95/0xa0
Nov 29 12:08:20 arh-ibstorage1-ib kernel: [ 3873.919491] [<ffffffff8160c1a7>] __alloc_skb+0x87/0x2a0
Nov 29 12:08:20 arh-ibstorage1-ib kernel: [ 3873.919501] [<ffffffff81668901>] sk_stream_alloc_skb+0x41/0x130
Nov 29 12:08:20 arh-ibstorage1-ib kernel: [ 3873.919507] [<ffffffff8166945c>] tcp_sendmsg+0x3ac/0xdc0
Nov 29 12:08:20 arh-ibstorage1-ib kernel: [ 3873.919515] [<ffffffff81697325>] ? inet_sendmsg+0x5/0x110
Nov 29 12:08:20 arh-ibstorage1-ib kernel: [ 3873.919520] [<ffffffff816973ef>] inet_sendmsg+0xcf/0x110
Nov 29 12:08:20 arh-ibstorage1-ib kernel: [ 3873.919526] [<ffffffff81697325>] ? inet_sendmsg+0x5/0x110
Nov 29 12:08:20 arh-ibstorage1-ib kernel: [ 3873.919534] [<ffffffff810a738d>] ? trace_hardirqs_on+0xd/0x10
Nov 29 12:08:20 arh-ibstorage1-ib kernel: [ 3873.919543] [<ffffffff8160364f>] sock_sendmsg+0xaf/0xc0
Nov 29 12:08:20 arh-ibstorage1-ib kernel: [ 3873.919550] [<ffffffff81607e73>] ? release_sock+0x193/0x1f0
Nov 29 12:08:20 arh-ibstorage1-ib kernel: [ 3873.919556] [<ffffffff8166aea0>] ? tcp_recvmsg+0x620/0xcb0
Nov 29 12:08:20 arh-ibstorage1-ib kernel: [ 3873.919561] [<ffffffff81697215>] ? inet_recvmsg+0x5/0x110
Nov 29 12:08:20 arh-ibstorage1-ib kernel: [ 3873.919567] [<ffffffff816036a0>] kernel_sendmsg+0x40/0x60
Nov 29 12:08:20 arh-ibstorage1-ib kernel: [ 3873.919574] [<ffffffff81605e7b>] sock_no_sendpage+0x9b/0xb0
Nov 29 12:08:20 arh-ibstorage1-ib kernel: [ 3873.919580] [<ffffffff81668a47>] tcp_sendpage+0x57/0x680
Nov 29 12:08:20 arh-ibstorage1-ib kernel: [ 3873.919585] [<ffffffff81697435>] ? inet_sendpage+0x5/0x130
Nov 29 12:08:20 arh-ibstorage1-ib kernel: [ 3873.919591] [<ffffffff81697512>] inet_sendpage+0xe2/0x130
Nov 29 12:08:20 arh-ibstorage1-ib kernel: [ 3873.919597] [<ffffffff81697435>] ? inet_sendpage+0x5/0x130
Nov 29 12:08:20 arh-ibstorage1-ib kernel: [ 3873.919603] [<ffffffff816006fb>] kernel_sendpage+0x1b/0x30
Nov 29 12:08:20 arh-ibstorage1-ib kernel: [ 3873.919613] [<ffffffffa05119ab>] ceph_tcp_sendpage+0x5b/0x110 [libceph]
Nov 29 12:08:20 arh-ibstorage1-ib kernel: [ 3873.919624] [<ffffffffa0512c28>] con_work+0xb08/0x2c10 [libceph]
Nov 29 12:08:20 arh-ibstorage1-ib kernel: [ 3873.919712] [<ffffffff811c8faa>] ? mem_cgroup_end_page_stat+0x3a/0x50
Nov 29 12:08:20 arh-ibstorage1-ib kernel: [ 3873.919719] [<ffffffff8116bc0c>] ? test_clear_page_writeback+0xfc/0x1e0
Nov 29 12:08:20 arh-ibstorage1-ib kernel: [ 3873.919732] [<ffffffff810724cc>] ? process_one_work+0x21c/0x500
Nov 29 12:08:20 arh-ibstorage1-ib kernel: [ 3873.919760] [<ffffffff810723f3>] ? process_one_work+0x143/0x500
Nov 29 12:08:20 arh-ibstorage1-ib kernel: [ 3873.919787] [<ffffffff81072301>] ? process_one_work+0x51/0x500
Nov 29 12:08:20 arh-ibstorage1-ib kernel: [ 3873.919804] [<ffffffff81072455>] process_one_work+0x1a5/0x500
Nov 29 12:08:20 arh-ibstorage1-ib kernel: [ 3873.919827] [<ffffffff810723f3>] ? process_one_work+0x143/0x500
Nov 29 12:08:20 arh-ibstorage1-ib kernel: [ 3873.919848] [<ffffffff81072b9c>] worker_thread+0x11c/0x4b0
Nov 29 12:08:20 arh-ibstorage1-ib kernel: [ 3873.919855] [<ffffffff81072a80>] ? rescuer_thread+0x290/0x290
Nov 29 12:08:20 arh-ibstorage1-ib kernel: [ 3873.919861] [<ffffffff81078a34>] kthread+0xe4/0x100
Nov 29 12:08:20 arh-ibstorage1-ib kernel: [ 3873.919870] [<ffffffff81078950>] ? flush_kthread_worker+0xf0/0xf0
Nov 29 12:08:20 arh-ibstorage1-ib kernel: [ 3873.919876] [<ffffffff81733e2c>] ret_from_fork+0x7c/0xb0
Nov 29 12:08:20 arh-ibstorage1-ib kernel: [ 3873.919884] [<ffffffff81078950>] ? flush_kthread_worker+0xf0/0xf0
Nov 29 12:08:20 arh-ibstorage1-ib kernel: [ 3873.919890] Mem-Info:
Nov 29 12:08:20 arh-ibstorage1-ib kernel: [ 3873.919893] Node 0 DMA per-cpu:
Nov 29 12:08:20 arh-ibstorage1-ib kernel: [ 3873.919898] CPU 0: hi: 0, btch: 1 usd: 0
Nov 29 12:08:20 arh-ibstorage1-ib kernel: [ 3873.919905] CPU 1: hi: 0, btch: 1 usd: 0
Nov 29 12:08:20 arh-ibstorage1-ib kernel: [ 3873.919909] CPU 2: hi: 0, btch: 1 usd: 0
Nov 29 12:08:20 arh-ibstorage1-ib kernel: [ 3873.919913] CPU 3: hi: 0, btch: 1 usd: 0
Nov 29 12:08:20 arh-ibstorage1-ib kernel: [ 3873.919916] CPU 4: hi: 0, btch: 1 usd: 0
Nov 29 12:08:20 arh-ibstorage1-ib kernel: [ 3873.919922] CPU 5: hi: 0, btch: 1 usd: 0
Nov 29 12:08:20 arh-ibstorage1-ib kernel: [ 3873.919926] CPU 6: hi: 0, btch: 1 usd: 0
Nov 29 12:08:20 arh-ibstorage1-ib kernel: [ 3873.919931] CPU 7: hi: 0, btch: 1 usd: 0
Nov 29 12:08:20 arh-ibstorage1-ib kernel: [ 3873.919934] CPU 8: hi: 0, btch: 1 usd: 0
Nov 29 12:08:20 arh-ibstorage1-ib kernel: [ 3873.919938] CPU 9: hi: 0, btch: 1 usd: 0
Nov 29 12:08:20 arh-ibstorage1-ib kernel: [ 3873.919945] CPU 10: hi: 0, btch: 1 usd: 0
Nov 29 12:08:20 arh-ibstorage1-ib kernel: [ 3873.919949] CPU 11: hi: 0, btch: 1 usd: 0
Nov 29 12:08:20 arh-ibstorage1-ib kernel: [ 3873.919953] Node 0 DMA32 per-cpu:
Nov 29 12:08:20 arh-ibstorage1-ib kernel: [ 3873.919957] CPU 0: hi: 186, btch: 31 usd: 0
Nov 29 12:08:20 arh-ibstorage1-ib kernel: [ 3873.919969] CPU 1: hi: 186, btch: 31 usd: 0
Nov 29 12:08:20 arh-ibstorage1-ib kernel: [ 3873.919984] CPU 2: hi: 186, btch: 31 usd: 0
Nov 29 12:08:20 arh-ibstorage1-ib kernel: [ 3873.919997] CPU 3: hi: 186, btch: 31 usd: 0
Nov 29 12:08:20 arh-ibstorage1-ib kernel: [ 3873.920004] CPU 4: hi: 186, btch: 31 usd: 0
Nov 29 12:08:20 arh-ibstorage1-ib kernel: [ 3873.920008] CPU 5: hi: 186, btch: 31 usd: 0
Nov 29 12:08:20 arh-ibstorage1-ib kernel: [ 3873.920013] CPU 6: hi: 186, btch: 31 usd: 0
Nov 29 12:08:20 arh-ibstorage1-ib kernel: [ 3873.920016] CPU 7: hi: 186, btch: 31 usd: 0
Nov 29 12:08:20 arh-ibstorage1-ib kernel: [ 3873.920021] CPU 8: hi: 186, btch: 31 usd: 0
Nov 29 12:08:20 arh-ibstorage1-ib kernel: [ 3873.920025] CPU 9: hi: 186, btch: 31 usd: 0
Nov 29 12:08:20 arh-ibstorage1-ib kernel: [ 3873.920029] CPU 10: hi: 186, btch: 31 usd: 0
Nov 29 12:08:20 arh-ibstorage1-ib kernel: [ 3873.920033] CPU 11: hi: 186, btch: 31 usd: 0
Nov 29 12:08:20 arh-ibstorage1-ib kernel: [ 3873.920036] Node 0 Normal per-cpu:
Nov 29 12:08:20 arh-ibstorage1-ib kernel: [ 3873.920049] CPU 0: hi: 186, btch: 31 usd: 27
Nov 29 12:08:20 arh-ibstorage1-ib kernel: [ 3873.920059] CPU 1: hi: 186, btch: 31 usd: 0
Nov 29 12:08:20 arh-ibstorage1-ib kernel: [ 3873.920063] CPU 2: hi: 186, btch: 31 usd: 0
Nov 29 12:08:20 arh-ibstorage1-ib kernel: [ 3873.920067] CPU 3: hi: 186, btch: 31 usd: 0
Nov 29 12:08:20 arh-ibstorage1-ib kernel: [ 3873.920073] CPU 4: hi: 186, btch: 31 usd: 0
Nov 29 12:08:20 arh-ibstorage1-ib kernel: [ 3873.920080] CPU 5: hi: 186, btch: 31 usd: 0
Nov 29 12:08:20 arh-ibstorage1-ib kernel: [ 3873.920084] CPU 6: hi: 186, btch: 31 usd: 0
Nov 29 12:08:20 arh-ibstorage1-ib kernel: [ 3873.920088] CPU 7: hi: 186, btch: 31 usd: 0
Nov 29 12:08:20 arh-ibstorage1-ib kernel: [ 3873.920093] CPU 8: hi: 186, btch: 31 usd: 0
Nov 29 12:08:20 arh-ibstorage1-ib kernel: [ 3873.920098] CPU 9: hi: 186, btch: 31 usd: 0
Nov 29 12:08:20 arh-ibstorage1-ib kernel: [ 3873.920103] CPU 10: hi: 186, btch: 31 usd: 0
Nov 29 12:08:20 arh-ibstorage1-ib kernel: [ 3873.920107] CPU 11: hi: 186, btch: 31 usd: 0
Nov 29 12:08:20 arh-ibstorage1-ib kernel: [ 3873.920111] Node 1 Normal per-cpu:
Nov 29 12:08:20 arh-ibstorage1-ib kernel: [ 3873.920115] CPU 0: hi: 186, btch: 31 usd: 0
Nov 29 12:08:20 arh-ibstorage1-ib kernel: [ 3873.920119] CPU 1: hi: 186, btch: 31 usd: 0
Nov 29 12:08:20 arh-ibstorage1-ib kernel: [ 3873.920123] CPU 2: hi: 186, btch: 31 usd: 0
Nov 29 12:08:20 arh-ibstorage1-ib kernel: [ 3873.920127] CPU 3: hi: 186, btch: 31 usd: 0
Nov 29 12:08:20 arh-ibstorage1-ib kernel: [ 3873.920131] CPU 4: hi: 186, btch: 31 usd: 0
Nov 29 12:08:20 arh-ibstorage1-ib kernel: [ 3873.920135] CPU 5: hi: 186, btch: 31 usd: 0
Nov 29 12:08:20 arh-ibstorage1-ib kernel: [ 3873.920139] CPU 6: hi: 186, btch: 31 usd: 0
Nov 29 12:08:20 arh-ibstorage1-ib kernel: [ 3873.920143] CPU 7: hi: 186, btch: 31 usd: 0
Nov 29 12:08:20 arh-ibstorage1-ib kernel: [ 3873.920147] CPU 8: hi: 186, btch: 31 usd: 0
Nov 29 12:08:20 arh-ibstorage1-ib kernel: [ 3873.920151] CPU 9: hi: 186, btch: 31 usd: 0
Nov 29 12:08:20 arh-ibstorage1-ib kernel: [ 3873.920155] CPU 10: hi: 186, btch: 31 usd: 9
Nov 29 12:08:20 arh-ibstorage1-ib kernel: [ 3873.920159] CPU 11: hi: 186, btch: 31 usd: 0
Nov 29 12:08:20 arh-ibstorage1-ib kernel: [ 3873.920167] active_anon:2199021 inactive_anon:394 isolated_anon:0
Nov 29 12:08:20 arh-ibstorage1-ib kernel: [ 3873.920167] active_file:195065 inactive_file:3309485 isolated_file:72
Nov 29 12:08:20 arh-ibstorage1-ib kernel: [ 3873.920167] unevictable:0 dirty:165220 writeback:444306 unstable:0
Nov 29 12:08:20 arh-ibstorage1-ib kernel: [ 3873.920167] free:77914 slab_reclaimable:120223 slab_unreclaimable:52519
Nov 29 12:08:20 arh-ibstorage1-ib kernel: [ 3873.920167] mapped:17629 shmem:438 pagetables:8919 bounce:0
Nov 29 12:08:20 arh-ibstorage1-ib kernel: [ 3873.920167] free_cma:0
Nov 29 12:08:20 arh-ibstorage1-ib kernel: [ 3873.920186] Node 0 DMA free:1776kB min:56kB low:68kB high:84kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15956kB managed:15872kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Nov 29 12:08:20 arh-ibstorage1-ib kernel: [ 3873.920233] lowmem_reserve[]: 0 1907 15985 15985
Nov 29 12:08:20 arh-ibstorage1-ib kernel: [ 3873.920241] Node 0 DMA32 free:79444kB min:7140kB low:8924kB high:10708kB active_anon:134928kB inactive_anon:36kB active_file:17884kB inactive_file:1629612kB unevictable:0kB isolated(anon):0kB isolated(file):32kB present:2047564kB managed:1955656kB mlocked:0kB dirty:37688kB writeback:150948kB mapped:44kB shmem:36kB slab_reclaimable:59096kB slab_unreclaimable:13664kB kernel_stack:5904kB pagetables:200kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Nov 29 12:08:20 arh-ibstorage1-ib kernel: [ 3873.920267] lowmem_reserve[]: 0 0 14078 14078
Nov 29 12:08:20 arh-ibstorage1-ib kernel: [ 3873.920273] Node 0 Normal free:146708kB min:52716kB low:65892kB high:79072kB active_anon:4098684kB inactive_anon:1344kB active_file:441700kB inactive_file:9205276kB unevictable:0kB isolated(anon):0kB isolated(file):256kB present:14680064kB managed:14416344kB mlocked:0kB dirty:485684kB writeback:1217180kB mapped:42512kB shmem:1456kB slab_reclaimable:207664kB slab_unreclaimable:81456kB kernel_stack:23856kB pagetables:16528kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Nov 29 12:08:20 arh-ibstorage1-ib kernel: [ 3873.920300] lowmem_reserve[]: 0 0 0 0
Nov 29 12:08:20 arh-ibstorage1-ib kernel: [ 3873.920306] Node 1 Normal free:83144kB min:30192kB low:37740kB high:45288kB active_anon:4562472kB inactive_anon:196kB active_file:320676kB inactive_file:2403556kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:8388608kB managed:8256116kB mlocked:0kB dirty:137508kB writeback:409096kB mapped:27960kB shmem:260kB slab_reclaimable:214132kB slab_unreclaimable:114940kB kernel_stack:31072kB pagetables:18948kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:128 all_unreclaimable? no
Nov 29 12:08:20 arh-ibstorage1-ib kernel: [ 3873.920359] lowmem_reserve[]: 0 0 0 0
Nov 29 12:08:20 arh-ibstorage1-ib kernel: [ 3873.920366] Node 0 DMA: 0*4kB 0*8kB 1*16kB (U) 1*32kB (U) 1*64kB (U) 1*128kB (U) 2*256kB (U) 4*512kB (U) 1*1024kB (U) 0*2048kB 0*4096kB = 3824kB
Nov 29 12:08:20 arh-ibstorage1-ib kernel: [ 3873.920391] Node 0 DMA32: 1017*4kB (UEM) 1062*8kB (UEM) 1445*16kB (UEM) 872*32kB (UEM) 245*64kB (UEM) 1*128kB (R) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 79396kB
Nov 29 12:08:20 arh-ibstorage1-ib kernel: [ 3873.920415] Node 0 Normal: 15098*4kB (UEM) 10249*8kB (UEM) 229*16kB (UEM) 22*32kB (EM) 5*64kB (UEM) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 147072kB
Nov 29 12:08:20 arh-ibstorage1-ib kernel: [ 3873.920437] Node 1 Normal: 13335*4kB (UEM) 1513*8kB (UEM) 587*16kB (UEM) 214*32kB (UEMR) 51*64kB (UMR) 2*128kB (R) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 85204kB
Nov 29 12:08:20 arh-ibstorage1-ib kernel: [ 3873.920462] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Nov 29 12:08:20 arh-ibstorage1-ib kernel: [ 3873.920469] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Nov 29 12:08:20 arh-ibstorage1-ib kernel: [ 3873.920475] 3505220 total pagecache pages
Nov 29 12:08:20 arh-ibstorage1-ib kernel: [ 3873.920479] 0 pages in swap cache
Nov 29 12:08:20 arh-ibstorage1-ib kernel: [ 3873.920483] Swap cache stats: add 0, delete 0, find 0/0
Nov 29 12:08:20 arh-ibstorage1-ib kernel: [ 3873.920486] Free swap = 0kB
Nov 29 12:08:20 arh-ibstorage1-ib kernel: [ 3873.920489] Total swap = 0kB
Nov 29 12:08:20 arh-ibstorage1-ib kernel: [ 3873.920493] 6283048 pages RAM
Nov 29 12:08:20 arh-ibstorage1-ib kernel: [ 3873.920496] 0 pages HighMem/MovableOnly
Nov 29 12:08:20 arh-ibstorage1-ib kernel: [ 3873.920500] 33123 pages reserved
Nov 29 12:08:20 arh-ibstorage1-ib kernel: [ 3873.920503] 0 pages hwpoisoned
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.454322] kworker/1:2: page allocation failure: order:5, mode:0x206050
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.454335] kworker/2:1: page allocation failure: order:5, mode:0x206050
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.454337] CPU: 1 PID: 9531 Comm: kworker/1:2 Tainted: G E 3.17.4-ceph-00638-g0f25ebb #1
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.454340] Hardware name: Supermicro X9DRH-7TF/7F/iTF/iF/X9DRH-7TF/7F/iTF/iF, BIOS 3.0b 04/28/2014
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.454356] Workqueue: ceph-msgr con_work [libceph]
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.454359] 0000000000000005 ffff88046828b478 ffffffff8172a4cf ffff88046fc4f258
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.454362] 0000000000206050 ffff88046828b508 ffffffff8116483e 0000000000000000
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.454364] 0000000400000000 ffff88047fffcf48 0000001fffffffe0 0000000000000010
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.454365] Call Trace:
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.454371] [<ffffffff8172a4cf>] dump_stack+0x4e/0x68
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.454376] [<ffffffff8116483e>] warn_alloc_failed+0xee/0x140
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.454380] [<ffffffff8116908b>] __alloc_pages_nodemask+0x96b/0xb70
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.454384] [<ffffffff817337cb>] ? _raw_spin_unlock+0x2b/0x40
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.454387] [<ffffffff811692d9>] alloc_kmem_pages_node+0x49/0x60
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.454390] [<ffffffff8172735f>] kmalloc_large_node+0x28/0x56
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.454394] [<ffffffff811b8fbc>] __kmalloc_node_track_caller+0x1cc/0x210
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.454399] [<ffffffff8160c1a7>] ? __alloc_skb+0x87/0x2a0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.454401] [<ffffffff8160bfd5>] __kmalloc_reserve.isra.56+0x95/0xa0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.454403] [<ffffffff8160c1a7>] __alloc_skb+0x87/0x2a0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.454408] [<ffffffff81668901>] sk_stream_alloc_skb+0x41/0x130
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.454411] [<ffffffff8166945c>] tcp_sendmsg+0x3ac/0xdc0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.454414] [<ffffffff81697325>] ? inet_sendmsg+0x5/0x110
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.454417] [<ffffffff816973ef>] inet_sendmsg+0xcf/0x110
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.454419] [<ffffffff81697325>] ? inet_sendmsg+0x5/0x110
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.454423] [<ffffffff8160364f>] sock_sendmsg+0xaf/0xc0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.454426] [<ffffffff817336e9>] ? _raw_spin_unlock_bh+0x39/0x40
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.454429] [<ffffffff81669181>] ? tcp_sendmsg+0xd1/0xdc0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.454433] [<ffffffff816036a0>] kernel_sendmsg+0x40/0x60
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.454436] [<ffffffff81605e7b>] sock_no_sendpage+0x9b/0xb0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.454439] [<ffffffff81668a47>] tcp_sendpage+0x57/0x680
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.454441] [<ffffffff81697435>] ? inet_sendpage+0x5/0x130
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.454445] [<ffffffff810c3fe8>] ? rcu_irq_exit+0x68/0xb0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.454447] [<ffffffff81697512>] inet_sendpage+0xe2/0x130
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.454449] [<ffffffff81697435>] ? inet_sendpage+0x5/0x130
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.454452] [<ffffffff816006fb>] kernel_sendpage+0x1b/0x30
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.454459] [<ffffffffa05119ab>] ceph_tcp_sendpage+0x5b/0x110 [libceph]
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.454466] [<ffffffffa0512c28>] con_work+0xb08/0x2c10 [libceph]
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.454469] [<ffffffff8107e46f>] ? finish_task_switch+0x3f/0x120
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.454472] [<ffffffff8107e46f>] ? finish_task_switch+0x3f/0x120
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.454475] [<ffffffff810723f3>] ? process_one_work+0x143/0x500
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.454478] [<ffffffff81072301>] ? process_one_work+0x51/0x500
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.454481] [<ffffffff81072455>] process_one_work+0x1a5/0x500
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.454484] [<ffffffff810723f3>] ? process_one_work+0x143/0x500
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.454488] [<ffffffff81072b9c>] worker_thread+0x11c/0x4b0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.454492] [<ffffffff81072a80>] ? rescuer_thread+0x290/0x290
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.454495] [<ffffffff81078a34>] kthread+0xe4/0x100
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.454498] [<ffffffff81078950>] ? flush_kthread_worker+0xf0/0xf0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.454499] [<ffffffff81733e2c>] ret_from_fork+0x7c/0xb0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.454502] [<ffffffff81078950>] ? flush_kthread_worker+0xf0/0xf0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.454503] Mem-Info:
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.454505] Node 0 DMA per-cpu:
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.454506] CPU 0: hi: 0, btch: 1 usd: 0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.454507] CPU 1: hi: 0, btch: 1 usd: 0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.454509] CPU 2: hi: 0, btch: 1 usd: 0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.454510] CPU 3: hi: 0, btch: 1 usd: 0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.454510] CPU 4: hi: 0, btch: 1 usd: 0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.454511] CPU 5: hi: 0, btch: 1 usd: 0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.454513] CPU 6: hi: 0, btch: 1 usd: 0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.454514] CPU 7: hi: 0, btch: 1 usd: 0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.454515] CPU 8: hi: 0, btch: 1 usd: 0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.454515] CPU 9: hi: 0, btch: 1 usd: 0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.454517] CPU 10: hi: 0, btch: 1 usd: 0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.454518] CPU 11: hi: 0, btch: 1 usd: 0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.454519] Node 0 DMA32 per-cpu:
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.454521] CPU 0: hi: 186, btch: 31 usd: 0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.454522] CPU 1: hi: 186, btch: 31 usd: 0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.454523] CPU 2: hi: 186, btch: 31 usd: 0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.454524] CPU 3: hi: 186, btch: 31 usd: 0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.454525] CPU 4: hi: 186, btch: 31 usd: 0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.454526] CPU 5: hi: 186, btch: 31 usd: 0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.454527] CPU 6: hi: 186, btch: 31 usd: 0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.454529] CPU 7: hi: 186, btch: 31 usd: 0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.454530] CPU 8: hi: 186, btch: 31 usd: 0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.454530] CPU 9: hi: 186, btch: 31 usd: 0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.454531] CPU 10: hi: 186, btch: 31 usd: 0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.454533] CPU 11: hi: 186, btch: 31 usd: 0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.454534] Node 0 Normal per-cpu:
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.454535] CPU 0: hi: 186, btch: 31 usd: 0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.454537] CPU 1: hi: 186, btch: 31 usd: 0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.454537] CPU 2: hi: 186, btch: 31 usd: 0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.454538] CPU 3: hi: 186, btch: 31 usd: 0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.454539] CPU 4: hi: 186, btch: 31 usd: 0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.454541] CPU 5: hi: 186, btch: 31 usd: 0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.454542] CPU 6: hi: 186, btch: 31 usd: 0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.454542] CPU 7: hi: 186, btch: 31 usd: 0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.454544] CPU 8: hi: 186, btch: 31 usd: 18
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.454545] CPU 9: hi: 186, btch: 31 usd: 17
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.454546] CPU 10: hi: 186, btch: 31 usd: 0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.454548] CPU 11: hi: 186, btch: 31 usd: 0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.454549] Node 1 Normal per-cpu:
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.454551] CPU 0: hi: 186, btch: 31 usd: 0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.454552] CPU 1: hi: 186, btch: 31 usd: 0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.454552] CPU 2: hi: 186, btch: 31 usd: 0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.454553] CPU 3: hi: 186, btch: 31 usd: 0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.454555] CPU 4: hi: 186, btch: 31 usd: 0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.454556] CPU 5: hi: 186, btch: 31 usd: 0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.454557] CPU 6: hi: 186, btch: 31 usd: 0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.454558] CPU 7: hi: 186, btch: 31 usd: 0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.454559] CPU 8: hi: 186, btch: 31 usd: 0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.454560] CPU 9: hi: 186, btch: 31 usd: 0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.454562] CPU 10: hi: 186, btch: 31 usd: 0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.454563] CPU 11: hi: 186, btch: 31 usd: 0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.454567] active_anon:2196174 inactive_anon:394 isolated_anon:0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.454567] active_file:195320 inactive_file:3309652 isolated_file:187
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.454567] unevictable:0 dirty:233742 writeback:328941 unstable:0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.454567] free:77198 slab_reclaimable:124331 slab_unreclaimable:52458
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.454567] mapped:17398 shmem:438 pagetables:8773 bounce:0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.454567] free_cma:0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.454573] Node 0 DMA free:112kB min:56kB low:68kB high:84kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15956kB managed:15872kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.454576] lowmem_reserve[]: 0 1907 15985 15985
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.454582] Node 0 DMA32 free:76784kB min:7140kB low:8924kB high:10708kB active_anon:134756kB inactive_anon:36kB active_file:17532kB inactive_file:1624252kB unevictable:0kB isolated(anon):0kB isolated(file):620kB present:2047564kB managed:1955656kB mlocked:0kB dirty:87680kB writeback:150892kB mapped:44kB shmem:36kB slab_reclaimable:66664kB slab_unreclaimable:13552kB kernel_stack:5904kB pagetables:200kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.454585] lowmem_reserve[]: 0 0 14078 14078
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.454590] Node 0 Normal free:147500kB min:52716kB low:65892kB high:79072kB active_anon:4092552kB inactive_anon:1344kB active_file:441772kB inactive_file:9201572kB unevictable:0kB isolated(anon):0kB isolated(file):128kB present:14680064kB managed:14416344kB mlocked:0kB dirty:605856kB writeback:851400kB mapped:41708kB shmem:1456kB slab_reclaimable:210784kB slab_unreclaimable:81980kB kernel_stack:23856kB pagetables:16060kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.454593] lowmem_reserve[]: 0 0 0 0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.454598] Node 1 Normal free:84396kB min:30192kB low:37740kB high:45288kB active_anon:4557388kB inactive_anon:196kB active_file:321976kB inactive_file:2412784kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:8388608kB managed:8256116kB mlocked:0kB dirty:241432kB writeback:313472kB mapped:27840kB shmem:260kB slab_reclaimable:219876kB slab_unreclaimable:114284kB kernel_stack:30624kB pagetables:18832kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.454601] lowmem_reserve[]: 0 0 0 0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.454611] Node 0 DMA: 0*4kB 0*8kB 1*16kB (U) 1*32kB (U) 1*64kB (U) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 112kB
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.454622] Node 0 DMA32: 1826*4kB (UEM) 2194*8kB (UEM) 858*16kB (UEM) 662*32kB (UEM) 255*64kB (UEM) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 76088kB
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.454632] Node 0 Normal: 12357*4kB (UEM) 3244*8kB (UEM) 2531*16kB (UEM) 876*32kB (UM) 39*64kB (UEM) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 146404kB
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.454642] Node 1 Normal: 14454*4kB (UEM) 1033*8kB (UEM) 869*16kB (UEM) 173*32kB (EMR) 1*64kB (R) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 85584kB
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.454644] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.454645] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.454646] 3506090 total pagecache pages
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.454647] 0 pages in swap cache
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.454648] Swap cache stats: add 0, delete 0, find 0/0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.454649] Free swap = 0kB
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.454649] Total swap = 0kB
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.454651] 6283048 pages RAM
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.454651] 0 pages HighMem/MovableOnly
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.454652] 33123 pages reserved
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.454652] 0 pages hwpoisoned
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.454899] CPU: 2 PID: 4344 Comm: kworker/2:1 Tainted: G E 3.17.4-ceph-00638-g0f25ebb #1
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.454905] Hardware name: Supermicro X9DRH-7TF/7F/iTF/iF/X9DRH-7TF/7F/iTF/iF, BIOS 3.0b 04/28/2014
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.454919] Workqueue: ceph-msgr con_work [libceph]
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.454924] 0000000000000005 ffff88006d6fb478 ffffffff8172a4cf ffff88046fc8f258
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.454932] 0000000000206050 ffff88006d6fb508 ffffffff8116483e 0000000000000000
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.454940] 0000000400000000 ffff88047fffcf48 0000001fffffffe0 0000000000000010
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.454948] Call Trace:
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.454954] [<ffffffff8172a4cf>] dump_stack+0x4e/0x68
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.454961] [<ffffffff8116483e>] warn_alloc_failed+0xee/0x140
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.454967] [<ffffffff8116908b>] __alloc_pages_nodemask+0x96b/0xb70
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.454974] [<ffffffff811692d9>] alloc_kmem_pages_node+0x49/0x60
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.454981] [<ffffffff8172735f>] kmalloc_large_node+0x28/0x56
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.454987] [<ffffffff811b8fbc>] __kmalloc_node_track_caller+0x1cc/0x210
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.454994] [<ffffffff8160c1a7>] ? __alloc_skb+0x87/0x2a0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.455000] [<ffffffff8160bfd5>] __kmalloc_reserve.isra.56+0x95/0xa0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.455005] [<ffffffff8160c1a7>] __alloc_skb+0x87/0x2a0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.455012] [<ffffffff81668901>] sk_stream_alloc_skb+0x41/0x130
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.455018] [<ffffffff8166945c>] tcp_sendmsg+0x3ac/0xdc0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.455024] [<ffffffff81697325>] ? inet_sendmsg+0x5/0x110
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.455030] [<ffffffff816973ef>] inet_sendmsg+0xcf/0x110
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.455036] [<ffffffff81697325>] ? inet_sendmsg+0x5/0x110
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.455042] [<ffffffff8160364f>] sock_sendmsg+0xaf/0xc0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.455049] [<ffffffff817336e9>] ? _raw_spin_unlock_bh+0x39/0x40
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.455055] [<ffffffff81669181>] ? tcp_sendmsg+0xd1/0xdc0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.455061] [<ffffffff816036a0>] kernel_sendmsg+0x40/0x60
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.455067] [<ffffffff81605e7b>] sock_no_sendpage+0x9b/0xb0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.455073] [<ffffffff81668a47>] tcp_sendpage+0x57/0x680
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.455079] [<ffffffff81697435>] ? inet_sendpage+0x5/0x130
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.455088] [<ffffffffa0512175>] ? con_work+0x55/0x2c10 [libceph]
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.455095] [<ffffffff81697512>] inet_sendpage+0xe2/0x130
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.455100] [<ffffffff81697435>] ? inet_sendpage+0x5/0x130
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.455106] [<ffffffff816006fb>] kernel_sendpage+0x1b/0x30
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.455116] [<ffffffffa05119ab>] ceph_tcp_sendpage+0x5b/0x110 [libceph]
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.455126] [<ffffffffa0512c28>] con_work+0xb08/0x2c10 [libceph]
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.455132] [<ffffffff8107e46f>] ? finish_task_switch+0x3f/0x120
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.455138] [<ffffffff8107e46f>] ? finish_task_switch+0x3f/0x120
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.455145] [<ffffffff810723f3>] ? process_one_work+0x143/0x500
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.455151] [<ffffffff81072301>] ? process_one_work+0x51/0x500
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.455158] [<ffffffff81072455>] process_one_work+0x1a5/0x500
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.455164] [<ffffffff810723f3>] ? process_one_work+0x143/0x500
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.455171] [<ffffffff81072b9c>] worker_thread+0x11c/0x4b0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.455177] [<ffffffff81072a80>] ? rescuer_thread+0x290/0x290
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.455183] [<ffffffff81078a34>] kthread+0xe4/0x100
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.455189] [<ffffffff81078950>] ? flush_kthread_worker+0xf0/0xf0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.455195] [<ffffffff81733e2c>] ret_from_fork+0x7c/0xb0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.455200] [<ffffffff81078950>] ? flush_kthread_worker+0xf0/0xf0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.455205] Mem-Info:
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.455208] Node 0 DMA per-cpu:
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.455213] CPU 0: hi: 0, btch: 1 usd: 0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.455217] CPU 1: hi: 0, btch: 1 usd: 0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.455221] CPU 2: hi: 0, btch: 1 usd: 0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.455225] CPU 3: hi: 0, btch: 1 usd: 0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.455229] CPU 4: hi: 0, btch: 1 usd: 0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.455233] CPU 5: hi: 0, btch: 1 usd: 0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.455236] CPU 6: hi: 0, btch: 1 usd: 0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.455240] CPU 7: hi: 0, btch: 1 usd: 0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.455244] CPU 8: hi: 0, btch: 1 usd: 0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.455248] CPU 9: hi: 0, btch: 1 usd: 0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.455252] CPU 10: hi: 0, btch: 1 usd: 0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.455256] CPU 11: hi: 0, btch: 1 usd: 0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.455260] Node 0 DMA32 per-cpu:
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.455264] CPU 0: hi: 186, btch: 31 usd: 0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.455268] CPU 1: hi: 186, btch: 31 usd: 0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.455272] CPU 2: hi: 186, btch: 31 usd: 0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.455276] CPU 3: hi: 186, btch: 31 usd: 0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.455280] CPU 4: hi: 186, btch: 31 usd: 0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.455284] CPU 5: hi: 186, btch: 31 usd: 0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.455288] CPU 6: hi: 186, btch: 31 usd: 0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.455292] CPU 7: hi: 186, btch: 31 usd: 0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.455296] CPU 8: hi: 186, btch: 31 usd: 0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.455300] CPU 9: hi: 186, btch: 31 usd: 0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.455304] CPU 10: hi: 186, btch: 31 usd: 0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.455308] CPU 11: hi: 186, btch: 31 usd: 0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.455312] Node 0 Normal per-cpu:
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.455316] CPU 0: hi: 186, btch: 31 usd: 0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.455320] CPU 1: hi: 186, btch: 31 usd: 0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.455324] CPU 2: hi: 186, btch: 31 usd: 0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.455328] CPU 3: hi: 186, btch: 31 usd: 0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.455332] CPU 4: hi: 186, btch: 31 usd: 0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.455336] CPU 5: hi: 186, btch: 31 usd: 0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.455340] CPU 6: hi: 186, btch: 31 usd: 0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.455344] CPU 7: hi: 186, btch: 31 usd: 0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.455348] CPU 8: hi: 186, btch: 31 usd: 13
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.455352] CPU 9: hi: 186, btch: 31 usd: 14
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.455356] CPU 10: hi: 186, btch: 31 usd: 0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.455360] CPU 11: hi: 186, btch: 31 usd: 0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.455364] Node 1 Normal per-cpu:
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.455368] CPU 0: hi: 186, btch: 31 usd: 0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.455372] CPU 1: hi: 186, btch: 31 usd: 0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.455376] CPU 2: hi: 186, btch: 31 usd: 0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.455380] CPU 3: hi: 186, btch: 31 usd: 0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.455384] CPU 4: hi: 186, btch: 31 usd: 0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.455388] CPU 5: hi: 186, btch: 31 usd: 0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.455392] CPU 6: hi: 186, btch: 31 usd: 0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.455396] CPU 7: hi: 186, btch: 31 usd: 0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.455400] CPU 8: hi: 186, btch: 31 usd: 0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.455404] CPU 9: hi: 186, btch: 31 usd: 0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.455408] CPU 10: hi: 186, btch: 31 usd: 0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.455412] CPU 11: hi: 186, btch: 31 usd: 0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.455419] active_anon:2196174 inactive_anon:394 isolated_anon:0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.455419] active_file:195320 inactive_file:3310156 isolated_file:187
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.455419] unevictable:0 dirty:234118 writeback:328941 unstable:0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.455419] free:76896 slab_reclaimable:124331 slab_unreclaimable:52458
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.455419] mapped:17398 shmem:438 pagetables:8773 bounce:0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.455419] free_cma:0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.455438] Node 0 DMA free:112kB min:56kB low:68kB high:84kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15956kB managed:15872kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.455461] lowmem_reserve[]: 0 1907 15985 15985
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.455469] Node 0 DMA32 free:76784kB min:7140kB low:8924kB high:10708kB active_anon:134756kB inactive_anon:36kB active_file:17532kB inactive_file:1624252kB unevictable:0kB isolated(anon):0kB isolated(file):620kB present:2047564kB managed:1955656kB mlocked:0kB dirty:87680kB writeback:150892kB mapped:44kB shmem:36kB slab_reclaimable:66664kB slab_unreclaimable:13552kB kernel_stack:5904kB pagetables:200kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.455495] lowmem_reserve[]: 0 0 14078 14078
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.455501] Node 0 Normal free:146292kB min:52716kB low:65892kB high:79072kB active_anon:4092552kB inactive_anon:1344kB active_file:441772kB inactive_file:9203588kB unevictable:0kB isolated(anon):0kB isolated(file):128kB present:14680064kB managed:14416344kB mlocked:0kB dirty:607360kB writeback:851400kB mapped:41708kB shmem:1456kB slab_reclaimable:210784kB slab_unreclaimable:81980kB kernel_stack:23856kB pagetables:16060kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.455527] lowmem_reserve[]: 0 0 0 0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.455534] Node 1 Normal free:84396kB min:30192kB low:37740kB high:45288kB active_anon:4557388kB inactive_anon:196kB active_file:321976kB inactive_file:2412784kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:8388608kB managed:8256116kB mlocked:0kB dirty:241432kB writeback:313472kB mapped:27840kB shmem:260kB slab_reclaimable:219876kB slab_unreclaimable:114284kB kernel_stack:30624kB pagetables:18832kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.455559] lowmem_reserve[]: 0 0 0 0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.455565] Node 0 DMA: 0*4kB 0*8kB 1*16kB (U) 1*32kB (U) 1*64kB (U) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 112kB
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.455585] Node 0 DMA32: 1826*4kB (UEM) 2194*8kB (UEM) 858*16kB (UEM) 662*32kB (UEM) 255*64kB (UEM) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 76088kB
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.455608] Node 0 Normal: 12357*4kB (UEM) 3027*8kB (UEM) 2531*16kB (UEM) 876*32kB (UM) 39*64kB (UEM) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 144668kB
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.455633] Node 1 Normal: 14454*4kB (UEM) 1033*8kB (UEM) 868*16kB (UEM) 173*32kB (EMR) 1*64kB (R) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 85568kB
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.455658] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.455664] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.455670] 3506654 total pagecache pages
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.455674] 0 pages in swap cache
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.455677] Swap cache stats: add 0, delete 0, find 0/0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.455681] Free swap = 0kB
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.455684] Total swap = 0kB
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.455688] 6283048 pages RAM
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.455691] 0 pages HighMem/MovableOnly
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.455694] 33123 pages reserved
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.455697] 0 pages hwpoisoned
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.473525] kworker/5:4: page allocation failure: order:5, mode:0x206050
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.473538] CPU: 5 PID: 12213 Comm: kworker/5:4 Tainted: G E 3.17.4-ceph-00638-g0f25ebb #1
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.473545] Hardware name: Supermicro X9DRH-7TF/7F/iTF/iF/X9DRH-7TF/7F/iTF/iF, BIOS 3.0b 04/28/2014
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.473564] Workqueue: ceph-msgr con_work [libceph]
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.473570] 0000000000000005 ffff88049d4e7478 ffffffff8172a4cf ffff88046fd4f258
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.473578] 0000000000206050 ffff88049d4e7508 ffffffff8116483e 0000000000000000
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.473586] 0000000400000000 ffff88047fffcf48 0000001fffffffe0 0000000000000010
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.473594] Call Trace:
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.473602] [<ffffffff8172a4cf>] dump_stack+0x4e/0x68
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.473610] [<ffffffff8116483e>] warn_alloc_failed+0xee/0x140
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.473617] [<ffffffff8116908b>] __alloc_pages_nodemask+0x96b/0xb70
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.473624] [<ffffffff811692d9>] alloc_kmem_pages_node+0x49/0x60
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.473631] [<ffffffff8172735f>] kmalloc_large_node+0x28/0x56
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.473638] [<ffffffff811b8fbc>] __kmalloc_node_track_caller+0x1cc/0x210
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.473646] [<ffffffff8160c1a7>] ? __alloc_skb+0x87/0x2a0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.473653] [<ffffffff81605d63>] ? lock_sock_nested+0x43/0xc0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.473659] [<ffffffff8160bfd5>] __kmalloc_reserve.isra.56+0x95/0xa0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.473664] [<ffffffff8160c1a7>] __alloc_skb+0x87/0x2a0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.473672] [<ffffffff81668901>] sk_stream_alloc_skb+0x41/0x130
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.473678] [<ffffffff8166945c>] tcp_sendmsg+0x3ac/0xdc0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.473685] [<ffffffff81697325>] ? inet_sendmsg+0x5/0x110
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.473691] [<ffffffff816973ef>] inet_sendmsg+0xcf/0x110
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.473697] [<ffffffff81697325>] ? inet_sendmsg+0x5/0x110
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.473703] [<ffffffff8160364f>] sock_sendmsg+0xaf/0xc0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.473710] [<ffffffff817336e9>] ? _raw_spin_unlock_bh+0x39/0x40
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.473717] [<ffffffff81669181>] ? tcp_sendmsg+0xd1/0xdc0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.473723] [<ffffffff816036a0>] kernel_sendmsg+0x40/0x60
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.473729] [<ffffffff81605e7b>] sock_no_sendpage+0x9b/0xb0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.473735] [<ffffffff81668a47>] tcp_sendpage+0x57/0x680
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.473740] [<ffffffff81697435>] ? inet_sendpage+0x5/0x130
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.473750] [<ffffffffa0512175>] ? con_work+0x55/0x2c10 [libceph]
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.473757] [<ffffffff81697512>] inet_sendpage+0xe2/0x130
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.473762] [<ffffffff81697435>] ? inet_sendpage+0x5/0x130
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.473768] [<ffffffff816006fb>] kernel_sendpage+0x1b/0x30
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.473778] [<ffffffffa05119ab>] ceph_tcp_sendpage+0x5b/0x110 [libceph]
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.473788] [<ffffffffa0512c28>] con_work+0xb08/0x2c10 [libceph]
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.473796] [<ffffffff8107e46f>] ? finish_task_switch+0x3f/0x120
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.473801] [<ffffffff8107e46f>] ? finish_task_switch+0x3f/0x120
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.473810] [<ffffffff810723f3>] ? process_one_work+0x143/0x500
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.473816] [<ffffffff81072301>] ? process_one_work+0x51/0x500
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.473822] [<ffffffff81072455>] process_one_work+0x1a5/0x500
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.473828] [<ffffffff810723f3>] ? process_one_work+0x143/0x500
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.473836] [<ffffffff81072b9c>] worker_thread+0x11c/0x4b0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.473842] [<ffffffff81072a80>] ? rescuer_thread+0x290/0x290
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.473848] [<ffffffff81078a34>] kthread+0xe4/0x100
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.473854] [<ffffffff81078950>] ? flush_kthread_worker+0xf0/0xf0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.473860] [<ffffffff81733e2c>] ret_from_fork+0x7c/0xb0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.473866] [<ffffffff81078950>] ? flush_kthread_worker+0xf0/0xf0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.473870] Mem-Info:
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.473873] Node 0 DMA per-cpu:
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.473878] CPU 0: hi: 0, btch: 1 usd: 0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.473882] CPU 1: hi: 0, btch: 1 usd: 0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.473886] CPU 2: hi: 0, btch: 1 usd: 0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.473890] CPU 3: hi: 0, btch: 1 usd: 0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.473894] CPU 4: hi: 0, btch: 1 usd: 0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.473898] CPU 5: hi: 0, btch: 1 usd: 0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.473902] CPU 6: hi: 0, btch: 1 usd: 0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.473906] CPU 7: hi: 0, btch: 1 usd: 0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.473910] CPU 8: hi: 0, btch: 1 usd: 0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.473914] CPU 9: hi: 0, btch: 1 usd: 0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.473918] CPU 10: hi: 0, btch: 1 usd: 0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.473922] CPU 11: hi: 0, btch: 1 usd: 0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.473925] Node 0 DMA32 per-cpu:
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.473930] CPU 0: hi: 186, btch: 31 usd: 0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.473934] CPU 1: hi: 186, btch: 31 usd: 0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.473938] CPU 2: hi: 186, btch: 31 usd: 0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.473942] CPU 3: hi: 186, btch: 31 usd: 0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.473946] CPU 4: hi: 186, btch: 31 usd: 0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.473950] CPU 5: hi: 186, btch: 31 usd: 0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.473954] CPU 6: hi: 186, btch: 31 usd: 0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.473958] CPU 7: hi: 186, btch: 31 usd: 0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.473962] CPU 8: hi: 186, btch: 31 usd: 0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.473965] CPU 9: hi: 186, btch: 31 usd: 0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.473970] CPU 10: hi: 186, btch: 31 usd: 0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.473973] CPU 11: hi: 186, btch: 31 usd: 0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.473977] Node 0 Normal per-cpu:
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.473981] CPU 0: hi: 186, btch: 31 usd: 0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.473985] CPU 1: hi: 186, btch: 31 usd: 0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.473989] CPU 2: hi: 186, btch: 31 usd: 0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.473993] CPU 3: hi: 186, btch: 31 usd: 30
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.473997] CPU 4: hi: 186, btch: 31 usd: 0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.474001] CPU 5: hi: 186, btch: 31 usd: 0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.474005] CPU 6: hi: 186, btch: 31 usd: 0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.474009] CPU 7: hi: 186, btch: 31 usd: 0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.474013] CPU 8: hi: 186, btch: 31 usd: 0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.474017] CPU 9: hi: 186, btch: 31 usd: 0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.474021] CPU 10: hi: 186, btch: 31 usd: 0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.474025] CPU 11: hi: 186, btch: 31 usd: 0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.474029] Node 1 Normal per-cpu:
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.474033] CPU 0: hi: 186, btch: 31 usd: 0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.474037] CPU 1: hi: 186, btch: 31 usd: 0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.474041] CPU 2: hi: 186, btch: 31 usd: 0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.474045] CPU 3: hi: 186, btch: 31 usd: 0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.474049] CPU 4: hi: 186, btch: 31 usd: 0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.474053] CPU 5: hi: 186, btch: 31 usd: 0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.474056] CPU 6: hi: 186, btch: 31 usd: 0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.474060] CPU 7: hi: 186, btch: 31 usd: 0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.474064] CPU 8: hi: 186, btch: 31 usd: 0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.474068] CPU 9: hi: 186, btch: 31 usd: 0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.474072] CPU 10: hi: 186, btch: 31 usd: 0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.474076] CPU 11: hi: 186, btch: 31 usd: 0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.474083] active_anon:2196174 inactive_anon:394 isolated_anon:0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.474083] active_file:195320 inactive_file:3310363 isolated_file:187
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.474083] unevictable:0 dirty:219742 writeback:342697 unstable:0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.474083] free:76374 slab_reclaimable:124331 slab_unreclaimable:52458
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.474083] mapped:17398 shmem:438 pagetables:8773 bounce:0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.474083] free_cma:0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.474102] Node 0 DMA free:112kB min:56kB low:68kB high:84kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15956kB managed:15872kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.474125] lowmem_reserve[]: 0 1907 15985 15985
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.474133] Node 0 DMA32 free:76600kB min:7140kB low:8924kB high:10708kB active_anon:134756kB inactive_anon:36kB active_file:17532kB inactive_file:1624252kB unevictable:0kB isolated(anon):0kB isolated(file):620kB present:2047564kB managed:1955656kB mlocked:0kB dirty:84320kB writeback:154028kB mapped:44kB shmem:36kB slab_reclaimable:66664kB slab_unreclaimable:13552kB kernel_stack:5904kB pagetables:200kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.474158] lowmem_reserve[]: 0 0 14078 14078
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.474165] Node 0 Normal free:145452kB min:52716kB low:65892kB high:79072kB active_anon:4092552kB inactive_anon:1344kB active_file:441772kB inactive_file:9204416kB unevictable:0kB isolated(anon):0kB isolated(file):128kB present:14680064kB managed:14416344kB mlocked:0kB dirty:574272kB writeback:882232kB mapped:41708kB shmem:1456kB slab_reclaimable:210784kB slab_unreclaimable:81980kB kernel_stack:23856kB pagetables:16060kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.474191] lowmem_reserve[]: 0 0 0 0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.474197] Node 1 Normal free:83332kB min:30192kB low:37740kB high:45288kB active_anon:4557388kB inactive_anon:196kB active_file:321976kB inactive_file:2412784kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:8388608kB managed:8256116kB mlocked:0kB dirty:220376kB writeback:334528kB mapped:27840kB shmem:260kB slab_reclaimable:219876kB slab_unreclaimable:114284kB kernel_stack:30624kB pagetables:18832kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.474223] lowmem_reserve[]: 0 0 0 0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.474229] Node 0 DMA: 0*4kB 0*8kB 1*16kB (U) 1*32kB (U) 1*64kB (U) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 112kB
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.474249] Node 0 DMA32: 1826*4kB (UEM) 2195*8kB (UEM) 858*16kB (UEM) 662*32kB (UEM) 257*64kB (UEM) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 76224kB
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.474273] Node 0 Normal: 12379*4kB (UEM) 2944*8kB (UEM) 2535*16kB (UEM) 890*32kB (UM) 39*64kB (UEM) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 144604kB
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.474295] Node 1 Normal: 14244*4kB (UEM) 1033*8kB (UEM) 868*16kB (UEM) 173*32kB (EMR) 1*64kB (R) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 84728kB
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.474319] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.474325] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.474331] 3506654 total pagecache pages
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.474335] 0 pages in swap cache
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.474339] Swap cache stats: add 0, delete 0, find 0/0
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.474342] Free swap = 0kB
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.474345] Total swap = 0kB
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.474349] 6283048 pages RAM
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.474352] 0 pages HighMem/MovableOnly
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.474355] 33123 pages reserved
Nov 29 12:08:26 arh-ibstorage1-ib kernel: [ 3880.474358] 0 pages hwpoisoned
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.071004] kworker/6:4: page allocation failure: order:5, mode:0x206050
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.071021] CPU: 6 PID: 9672 Comm: kworker/6:4 Tainted: G E 3.17.4-ceph-00638-g0f25ebb #1
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.071028] Hardware name: Supermicro X9DRH-7TF/7F/iTF/iF/X9DRH-7TF/7F/iTF/iF, BIOS 3.0b 04/28/2014
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.071047] Workqueue: ceph-msgr con_work [libceph]
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.071054] 0000000000000005 ffff880003b27478 ffffffff8172a4cf ffff88067fc0f258
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.071062] 0000000000206050 ffff880003b27508 ffffffff8116483e 0000000000000000
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.071070] 0000000400000000 ffff88067fff9f48 0000001fffffffe0 0000000000000010
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.071079] Call Trace:
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.071089] [<ffffffff8172a4cf>] dump_stack+0x4e/0x68
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.071098] [<ffffffff8116483e>] warn_alloc_failed+0xee/0x140
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.071104] [<ffffffff8116908b>] __alloc_pages_nodemask+0x96b/0xb70
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.071111] [<ffffffff811692d9>] alloc_kmem_pages_node+0x49/0x60
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.071119] [<ffffffff8172735f>] kmalloc_large_node+0x28/0x56
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.071127] [<ffffffff811b8fbc>] __kmalloc_node_track_caller+0x1cc/0x210
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.071136] [<ffffffff8160c1a7>] ? __alloc_skb+0x87/0x2a0
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.071144] [<ffffffff81605d63>] ? lock_sock_nested+0x43/0xc0
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.071150] [<ffffffff8160bfd5>] __kmalloc_reserve.isra.56+0x95/0xa0
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.071156] [<ffffffff8160c1a7>] __alloc_skb+0x87/0x2a0
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.071165] [<ffffffff81668901>] sk_stream_alloc_skb+0x41/0x130
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.071172] [<ffffffff8166945c>] tcp_sendmsg+0x3ac/0xdc0
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.071179] [<ffffffff81697325>] ? inet_sendmsg+0x5/0x110
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.071185] [<ffffffff816973ef>] inet_sendmsg+0xcf/0x110
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.071190] [<ffffffff81697325>] ? inet_sendmsg+0x5/0x110
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.071199] [<ffffffff810a738d>] ? trace_hardirqs_on+0xd/0x10
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.071206] [<ffffffff8160364f>] sock_sendmsg+0xaf/0xc0
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.071213] [<ffffffff81607e73>] ? release_sock+0x193/0x1f0
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.071218] [<ffffffff8166aea0>] ? tcp_recvmsg+0x620/0xcb0
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.071224] [<ffffffff81697215>] ? inet_recvmsg+0x5/0x110
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.071230] [<ffffffff816036a0>] kernel_sendmsg+0x40/0x60
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.071236] [<ffffffff81605e7b>] sock_no_sendpage+0x9b/0xb0
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.071243] [<ffffffff81668a47>] tcp_sendpage+0x57/0x680
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.071248] [<ffffffff81697435>] ? inet_sendpage+0x5/0x130
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.071255] [<ffffffff81734c60>] ? retint_restore_args+0xe/0xe
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.071261] [<ffffffff81697512>] inet_sendpage+0xe2/0x130
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.071267] [<ffffffff81697435>] ? inet_sendpage+0x5/0x130
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.071273] [<ffffffff816006fb>] kernel_sendpage+0x1b/0x30
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.071284] [<ffffffffa05119ab>] ceph_tcp_sendpage+0x5b/0x110 [libceph]
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.071294] [<ffffffffa0512c28>] con_work+0xb08/0x2c10 [libceph]
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.071302] [<ffffffff8107e46f>] ? finish_task_switch+0x3f/0x120
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.071311] [<ffffffff810724cc>] ? process_one_work+0x21c/0x500
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.071317] [<ffffffff810723f3>] ? process_one_work+0x143/0x500
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.071324] [<ffffffff81072301>] ? process_one_work+0x51/0x500
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.071330] [<ffffffff81072455>] process_one_work+0x1a5/0x500
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.071336] [<ffffffff810723f3>] ? process_one_work+0x143/0x500
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.071344] [<ffffffff81072b9c>] worker_thread+0x11c/0x4b0
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.071350] [<ffffffff81072a80>] ? rescuer_thread+0x290/0x290
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.071356] [<ffffffff81078a34>] kthread+0xe4/0x100
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.071363] [<ffffffff81078950>] ? flush_kthread_worker+0xf0/0xf0
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.071369] [<ffffffff81733e2c>] ret_from_fork+0x7c/0xb0
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.071374] [<ffffffff81078950>] ? flush_kthread_worker+0xf0/0xf0
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.071379] Mem-Info:
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.071382] Node 0 DMA per-cpu:
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.071387] CPU 0: hi: 0, btch: 1 usd: 0
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.071391] CPU 1: hi: 0, btch: 1 usd: 0
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.071396] CPU 2: hi: 0, btch: 1 usd: 0
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.071400] CPU 3: hi: 0, btch: 1 usd: 0
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.071404] CPU 4: hi: 0, btch: 1 usd: 0
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.071408] CPU 5: hi: 0, btch: 1 usd: 0
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.071412] CPU 6: hi: 0, btch: 1 usd: 0
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.071416] CPU 7: hi: 0, btch: 1 usd: 0
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.071420] CPU 8: hi: 0, btch: 1 usd: 0
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.071424] CPU 9: hi: 0, btch: 1 usd: 0
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.071428] CPU 10: hi: 0, btch: 1 usd: 0
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.071432] CPU 11: hi: 0, btch: 1 usd: 0
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.071435] Node 0 DMA32 per-cpu:
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.071440] CPU 0: hi: 186, btch: 31 usd: 0
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.071450] CPU 1: hi: 186, btch: 31 usd: 0
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.071454] CPU 2: hi: 186, btch: 31 usd: 0
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.071458] CPU 3: hi: 186, btch: 31 usd: 0
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.071462] CPU 4: hi: 186, btch: 31 usd: 0
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.071469] CPU 5: hi: 186, btch: 31 usd: 0
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.071473] CPU 6: hi: 186, btch: 31 usd: 0
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.071477] CPU 7: hi: 186, btch: 31 usd: 0
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.071481] CPU 8: hi: 186, btch: 31 usd: 0
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.071489] CPU 9: hi: 186, btch: 31 usd: 0
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.071511] CPU 10: hi: 186, btch: 31 usd: 0
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.071515] CPU 11: hi: 186, btch: 31 usd: 0
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.071530] Node 0 Normal per-cpu:
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.071535] CPU 0: hi: 186, btch: 31 usd: 0
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.071539] CPU 1: hi: 186, btch: 31 usd: 0
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.071543] CPU 2: hi: 186, btch: 31 usd: 0
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.071547] CPU 3: hi: 186, btch: 31 usd: 0
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.071551] CPU 4: hi: 186, btch: 31 usd: 155
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.071555] CPU 5: hi: 186, btch: 31 usd: 0
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.071559] CPU 6: hi: 186, btch: 31 usd: 0
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.071563] CPU 7: hi: 186, btch: 31 usd: 0
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.071567] CPU 8: hi: 186, btch: 31 usd: 0
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.071571] CPU 9: hi: 186, btch: 31 usd: 0
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.071591] CPU 10: hi: 186, btch: 31 usd: 0
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.071595] CPU 11: hi: 186, btch: 31 usd: 0
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.071601] Node 1 Normal per-cpu:
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.071605] CPU 0: hi: 186, btch: 31 usd: 0
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.071619] CPU 1: hi: 186, btch: 31 usd: 0
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.071623] CPU 2: hi: 186, btch: 31 usd: 0
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.071627] CPU 3: hi: 186, btch: 31 usd: 0
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.071631] CPU 4: hi: 186, btch: 31 usd: 1
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.071636] CPU 5: hi: 186, btch: 31 usd: 0
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.071640] CPU 6: hi: 186, btch: 31 usd: 0
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.071649] CPU 7: hi: 186, btch: 31 usd: 0
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.071653] CPU 8: hi: 186, btch: 31 usd: 0
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.071657] CPU 9: hi: 186, btch: 31 usd: 0
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.071661] CPU 10: hi: 186, btch: 31 usd: 0
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.071666] CPU 11: hi: 186, btch: 31 usd: 0
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.071732] active_anon:2195326 inactive_anon:394 isolated_anon:0
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.071732] active_file:195787 inactive_file:3303562 isolated_file:137
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.071732] unevictable:0 dirty:223194 writeback:345997 unstable:0
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.071732] free:77840 slab_reclaimable:129269 slab_unreclaimable:52460
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.071732] mapped:17237 shmem:438 pagetables:8773 bounce:0
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.071732] free_cma:0
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.071755] Node 0 DMA free:112kB min:56kB low:68kB high:84kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15956kB managed:15872kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.071786] lowmem_reserve[]: 0 1907 15985 15985
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.071832] Node 0 DMA32 free:77256kB min:7140kB low:8924kB high:10708kB active_anon:134416kB inactive_anon:36kB active_file:17540kB inactive_file:1592960kB unevictable:0kB isolated(anon):0kB isolated(file):548kB present:2047564kB managed:1955656kB mlocked:0kB dirty:86020kB writeback:148912kB mapped:44kB shmem:36kB slab_reclaimable:82696kB slab_unreclaimable:15320kB kernel_stack:5904kB pagetables:200kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.071922] lowmem_reserve[]: 0 0 14078 14078
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.071931] Node 0 Normal free:149840kB min:52716kB low:65892kB high:79072kB active_anon:4105856kB inactive_anon:1344kB active_file:442184kB inactive_file:9195540kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:14680064kB managed:14416344kB mlocked:0kB dirty:592832kB writeback:891468kB mapped:40516kB shmem:1456kB slab_reclaimable:211208kB slab_unreclaimable:81316kB kernel_stack:23888kB pagetables:16060kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:172 all_unreclaimable? no
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.071957] lowmem_reserve[]: 0 0 0 0
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.071964] Node 1 Normal free:84656kB min:30192kB low:37740kB high:45288kB active_anon:4541032kB inactive_anon:196kB active_file:323424kB inactive_file:2425748kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:8388608kB managed:8256116kB mlocked:0kB dirty:213172kB writeback:345112kB mapped:28388kB shmem:260kB slab_reclaimable:223172kB slab_unreclaimable:113188kB kernel_stack:30640kB pagetables:18832kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.071989] lowmem_reserve[]: 0 0 0 0
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.071999] Node 0 DMA: 0*4kB 0*8kB 1*16kB (U) 1*32kB (U) 1*64kB (U) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 112kB
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.072182] Node 0 DMA32: 969*4kB (UEM) 1038*8kB (UEM) 1170*16kB (UM) 576*32kB (UEM) 424*64kB (UEM) 1*128kB (U) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 76596kB
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.072207] Node 0 Normal: 14979*4kB (UE) 2131*8kB (UEM) 4127*16kB (UEM) 150*32kB (UEMR) 40*64kB (UEMR) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 150356kB
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.072231] Node 1 Normal: 8745*4kB (UE) 1644*8kB (UEM) 1904*16kB (UM) 110*32kB (UEMR) 44*64kB (UEMR) 3*128kB (R) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 85316kB
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.072258] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.072279] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.072287] 3500322 total pagecache pages
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.072292] 0 pages in swap cache
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.072296] Swap cache stats: add 0, delete 0, find 0/0
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.072300] Free swap = 0kB
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.072302] Total swap = 0kB
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.072309] 6283048 pages RAM
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.072312] 0 pages HighMem/MovableOnly
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.072315] 33123 pages reserved
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.072318] 0 pages hwpoisoned
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.076149] kworker/6:4: page allocation failure: order:5, mode:0x206050
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.076185] CPU: 6 PID: 9672 Comm: kworker/6:4 Tainted: G E 3.17.4-ceph-00638-g0f25ebb #1
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.076192] Hardware name: Supermicro X9DRH-7TF/7F/iTF/iF/X9DRH-7TF/7F/iTF/iF, BIOS 3.0b 04/28/2014
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.076212] Workqueue: ceph-msgr con_work [libceph]
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.076223] 0000000000000005 ffff880003b27478 ffffffff8172a4cf ffff88067fc0f258
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.076232] 0000000000206050 ffff880003b27508 ffffffff8116483e 0000000000000000
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.076240] 0000000400000000 ffff88067fff9f48 0000001fffffffe0 0000000000000010
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.076249] Call Trace:
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.076259] [<ffffffff8172a4cf>] dump_stack+0x4e/0x68
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.076269] [<ffffffff8116483e>] warn_alloc_failed+0xee/0x140
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.076282] [<ffffffff8116908b>] __alloc_pages_nodemask+0x96b/0xb70
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.076293] [<ffffffff811692d9>] alloc_kmem_pages_node+0x49/0x60
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.076302] [<ffffffff8172735f>] kmalloc_large_node+0x28/0x56
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.076315] [<ffffffff811b8fbc>] __kmalloc_node_track_caller+0x1cc/0x210
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.076324] [<ffffffff8160c1a7>] ? __alloc_skb+0x87/0x2a0
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.076337] [<ffffffff81605d63>] ? lock_sock_nested+0x43/0xc0
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.076343] [<ffffffff8160bfd5>] __kmalloc_reserve.isra.56+0x95/0xa0
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.076354] [<ffffffff8160c1a7>] __alloc_skb+0x87/0x2a0
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.076364] [<ffffffff81668901>] sk_stream_alloc_skb+0x41/0x130
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.076377] [<ffffffff8166945c>] tcp_sendmsg+0x3ac/0xdc0
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.076388] [<ffffffff81697325>] ? inet_sendmsg+0x5/0x110
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.076394] [<ffffffff816973ef>] inet_sendmsg+0xcf/0x110
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.076400] [<ffffffff81697325>] ? inet_sendmsg+0x5/0x110
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.076411] [<ffffffff810a738d>] ? trace_hardirqs_on+0xd/0x10
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.076419] [<ffffffff8160364f>] sock_sendmsg+0xaf/0xc0
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.076429] [<ffffffff81607e73>] ? release_sock+0x193/0x1f0
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.076435] [<ffffffff8166aea0>] ? tcp_recvmsg+0x620/0xcb0
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.076441] [<ffffffff81697215>] ? inet_recvmsg+0x5/0x110
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.076452] [<ffffffff816036a0>] kernel_sendmsg+0x40/0x60
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.076458] [<ffffffff81605e7b>] sock_no_sendpage+0x9b/0xb0
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.076468] [<ffffffff81668a47>] tcp_sendpage+0x57/0x680
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.076474] [<ffffffff81697435>] ? inet_sendpage+0x5/0x130
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.076481] [<ffffffff81734c60>] ? retint_restore_args+0xe/0xe
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.076491] [<ffffffff81697512>] inet_sendpage+0xe2/0x130
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.076497] [<ffffffff81697435>] ? inet_sendpage+0x5/0x130
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.076508] [<ffffffff816006fb>] kernel_sendpage+0x1b/0x30
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.076519] [<ffffffffa05119ab>] ceph_tcp_sendpage+0x5b/0x110 [libceph]
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.076559] [<ffffffffa0000012>] ? crc32c+0x12/0x58 [libcrc32c]
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.076569] [<ffffffffa0512c28>] con_work+0xb08/0x2c10 [libceph]
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.076578] [<ffffffff8107e46f>] ? finish_task_switch+0x3f/0x120
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.076588] [<ffffffff810724cc>] ? process_one_work+0x21c/0x500
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.076595] [<ffffffff810723f3>] ? process_one_work+0x143/0x500
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.076601] [<ffffffff81072301>] ? process_one_work+0x51/0x500
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.076607] [<ffffffff81072455>] process_one_work+0x1a5/0x500
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.076613] [<ffffffff810723f3>] ? process_one_work+0x143/0x500
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.076621] [<ffffffff81072b9c>] worker_thread+0x11c/0x4b0
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.076628] [<ffffffff81072a80>] ? rescuer_thread+0x290/0x290
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.076635] [<ffffffff81078a34>] kthread+0xe4/0x100
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.076641] [<ffffffff81078950>] ? flush_kthread_worker+0xf0/0xf0
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.076648] [<ffffffff81733e2c>] ret_from_fork+0x7c/0xb0
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.076653] [<ffffffff81078950>] ? flush_kthread_worker+0xf0/0xf0
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.076659] Mem-Info:
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.076662] Node 0 DMA per-cpu:
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.076667] CPU 0: hi: 0, btch: 1 usd: 0
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.076671] CPU 1: hi: 0, btch: 1 usd: 0
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.076675] CPU 2: hi: 0, btch: 1 usd: 0
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.076680] CPU 3: hi: 0, btch: 1 usd: 0
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.076684] CPU 4: hi: 0, btch: 1 usd: 0
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.076688] CPU 5: hi: 0, btch: 1 usd: 0
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.076692] CPU 6: hi: 0, btch: 1 usd: 0
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.076696] CPU 7: hi: 0, btch: 1 usd: 0
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.076701] CPU 8: hi: 0, btch: 1 usd: 0
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.076705] CPU 9: hi: 0, btch: 1 usd: 0
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.076709] CPU 10: hi: 0, btch: 1 usd: 0
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.076713] CPU 11: hi: 0, btch: 1 usd: 0
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.076717] Node 0 DMA32 per-cpu:
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.076722] CPU 0: hi: 186, btch: 31 usd: 0
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.076726] CPU 1: hi: 186, btch: 31 usd: 0
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.076730] CPU 2: hi: 186, btch: 31 usd: 0
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.076734] CPU 3: hi: 186, btch: 31 usd: 0
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.076738] CPU 4: hi: 186, btch: 31 usd: 0
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.076742] CPU 5: hi: 186, btch: 31 usd: 0
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.076746] CPU 6: hi: 186, btch: 31 usd: 0
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.076750] CPU 7: hi: 186, btch: 31 usd: 0
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.076754] CPU 8: hi: 186, btch: 31 usd: 0
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.076758] CPU 9: hi: 186, btch: 31 usd: 0
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.076762] CPU 10: hi: 186, btch: 31 usd: 0
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.076766] CPU 11: hi: 186, btch: 31 usd: 0
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.076770] Node 0 Normal per-cpu:
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.076774] CPU 0: hi: 186, btch: 31 usd: 0
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.076778] CPU 1: hi: 186, btch: 31 usd: 0
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.076782] CPU 2: hi: 186, btch: 31 usd: 0
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.076787] CPU 3: hi: 186, btch: 31 usd: 0
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.076791] CPU 4: hi: 186, btch: 31 usd: 48
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.076795] CPU 5: hi: 186, btch: 31 usd: 10
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.076799] CPU 6: hi: 186, btch: 31 usd: 0
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.076803] CPU 7: hi: 186, btch: 31 usd: 0
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.076807] CPU 8: hi: 186, btch: 31 usd: 4
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.076812] CPU 9: hi: 186, btch: 31 usd: 0
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.076816] CPU 10: hi: 186, btch: 31 usd: 0
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.076820] CPU 11: hi: 186, btch: 31 usd: 0
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.076824] Node 1 Normal per-cpu:
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.076828] CPU 0: hi: 186, btch: 31 usd: 0
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.076832] CPU 1: hi: 186, btch: 31 usd: 0
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.076836] CPU 2: hi: 186, btch: 31 usd: 0
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.076840] CPU 3: hi: 186, btch: 31 usd: 0
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.076845] CPU 4: hi: 186, btch: 31 usd: 0
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.076848] CPU 5: hi: 186, btch: 31 usd: 0
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.076852] CPU 6: hi: 186, btch: 31 usd: 0
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.076856] CPU 7: hi: 186, btch: 31 usd: 0
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.076860] CPU 8: hi: 186, btch: 31 usd: 0
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.076864] CPU 9: hi: 186, btch: 31 usd: 0
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.076868] CPU 10: hi: 186, btch: 31 usd: 0
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.076872] CPU 11: hi: 186, btch: 31 usd: 0
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.076879] active_anon:2195326 inactive_anon:394 isolated_anon:0
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.076879] active_file:195787 inactive_file:3303902 isolated_file:137
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.076879] unevictable:0 dirty:221502 writeback:347689 unstable:0
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.076879] free:77730 slab_reclaimable:129269 slab_unreclaimable:52460
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.076879] mapped:17237 shmem:438 pagetables:8773 bounce:0
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.076879] free_cma:0
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.076898] Node 0 DMA free:112kB min:56kB low:68kB high:84kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15956kB managed:15872kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.076923] lowmem_reserve[]: 0 1907 15985 15985
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.076931] Node 0 DMA32 free:76816kB min:7140kB low:8924kB high:10708kB active_anon:134416kB inactive_anon:36kB active_file:17540kB inactive_file:1592808kB unevictable:0kB isolated(anon):0kB isolated(file):548kB present:2047564kB managed:1955656kB mlocked:0kB dirty:86020kB writeback:148912kB mapped:44kB shmem:36kB slab_reclaimable:82696kB slab_unreclaimable:15320kB kernel_stack:5904kB pagetables:200kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.077045] lowmem_reserve[]: 0 0 14078 14078
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.077053] Node 0 Normal free:147632kB min:52716kB low:65892kB high:79072kB active_anon:4105856kB inactive_anon:1344kB active_file:442184kB inactive_file:9197052kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:14680064kB managed:14416344kB mlocked:0kB dirty:586816kB writeback:896732kB mapped:40516kB shmem:1456kB slab_reclaimable:211208kB slab_unreclaimable:81316kB kernel_stack:23888kB pagetables:16060kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.077079] lowmem_reserve[]: 0 0 0 0
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.077086] Node 1 Normal free:84732kB min:30192kB low:37740kB high:45288kB active_anon:4541032kB inactive_anon:196kB active_file:323424kB inactive_file:2425748kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:8388608kB managed:8256116kB mlocked:0kB dirty:213172kB writeback:345112kB mapped:28388kB shmem:260kB slab_reclaimable:223172kB slab_unreclaimable:113188kB kernel_stack:30640kB pagetables:18832kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.077111] lowmem_reserve[]: 0 0 0 0
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.077118] Node 0 DMA: 0*4kB 0*8kB 1*16kB (U) 1*32kB (U) 1*64kB (U) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 112kB
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.077139] Node 0 DMA32: 970*4kB (UEM) 1043*8kB (UEM) 1171*16kB (UM) 576*32kB (UEM) 424*64kB (UEM) 6*128kB (U) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 77296kB
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.077163] Node 0 Normal: 15021*4kB (UE) 1775*8kB (UEM) 4127*16kB (UM) 150*32kB (UMR) 40*64kB (UMR) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 147676kB
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.077186] Node 1 Normal: 8776*4kB (UE) 1644*8kB (UEM) 1905*16kB (UEM) 108*32kB (UEMR) 44*64kB (UEMR) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 85008kB
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.077210] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.077216] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.077222] 3501074 total pagecache pages
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.077227] 0 pages in swap cache
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.077230] Swap cache stats: add 0, delete 0, find 0/0
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.077234] Free swap = 0kB
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.077237] Total swap = 0kB
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.077241] 6283048 pages RAM
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.077244] 0 pages HighMem/MovableOnly
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.077247] 33123 pages reserved
Nov 29 12:08:36 arh-ibstorage1-ib kernel: [ 3890.077250] 0 pages hwpoisoned
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
