Hi Dave,
Yet another crash (following assertion trace see further down), hopefully with
some more useful information.
The server is always running under some memory pressure (also expressed by hpsa
having a hard time to allocate order-4 memory):
[1018038.443525] hpacucli: page allocation failure: order:4, mode:0xc0d0
[1018038.443528] CPU: 0 PID: 12482 Comm: hpacucli Not tainted 3.18.5-x86_64 #2
[1018038.443530] Hardware name: HP ProLiant DL360 G6, BIOS P64 07/02/2013
[1018038.443533] 0000000000000000 ffff8800025bb978 ffffffff817b2be3 000000000000d4d4
[1018038.443535] 000000000000c0d0 ffff8800025bba08 ffffffff81104e9f 0000000000000004
[1018038.443537] ffff8800ac2ae9e0 ffff8800025bba08 ffffffff817b0752 ffff8800025bb9d0
[1018038.443538] Call Trace:
[1018038.443544] [<ffffffff817b2be3>] dump_stack+0x46/0x58
[1018038.443549] [<ffffffff81104e9f>] warn_alloc_failed+0xcf/0x120
[1018038.443553] [<ffffffff817b0752>] ? __alloc_pages_direct_compact+0x61/0x18e
[1018038.443556] [<ffffffff811074d7>] __alloc_pages_nodemask+0x577/0x940
[1018038.443559] [<ffffffff81107962>] alloc_kmem_pages+0x12/0x20
[1018038.443562] [<ffffffff8111aaf3>] kmalloc_order+0x13/0x50
[1018038.443566] [<ffffffff815740f6>] hpsa_scan_start+0x156/0x1830
[1018038.443568] [<ffffffff81588c26>] ? sg_open+0x376/0x5b0
[1018038.443572] [<ffffffff81150515>] ? chrdev_open+0xa5/0x1a0
[1018038.443576] [<ffffffff811555fa>] ? generic_permission+0xda/0x1e0
[1018038.443578] [<ffffffff81150470>] ? cdev_put+0x30/0x30
[1018038.443580] [<ffffffff81575e7f>] hpsa_ioctl+0x1af/0xcd0
[1018038.443582] [<ffffffff81159035>] ? do_last.isra.60+0x175/0xf60
[1018038.443584] [<ffffffff81156d29>] ? link_path_walk+0x69/0x7e0
[1018038.443587] [<ffffffff8113a3e6>] ? check_bytes_and_report+0x36/0x110
[1018038.443590] [<ffffffff8113aec3>] ? check_object+0xa3/0x250
[1018038.443594] [<ffffffff814d27aa>] scsi_ioctl+0x30a/0x410
[1018038.443596] [<ffffffff8158982b>] sg_ioctl+0x1bb/0xc50
[1018038.443598] [<ffffffff8115d708>] do_vfs_ioctl+0x2c8/0x4a0
[1018038.443601] [<ffffffff8113c22f>] ? kmem_cache_free+0xef/0x120
[1018038.443602] [<ffffffff8115a401>] ? final_putname+0x21/0x50
[1018038.443605] [<ffffffff8114b428>] ? do_sys_open+0x198/0x220
[1018038.443607] [<ffffffff8115d91c>] SyS_ioctl+0x3c/0x80
[1018038.443610] [<ffffffff817bb756>] system_call_fastpath+0x16/0x1b
[1018038.443611] Mem-Info:
[1018038.443612] DMA per-cpu:
[1018038.443613] CPU 0: hi: 0, btch: 1 usd: 0
[1018038.443614] CPU 1: hi: 0, btch: 1 usd: 0
[1018038.443615] CPU 2: hi: 0, btch: 1 usd: 0
[1018038.443616] CPU 3: hi: 0, btch: 1 usd: 0
[1018038.443617] CPU 4: hi: 0, btch: 1 usd: 0
[1018038.443618] CPU 5: hi: 0, btch: 1 usd: 0
[1018038.443619] CPU 6: hi: 0, btch: 1 usd: 0
[1018038.443620] CPU 7: hi: 0, btch: 1 usd: 0
[1018038.443621] DMA32 per-cpu:
[1018038.443622] CPU 0: hi: 186, btch: 31 usd: 0
[1018038.443623] CPU 1: hi: 186, btch: 31 usd: 0
[1018038.443624] CPU 2: hi: 186, btch: 31 usd: 0
[1018038.443625] CPU 3: hi: 186, btch: 31 usd: 0
[1018038.443626] CPU 4: hi: 186, btch: 31 usd: 0
[1018038.443627] CPU 5: hi: 186, btch: 31 usd: 0
[1018038.443628] CPU 6: hi: 186, btch: 31 usd: 0
[1018038.443629] CPU 7: hi: 186, btch: 31 usd: 0
[1018038.443629] Normal per-cpu:
[1018038.443630] CPU 0: hi: 186, btch: 31 usd: 0
[1018038.443632] CPU 1: hi: 186, btch: 31 usd: 0
[1018038.443633] CPU 2: hi: 186, btch: 31 usd: 0
[1018038.443634] CPU 3: hi: 186, btch: 31 usd: 0
[1018038.443635] CPU 4: hi: 186, btch: 31 usd: 0
[1018038.443636] CPU 5: hi: 186, btch: 31 usd: 0
[1018038.443637] CPU 6: hi: 186, btch: 31 usd: 0
[1018038.443638] CPU 7: hi: 186, btch: 31 usd: 0
[1018038.443642] active_anon:175496 inactive_anon:196056 isolated_anon:0
[1018038.443642] active_file:320293 inactive_file:1080120 isolated_file:0
[1018038.443642] unevictable:8 dirty:3899 writeback:0 unstable:0
[1018038.443642] free:108229 slab_reclaimable:115980 slab_unreclaimable:17333
[1018038.443642] mapped:10797 shmem:40 pagetables:7135 bounce:0
[1018038.443642] free_cma:0
[1018038.443647] DMA free:15836kB min:260kB low:324kB high:388kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writ
eback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
[1018038.443649] lowmem_reserve[]: 0 3533 8000 8000
[1018038.443655] DMA32 free:256676kB min:59584kB low:74480kB high:89376kB active_anon:340428kB inactive_anon:387756kB active_file:667068kB inactive_file:1671452kB unevictable:32kB isolated(anon):0kB isolated(file):0kB present:3643520kB managed:3630440kB mlocked:32kB dirty:7568kB writeback:0kB mapped:25180kB shmem:84kB slab_reclaimable:235772kB slab_unreclaimable:28068kB kernel_stack:1616kB
pagetables:12260kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
[1018038.443656] lowmem_reserve[]: 0 0 4466 4466
[1018038.443662] Normal free:160404kB min:75316kB low:94144kB high:112972kB active_anon:361556kB inactive_anon:396468kB active_file:614104kB inactive_file:2649028kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:4718588kB managed:4574088kB mlocked:0kB dirty:8028kB writeback:0kB mapped:18008kB shmem:76kB slab_reclaimable:228148kB slab_unreclaimable:41264kB kernel_stack:2336kB
pagetables:16280kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
[1018038.443664] lowmem_reserve[]: 0 0 0 0
[1018038.443673] DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (U) 3*4096kB (MR) = 15836kB
[1018038.443681] DMA32: 14482*4kB (UEM) 14822*8kB (UEM) 4479*16kB (UEM) 245*32kB (UEM) 17*64kB (UEM) 5*128kB (EM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 257736kB
[1018038.443689] Normal: 34007*4kB (UEM) 1494*8kB (UEM) 79*16kB (UEM) 65*32kB (UEM) 98*64kB (UM) 26*128kB (UM) 1*256kB (M) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 161180kB
[1018038.443690] 1431624 total pagecache pages
[1018038.443691] 31052 pages in swap cache
[1018038.443692] Swap cache stats: add 487562, delete 456510, find 302043467/302273642
[1018038.443692] Free swap = 3038676kB
[1018038.443693] Total swap = 4192252kB
[1018038.443693] 2094525 pages RAM
[1018038.443694] 0 pages HighMem/MovableOnly
[1018038.443694] 39416 pages reserved
(above OOM complaint happened multiple times, no at each hpaccucli run but for
some runs multiple times in a row)
The XFS trace:
[1020178.803465] XFS: Assertion failed: xfs_isiflocked(ip), file: /data/kernel/linux-git/fs/xfs/xfs_inode.c, line: 3195
[1020178.817795] ------------[ cut here ]------------
[1020178.825680] WARNING: CPU: 6 PID: 630 at /data/kernel/linux-git/fs/xfs/xfs_message.c:99 asswarn+0x2e/0x30()
[1020178.839685] Modules linked in:
[1020178.839689] CPU: 6 PID: 630 Comm: xfsaild/sda3 Not tainted 3.18.5-x86_64 #2
[1020178.839690] Hardware name: HP ProLiant DL360 G6, BIOS P64 07/02/2013
[1020178.839693] 0000000000000009 ffff8802143a3c28 ffffffff817b2be3 000000000000a8a8
[1020178.839694] 0000000000000000 ffff8802143a3c68 ffffffff81088a97 ffff8802143a3c88
[1020178.839696] ffff8801008ab400 ffff880122d89dc0 0000000000000008 ffff880213c5bb00
[1020178.839698] Call Trace:
[1020178.839705] [<ffffffff817b2be3>] dump_stack+0x46/0x58
[1020178.839708] [<ffffffff81088a97>] warn_slowpath_common+0x77/0xa0
[1020178.839710] [<ffffffff81088b65>] warn_slowpath_null+0x15/0x20
[1020178.839712] [<ffffffff812599fe>] asswarn+0x2e/0x30
[1020178.839714] [<ffffffff812546e1>] xfs_iflush_int+0x81/0x480
[1020178.839716] [<ffffffff81254cd3>] xfs_iflush_cluster+0x1f3/0x3c0
[1020178.839718] [<ffffffff812584f3>] xfs_iflush+0x1e3/0x2f0
[1020178.839723] [<ffffffff817ba747>] ? schedule_timeout+0x107/0x190
[1020178.839726] [<ffffffff81269306>] xfs_inode_item_push+0x146/0x190
[1020178.839729] [<ffffffff81270d29>] xfsaild+0x1d9/0x4b0
[1020178.839731] [<ffffffff81270b50>] ? xfs_trans_ail_cursor_first+0x90/0x90
[1020178.839734] [<ffffffff810a0a34>] kthread+0xc4/0xe0
[1020178.839737] [<ffffffff810a0970>] ? kthread_create_on_node+0x170/0x170
[1020178.839739] [<ffffffff817bb6ac>] ret_from_fork+0x7c/0xb0
[1020178.839741] [<ffffffff810a0970>] ? kthread_create_on_node+0x170/0x170
[1020178.839743] ---[ end trace 8908bf05bbd238cf ]---
[1020179.700610] general protection fault: 0000 [#1] SMP
[1020179.708213] Modules linked in:
[1020179.712243] CPU: 0 PID: 175 Comm: kworker/0:1H Tainted: G W 3.18.5-x86_64 #2
[1020179.724201] Hardware name: HP ProLiant DL360 G6, BIOS P64 07/02/2013
[1020179.734151] Workqueue: xfslogd xfs_buf_ioend_work
[1020179.740276] task: ffff880212decba0 ti: ffff880213788000 task.ti: ffff880213788000
[1020179.752158] RIP: 0010:[<ffffffff81271477>] [<ffffffff81271477>] xfs_trans_ail_delete_bulk+0x87/0x1a0
[1020179.766094] RSP: 0018:ffff88021378bcc8 EFLAGS: 00010202
[1020179.772314] RAX: dead000000100100 RBX: ffff880213869980 RCX: ffff880213869990
[1020179.784156] RDX: 6b6b6b6b6b6b6b6b RSI: 6b6b6b6b6b6b6b6b RDI: ffff880213869980
[1020179.794234] RBP: ffff88021378bd18 R08: 0000000000000000 R09: 0000000000000bfd
[1020179.804273] R10: 0000000000001c1e R11: ffffea00006e0180 R12: ffff88001b806000
[1020179.814335] R13: ffff88021378bd38 R14: ffff880122d89dc0 R15: 0000000000000000
[1020179.826195] FS: 0000000000000000(0000) GS:ffff88021fa00000(0000) knlGS:0000000000000000
[1020179.838174] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[1020179.846235] CR2: 00007fabeb28cff8 CR3: 0000000001c15000 CR4: 00000000000007f0
[1020179.856292] Stack:
[1020179.860196] ffff88021378bd28 00000008810aeaf9 ffff880213869990 ffff88021378bd38
[1020179.870324] ffff880213869980 0000000000000002 0000000000000002 ffff880213869980
[1020179.882184] 0000000000000000 ffff8801bd4b36e0 ffff88021378bd88 ffffffff81269614
[1020179.892310] Call Trace:
[1020179.896275] [<ffffffff81269614>] xfs_iflush_done+0x1b4/0x1e0
[1020179.904318] [<ffffffff810b008c>] ? put_prev_entity+0x2c/0x3b0
[1020179.912395] [<ffffffff812675d1>] xfs_buf_do_callbacks+0x61/0x80
[1020179.922280] [<ffffffff81267879>] xfs_buf_iodone_callbacks+0x29/0x140
[1020179.932220] [<ffffffff81245c1b>] xfs_buf_ioend+0x9b/0xe0
[1020179.940235] [<ffffffff81245c70>] xfs_buf_ioend_work+0x10/0x20
[1020179.948328] [<ffffffff8109c564>] process_one_work+0x134/0x360
[1020179.956376] [<ffffffff8109cb49>] worker_thread+0x119/0x440
[1020179.964401] [<ffffffff8109ca30>] ? rescuer_thread+0x2a0/0x2a0
[1020179.974254] [<ffffffff810a0a34>] kthread+0xc4/0xe0
[1020179.980395] [<ffffffff810a0970>] ? kthread_create_on_node+0x170/0x170
[1020179.990359] [<ffffffff817bb6ac>] ret_from_fork+0x7c/0xb0
[1020179.998371] [<ffffffff810a0970>] ? kthread_create_on_node+0x170/0x170
[1020180.008332] Code: 1f 44 00 00 4d 8b 75 00 49 83 c5 08 41 f6 46 34 01 0f 84 9d 00 00 00 48 b8 00 01 10 00 00 00 ad de 49 8b 36 48 89 df 49 8b 56 08 <48> 89 56 08 48 89 32 4c 89 f6 49 89 06 48 b8 00 02 20 00 00 00
[1020180.036304] RIP [<ffffffff81271477>] xfs_trans_ail_delete_bulk+0x87/0x1a0
[1020180.046295] RSP <ffff88021378bcc8>
[1020180.221761] ---[ end trace 8908bf05bbd238d0 ]---
[1020180.230144] BUG: unable to handle kernel paging request at ffffffffffffffd8
[1020180.240440] IP: [<ffffffff810a0efc>] kthread_data+0xc/0x20
[1020180.248467] PGD 1c16067 PUD 1c18067 PMD 0
[1020180.254511] Oops: 0000 [#2] SMP
[1020180.258656] Modules linked in:
[1020180.264457] CPU: 0 PID: 175 Comm: kworker/0:1H Tainted: G D W 3.18.5-x86_64 #2
[1020180.276417] Hardware name: HP ProLiant DL360 G6, BIOS P64 07/02/2013
[1020180.284579] task: ffff880212decba0 ti: ffff880213788000 task.ti: ffff880213788000
[1020180.296517] RIP: 0010:[<ffffffff810a0efc>] [<ffffffff810a0efc>] kthread_data+0xc/0x20
[1020180.308433] RSP: 0018:ffff88021378ba50 EFLAGS: 00010092
[1020180.316420] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 000000000000000f
[1020180.326483] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff880212decba0
[1020180.336554] RBP: ffff88021378ba68 R08: 000000000000c000 R09: 000000000000bb23
[1020180.346618] R10: 0000000000000000 R11: ffffea00084f1a00 R12: ffff880212decfa8
[1020180.358445] R13: 0000000000000000 R14: 0000000000000000 R15: ffff880212decba0
[1020180.368507] FS: 0000000000000000(0000) GS:ffff88021fa00000(0000) knlGS:0000000000000000
[1020180.380496] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[1020180.388568] CR2: 0000000000000028 CR3: 0000000001c15000 CR4: 00000000000007f0
[1020180.398621] Stack:
[1020180.402532] ffffffff8109cee0 ffff88021378ba68 ffff88021fa10980 ffff88021378bad8
[1020180.412656] ffffffff817b7e53 ffff88021378ba88 ffff880212decba0 0000000000010980
[1020180.424540] ffffffff8108946d ffff88021378bfd8 0000000000010980 ffff88021378baa8
[1020180.434650] Call Trace:
[1020180.438585] [<ffffffff8109cee0>] ? wq_worker_sleeping+0x10/0xa0
[1020180.448461] [<ffffffff817b7e53>] __schedule+0x393/0x5e0
[1020180.454680] [<ffffffff8108946d>] ? release_task+0x2fd/0x3f0
[1020180.464506] [<ffffffff817b80c4>] schedule+0x24/0x70
[1020180.470659] [<ffffffff8108aa3a>] do_exit+0x65a/0x970
[1020180.478619] [<ffffffff81005f29>] oops_end+0x89/0xd0
[1020180.486555] [<ffffffff810060a6>] die+0x46/0x70
[1020180.492643] [<ffffffff81003b0a>] do_general_protection+0xca/0x150
[1020180.502534] [<ffffffff817bcd42>] general_protection+0x22/0x30
[1020180.510617] [<ffffffff81271477>] ? xfs_trans_ail_delete_bulk+0x87/0x1a0
[1020180.520616] [<ffffffff81271497>] ? xfs_trans_ail_delete_bulk+0xa7/0x1a0
[1020180.530604] [<ffffffff81269614>] xfs_iflush_done+0x1b4/0x1e0
[1020180.538667] [<ffffffff810b008c>] ? put_prev_entity+0x2c/0x3b0
[1020180.546738] [<ffffffff812675d1>] xfs_buf_do_callbacks+0x61/0x80
[1020180.556625] [<ffffffff81267879>] xfs_buf_iodone_callbacks+0x29/0x140
[1020180.566630] [<ffffffff81245c1b>] xfs_buf_ioend+0x9b/0xe0
[1020180.574567] [<ffffffff81245c70>] xfs_buf_ioend_work+0x10/0x20
[1020180.582643] [<ffffffff8109c564>] process_one_work+0x134/0x360
[1020180.590716] [<ffffffff8109cb49>] worker_thread+0x119/0x440
[1020180.598756] [<ffffffff8109ca30>] ? rescuer_thread+0x2a0/0x2a0
[1020180.608596] [<ffffffff810a0a34>] kthread+0xc4/0xe0
[1020180.614739] [<ffffffff810a0970>] ? kthread_create_on_node+0x170/0x170
[1020180.624709] [<ffffffff817bb6ac>] ret_from_fork+0x7c/0xb0
[1020180.632721] [<ffffffff810a0970>] ? kthread_create_on_node+0x170/0x170
[1020180.642671] Code: 48 89 e5 5d 48 8b 40 c8 48 c1 e8 02 83 e0 01 c3 66 66 66 66 66 66 2e 0f 1f 84 00 00 00 00 00 48 8b 87 b0 03 00 00 55 48 89 e5 5d <48> 8b 40 d8 c3 66 66 66 66 66 66 2e 0f 1f 84 00 00 00 00 00 55
[1020180.670633] RIP [<ffffffff810a0efc>] kthread_data+0xc/0x20
[1020180.678649] RSP <ffff88021378ba50>
[1020180.682790] CR2: ffffffffffffffd8
[1020180.688680] ---[ end trace 8908bf05bbd238d1 ]---
[1020180.694787] Fixing recursive fault but reboot is needed!
[1021280.341447] Clocksource tsc unstable (delta = 899898448637 ns)
[1025061.698138] INFO: rcu_sched detected stalls on CPUs/tasks: { 0 1 2 5 7} (detected by 3, t=482838 jiffies, g=32626328, c=32626327, q=3381)
[1025061.716261] Task dump for CPU 0:
[1025061.720357] kworker/0:1H D 0000000000000046 0 175 2 0x00000000
[1025061.730423] 00000001000f000d ffff880213c68840 ffff88021378b968 ffffffff810ae624
[1025061.742300] ffff880212decc08 ffff880212decc08 ffff88021fa109f0 0000000000000001
[1025061.752446] ffff880212decfa8 0000000000000000 0000000000000000 ffff880212decba0
[1025061.764260] Call Trace:
[1025061.766477] [<ffffffff810ae624>] ? update_curr+0xa4/0x100
[1025061.774495] [<ffffffff8103945c>] ? do_page_fault+0xc/0x10
[1025061.784271] [<ffffffff817bcd72>] ? page_fault+0x22/0x30
[1025061.790497] [<ffffffff810a0efc>] ? kthread_data+0xc/0x20
[1025061.798505] [<ffffffff8109cee0>] ? wq_worker_sleeping+0x10/0xa0
[1025061.808376] [<ffffffff817b7e53>] ? __schedule+0x393/0x5e0
[1025061.816395] [<ffffffff8108946d>] ? release_task+0x2fd/0x3f0
[1025061.824450] [<ffffffff817b80c4>] ? schedule+0x24/0x70
[1025061.832454] [<ffffffff8108aa3a>] ? do_exit+0x65a/0x970
[1025061.840379] [<ffffffff81005f29>] ? oops_end+0x89/0xd0
[1025061.848341] [<ffffffff810060a6>] ? die+0x46/0x70
[1025061.854462] [<ffffffff81003b0a>] ? do_general_protection+0xca/0x150
[1025061.864407] [<ffffffff817bcd42>] ? general_protection+0x22/0x30
[1025061.872522] [<ffffffff81271477>] ? xfs_trans_ail_delete_bulk+0x87/0x1a0
[1025061.882480] [<ffffffff81271497>] ? xfs_trans_ail_delete_bulk+0xa7/0x1a0
[1025061.892480] [<ffffffff81269614>] ? xfs_iflush_done+0x1b4/0x1e0
[1025061.902352] [<ffffffff810b008c>] ? put_prev_entity+0x2c/0x3b0
[1025061.910423] [<ffffffff812675d1>] ? xfs_buf_do_callbacks+0x61/0x80
[1025061.918549] [<ffffffff81267879>] ? xfs_buf_iodone_callbacks+0x29/0x140
[1025061.928543] [<ffffffff81245c1b>] ? xfs_buf_ioend+0x9b/0xe0
[1025061.936555] [<ffffffff81245c70>] ? xfs_buf_ioend_work+0x10/0x20
[1025061.946434] [<ffffffff8109c564>] ? process_one_work+0x134/0x360
[1025061.954540] [<ffffffff8109cb49>] ? worker_thread+0x119/0x440
[1025061.964373] [<ffffffff8109ca30>] ? rescuer_thread+0x2a0/0x2a0
[1025061.972446] [<ffffffff810a0a34>] ? kthread+0xc4/0xe0
[1025061.980430] [<ffffffff810a0970>] ? kthread_create_on_node+0x170/0x170
[1025061.988587] [<ffffffff817bb6ac>] ? ret_from_fork+0x7c/0xb0
[1025061.998393] [<ffffffff810a0970>] ? kthread_create_on_node+0x170/0x170
[1025062.006579] Task dump for CPU 1:
[1025062.012454] nrpe R running task 0 2998 1612 0x00000008
[1025062.022521] 00000001002e0018 ffff880059317c30 ffffffff81106beb ffffffff81c8b248
[1025062.032619] 0000000100000141 0000000000002200 0000000100200013 ffff880059317c50
[1025062.044507] ffffffff8113b7d8 ffff880216843010 ffff880216843030 ffffea0000af48a0
[1025062.054633] Call Trace:
[1025062.058578] [<ffffffff81106beb>] ? get_page_from_freelist+0x32b/0x6a0
[1025062.068530] [<ffffffff8113b7d8>] ? deactivate_slab+0x118/0x400
[1025062.076615] [<ffffffff8113b7d8>] ? deactivate_slab+0x118/0x400
[1025062.086502] [<ffffffff8113aec3>] ? check_object+0xa3/0x250
[1025062.094524] [<ffffffff8113a378>] ? init_object+0x38/0x70
[1025062.102512] [<ffffffff8109ebf9>] ? alloc_pid+0x29/0x480
[1025062.110514] [<ffffffff8108718b>] ? copy_process.part.52+0xb7b/0x1600
[1025062.120469] [<ffffffff810a44db>] ? task_rq_lock+0x4b/0xa0
[1025062.128484] [<ffffffff810aa5ad>] ? sched_move_task+0x9d/0x120
[1025062.136551] [<ffffffff810a681c>] ? set_task_cpu+0x2c/0xe0
[1025062.144576] [<ffffffff812bb7f0>] ? cpumask_next_and+0x30/0x50
[1025062.152664] [<ffffffff810acb7e>] ? select_task_rq_fair+0x1de/0x7d0
[1025062.162564] [<ffffffff810ac496>] ? sched_slice.isra.61+0x46/0x90
[1025062.170681] [<ffffffff810adf5a>] ? init_task_runnable_average+0x2a/0x60
[1025062.180678] [<ffffffff810a89ca>] ? wake_up_new_task+0x9a/0x140
[1025062.190547] [<ffffffff8116720d>] ? __alloc_fd+0x7d/0x120
[1025062.198541] [<ffffffff81087e0b>] ? do_fork+0x11b/0x2b0
[1025062.206517] [<ffffffff811672db>] ? get_unused_fd_flags+0x2b/0x30
[1025062.214654] [<ffffffff81154495>] ? __do_pipe_flags+0x65/0xb0
[1025062.222703] [<ffffffff811672ff>] ? __fd_install+0x1f/0x60
[1025062.230715] [<ffffffff81088021>] ? SyS_clone+0x11/0x20
[1025062.238689] [<ffffffff817bba39>] ? stub_clone+0x69/0x90
[1025062.246699] [<ffffffff817bb756>] ? system_call_fastpath+0x16/0x1b
[1025062.256582] Task dump for CPU 2:
[1025062.260700] xfsaild/sda3 R running task 0 630 2 0x00000008
[1025062.270738] ffff8802143a3d18 ffffffff810c96c6 ffff8802143a3d88 ffff8802143a3d88
[1025062.282649] ffffffff82807340 0000000000000282 ffff8802143a3d48 ffffffff810c9793
[1025062.292776] ffff8802143a3d68 0000000000000282 ffff8802143a3d88 ffff8802143a3d88
[1025062.304665] Call Trace:
[1025062.308578] [<ffffffff810c96c6>] ? lock_timer_base.isra.30+0x26/0x50
[1025062.316753] [<ffffffff810c9793>] try_to_del_timer_sync+0x43/0x60
[1025062.326667] [<ffffffff810c97fa>] del_timer_sync+0x4a/0x60
[1025062.334686] [<ffffffff817ba747>] schedule_timeout+0x107/0x190
[1025062.343966] [<ffffffff817bb28c>] ? _raw_spin_lock+0x1c/0x30
[1025062.351780] [<ffffffff81270c6f>] xfsaild+0x11f/0x4b0
[1025062.359727] [<ffffffff81270b50>] ? xfs_trans_ail_cursor_first+0x90/0x90
[1025062.369714] [<ffffffff810a0a34>] kthread+0xc4/0xe0
[1025062.377657] [<ffffffff810a0970>] ? kthread_create_on_node+0x170/0x170
[1025062.387590] [<ffffffff817bb6ac>] ret_from_fork+0x7c/0xb0
[1025062.395595] [<ffffffff810a0970>] ? kthread_create_on_node+0x170/0x170
[1025062.403791] Task dump for CPU 5:
[1025062.409690] swapper/5 R running task 0 0 1 0x00200000
[1025062.419715] ffff88021454ff18 ffffffff810d82c8 0000000000001000 0000000000001000
[1025062.429831] ffff88021fb90980 0000000000000000 0000000000200000 0100000000000000
[1025062.441734] ffff88021454ffd8 0000000000000000 ffff88021fb4bee0 ffffffffffffff1d
[1025062.451835] Call Trace:
[1025062.455782] [<ffffffff810d82c8>] ? __tick_nohz_idle_enter+0xd8/0x400
[1025062.465761] [<ffffffff810b86a2>] ? cpu_startup_entry+0x102/0x2e0
[1025062.473858] [<ffffffff810b8685>] ? cpu_startup_entry+0xe5/0x2e0
[1025062.483723] [<ffffffff8102cda3>] start_secondary+0x183/0x190
[1025062.491779] Task dump for CPU 7:
[1025062.497647] swapper/7 R running task 0 0 1 0x00200000
[1025062.507703] ffff880214557e98 ffffffff810d82c8 ffff880214557e68 00000001061abf32
[1025062.517826] 00000001061abf31 00000000145dd330 ffff880214557fd8 ffffffffffffffff
[1025062.529688] ffff88021fbcbee0 0000000000000096 ffff880214557ea8 0000000000000096
[1025062.539813] Call Trace:
[1025062.543770] [<ffffffff810d82c8>] ? __tick_nohz_idle_enter+0xd8/0x400
[1025062.553713] [<ffffffff810b8685>] ? cpu_startup_entry+0xe5/0x2e0
[1025062.561817] [<ffffffff8102cda3>] ? start_secondary+0x183/0x190
[1025062.586685] =============================================================================
[1025062.596941] BUG xfs_ili (Tainted: G D W ): Poison overwritten
[1025062.606904] -----------------------------------------------------------------------------
[1025062.606904]
[1025062.620946] INFO: 0xffff880122d89df8-0xffff880122d89e53. First byte 0x0 instead of 0x6b
[1025062.632938] INFO: Slab 0xffffea00048b6240 objects=23 used=0 fp=0xffff880122d89790 flags=0x8000000000000080
[1025062.646949] INFO: Object 0xffff880122d89dc0 @offset=3520 fp=0xffff880122d89bb0
[1025062.646949]
[1025062.660839] Bytes b4 ffff880122d89db0: b0 90 d8 22 01 88 ff ff 5a 5a 5a 5a 5a 5a 5a 5a ..."....ZZZZZZZZ
[1025062.674834] Object ffff880122d89dc0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[1025062.688799] Object ffff880122d89dd0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[1025062.700991] Object ffff880122d89de0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[1025062.714958] Object ffff880122d89df0: 6b 6b 6b 6b 6b 6b 6b 6b 00 00 00 00 00 00 00 00 kkkkkkkk........
[1025062.728935] Object ffff880122d89e00: 60 94 26 81 ff ff ff ff 6b 6b 6b 6b 6b 6b 6b 6b `.&.....kkkkkkkk
[1025062.742883] Object ffff880122d89e10: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[1025062.756849] Object ffff880122d89e20: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[1025062.770808] Object ffff880122d89e30: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[1025062.782997] Object ffff880122d89e40: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 01 00 6b 6b 6b 6b kkkkkkkkkk..kkkk
[1025062.796976] Object ffff880122d89e50: 00 00 00 00 6b 6b 6b a5 ....kkk.
[1025062.809043] Redzone ffff880122d89e58: bb bb bb bb bb bb bb bb ........
[1025062.822922] Padding ffff880122d89e68: 5a 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZ
[1025062.835006] CPU: 3 PID: 25139 Comm: kworker/3:2 Tainted: G B D W 3.18.5-x86_64 #2
[1025062.847001] Hardware name: HP ProLiant DL360 G6, BIOS P64 07/02/2013
[1025062.856957] Workqueue: xfs-reclaim/sda3 xfs_reclaim_worker
[1025062.864974] ffff880122d89dc0 ffff88011e7939a8 ffffffff817b2be3 00000000000000fc
[1025062.875094] ffff8800da170700 ffff88011e7939e8 ffffffff8113a288 0000000000000008
[1025062.886964] ffff880100000001 ffff880122d89e54 ffff8800da170700 000000000000006b
[1025062.897098] Call Trace:
[1025062.901052] [<ffffffff817b2be3>] dump_stack+0x46/0x58
[1025062.908996] [<ffffffff8113a288>] print_trailer+0x148/0x200
[1025062.917031] [<ffffffff8113a47f>] check_bytes_and_report+0xcf/0x110
[1025062.926948] [<ffffffff8113aff7>] check_object+0x1d7/0x250
[1025062.934963] [<ffffffff8113b1b0>] __free_slab+0x140/0x180
[1025062.942961] [<ffffffff8126a4d7>] ? xfs_inode_item_destroy+0x17/0x20
[1025062.951164] [<ffffffff8113b21c>] discard_slab+0x2c/0x40
[1025062.959168] [<ffffffff817b1a29>] __slab_free+0x297/0x2a6
[1025062.967116] [<ffffffff810c6ec2>] ? call_rcu_sched+0x12/0x20
[1025062.976944] [<ffffffff812c0db8>] ? __radix_tree_delete_node+0x58/0x150
[1025062.985135] [<ffffffff8113c22f>] kmem_cache_free+0xef/0x120
[1025062.994968] [<ffffffff8126a4d7>] xfs_inode_item_destroy+0x17/0x20
[1025063.003091] [<ffffffff8124d009>] xfs_inode_free+0x79/0x130
[1025063.011120] [<ffffffff8124d2cb>] xfs_reclaim_inode+0x20b/0x330
[1025063.020991] [<ffffffff8124d647>] xfs_reclaim_inodes_ag+0x257/0x370
[1025063.029126] [<ffffffff8124de48>] xfs_reclaim_worker+0x28/0x40
[1025063.038989] [<ffffffff8109c564>] process_one_work+0x134/0x360
[1025063.047059] [<ffffffff8109cb49>] worker_thread+0x119/0x440
[1025063.055105] [<ffffffff8109ca30>] ? rescuer_thread+0x2a0/0x2a0
[1025063.063192] [<ffffffff810a0a34>] kthread+0xc4/0xe0
[1025063.071079] [<ffffffff810a0970>] ? kthread_create_on_node+0x170/0x170
[1025063.081050] [<ffffffff817bb6ac>] ret_from_fork+0x7c/0xb0
[1025063.089063] [<ffffffff810a0970>] ? kthread_create_on_node+0x170/0x170
[1025063.099093] FIX xfs_ili: Restoring 0xffff880122d89df8-0xffff880122d89e53=0x6b
[1025063.099093]
Cheers,
Bruno
_______________________________________________
xfs mailing list
xfs@xxxxxxxxxxx
http://oss.sgi.com/mailman/listinfo/xfs
