Greetings,
I am using stock XFS from kernel 3.8.13, compiled with kmemleak
enabled. I am testing a particular scenario, in which the underlying
block device returns IO errors during XFS umount. Almost in all cases
this results in kernel crashes in various places, and sometimes
kmemleak complains, and sometimes CPU soft lockup happens. It always
happens after XFS messages like:
kernel: [ 600.190509] XFS (dm-22): metadata I/O error: block
0x7600030 ("xlog_iodone") error 125 numblks 64
kernel: [ 600.192267] XFS (dm-22): xfs_do_force_shutdown(0x2) called
from line 1115 of file
/mnt/compile/linux-stable/source/fs/xfs/xfs_log.c. Return address =
0xffffffffa05cffa1
kernel: [ 600.192319] XFS (dm-22): Log I/O Error Detected. Shutting
down filesystem
kernel: [ 600.192392] XFS (dm-22): Unable to update superblock
counters. Freespace may not be correct on next mount.
kernel: [ 600.192398] XFS (dm-22): xfs_log_force: error 5 returned.
kernel: [ 600.193687] XFS (?º.Z): Please umount the filesystem and
rectify the problem(s)
you can see here the garbage that XFS prints instead of the block device name.
In [1] and [2] I am attaching more kernel log from two such crashes.
Can anybody please advise on this?
Thanks,
Alex.
[1]
kernel: [ 600.190509] XFS (dm-22): metadata I/O error: block
0x7600030 ("xlog_iodone") error 125 numblks 64
kernel: [ 600.192267] XFS (dm-22): xfs_do_force_shutdown(0x2) called
from line 1115 of file
/mnt/compile/linux-stable/source/fs/xfs/xfs_log.c. Return address =
0xffffffffa05cffa1
kernel: [ 600.192319] XFS (dm-22): Log I/O Error Detected. Shutting
down filesystem
kernel: [ 600.192392] XFS (dm-22): Unable to update superblock
counters. Freespace may not be correct on next mount.
kernel: [ 600.192398] XFS (dm-22): xfs_log_force: error 5 returned.
kernel: [ 600.193687] XFS (?º.Z): Please umount the filesystem and
rectify the problem(s)
kernel: [ 600.223974] BUG: unable to handle kernel paging request at
0000000000010001
kernel: [ 600.225347] IP: [<ffffffff8117d823>] __kmalloc+0x83/0x1c0
kernel: [ 600.226381] PGD 486e9067 PUD 5eff6067 PMD 0
kernel: [ 600.227256] Oops: 0000 [#1] SMP
kernel: [ 600.227881] Modules linked in: xfs raid1 xfrm_user
xfrm4_tunnel tunnel4 ipcomp xfrm_ipcomp esp4 ah4 scst_vdisk(O)
iscsi_scst(O) scst(O) dm_zcache(O) dm_btrfs(O) btrfs(O) libcrc32c
dm_iostat(O) md_mod deflate zlib_deflate ctr twofish_generic
twofish_x86_64_3way glue_helper lrw xts gf128mul twofish_x86_64
twofish_common camellia_generic serpent_generic blowfish_generic
blowfish_x86_64 blowfish_common cast5_generic cast_common des_generic
xcbc rmd160 sha512_generic crypto_null af_key xfrm_algo dm_round_robin
kvm vfat fat ppdev nfsd(O) nfs_acl microcode dm_multipath psmouse
serio_raw parport_pc i2c_piix4 mac_hid lp nfsv4 auth_rpcgss parport
nfs fscache lockd sunrpc floppy
kernel: [ 600.227881] CPU 3
kernel: [ 600.227881] Pid: 26391, comm: udev-logger.sh Tainted: G
O 3.8.13-557-generic #1382000791 Bochs Bochs
kernel: [ 600.227881] RIP: 0010:[<ffffffff8117d823>]
[<ffffffff8117d823>] __kmalloc+0x83/0x1c0
kernel: [ 600.227881] RSP: 0018:ffff88005ae6fcd8 EFLAGS: 00010202
kernel: [ 600.227881] RAX: 0000000000000000 RBX: 0000000000000100
RCX: 000000000003b9be
kernel: [ 600.227881] RDX: 000000000003b9bd RSI: 0000000000000000
RDI: 0000000000016c60
kernel: [ 600.227881] RBP: ffff88005ae6fd28 R08: ffff88007fd96c60
R09: ffffffff812f4011
kernel: [ 600.227881] R10: 000000000000c6e0 R11: ffff88007fd80000
R12: ffff88007d001900
kernel: [ 600.227881] R13: 0000000000010001 R14: 00000000000000d0
R15: 0000000000000100
kernel: [ 600.227881] FS: 00007ffacca11700(0000)
GS:ffff88007fd80000(0000) knlGS:0000000000000000
kernel: [ 600.227881] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
kernel: [ 600.227881] CR2: 0000000000010001 CR3: 0000000069fab000
CR4: 00000000000006e0
kernel: [ 600.227881] DR0: 0000000000000000 DR1: 0000000000000000
DR2: 0000000000000000
kernel: [ 600.227881] DR3: 0000000000000000 DR6: 00000000ffff0ff0
DR7: 0000000000000400
kernel: [ 600.227881] Process udev-logger.sh (pid: 26391, threadinfo
ffff88005ae6e000, task ffff88005e94dc00)
kernel: [ 600.227881] Stack:
kernel: [ 600.227881] ffffffff812f4011 0000000000000000
ffff88005ae6fd58 ffffffff812be0cb
kernel: [ 600.227881] 00000000000112d0 0000000000000100
ffff880054536500 ffff88005ae6fe38
kernel: [ 600.227881] ffff880078c4bf10 0000000000000000
ffff88005ae6fd98 ffffffff812f4011
kernel: [ 600.227881] Call Trace:
kernel: [ 600.227881] [<ffffffff812f4011>] ? aa_path_name+0x91/0xe0
kernel: [ 600.227881] [<ffffffff812be0cb>] ? get_file_caps+0x17b/0x1a0
kernel: [ 600.227881] [<ffffffff812f4011>] aa_path_name+0x91/0xe0
kernel: [ 600.227881] [<ffffffff812f46e7>]
apparmor_bprm_set_creds+0x107/0x780
kernel: [ 600.227881] [<ffffffff8115d6ef>] ? vma_link+0xcf/0xe0
kernel: [ 600.227881] [<ffffffff812bf203>] security_bprm_set_creds+0x13/0x20
kernel: [ 600.227881] [<ffffffff81193698>] prepare_binprm+0xb8/0x1e0
kernel: [ 600.227881] [<ffffffff811953a2>]
do_execve_common.isra.24+0x232/0x330
kernel: [ 600.227881] [<ffffffff811954b8>] do_execve+0x18/0x20
kernel: [ 600.227881] [<ffffffff8119575d>] sys_execve+0x3d/0x60
kernel: [ 600.227881] [<ffffffff816abe89>] stub_execve+0x69/0xc0
kernel: [ 600.227881] Code: b0 4d 8b 04 24 65 4c 03 04 25 08 dc 00
00 49 8b 50 08 4d 8b 28 4d 85 ed 0f 84 03 01 00 00 49 63 44 24 20 49
8b 3c 24 48 8d 4a 01 <49> 8b 5c 05 00 4c 89 e8 65 48 0f c7 0f 0f 94 c0
84 c0 74 c2 49
kernel: [ 600.227881] RIP [<ffffffff8117d823>] __kmalloc+0x83/0x1c0
kernel: [ 600.227881] RSP <ffff88005ae6fcd8>
kernel: [ 600.227881] CR2: 0000000000010001
kernel: [ 600.283792] ---[ end trace ded81b9e3753ac4e ]---
kernel: [ 600.333036] BUG: unable to handle kernel paging request at
0000000000010001
kernel: [ 600.334395] IP: [<ffffffff8117ee4b>] kmem_cache_alloc+0x5b/0x170
kernel: [ 600.335518] PGD 486e9067 PUD 5eff6067 PMD 0
kernel: [ 600.336022] Oops: 0000 [#2] SMP
kernel: [ 600.336022] Modules linked in: xfs raid1 xfrm_user
xfrm4_tunnel tunnel4 ipcomp xfrm_ipcomp esp4 ah4 scst_vdisk(O)
iscsi_scst(O) scst(O) dm_zcache(O) dm_btrfs(O) btrfs(O) libcrc32c
dm_iostat(O) md_mod deflate zlib_deflate ctr twofish_generic
twofish_x86_64_3way glue_helper lrw xts gf128mul twofish_x86_64
twofish_common camellia_generic serpent_generic blowfish_generic
blowfish_x86_64 blowfish_common cast5_generic cast_common des_generic
xcbc rmd160 sha512_generic crypto_null af_key xfrm_algo dm_round_robin
kvm vfat fat ppdev nfsd(O) nfs_acl microcode dm_multipath psmouse
serio_raw parport_pc i2c_piix4 mac_hid lp nfsv4 auth_rpcgss parport
nfs fscache lockd sunrpc floppy
kernel: [ 600.336022] CPU 3
kernel: [ 600.336022] Pid: 26391, comm: udev-logger.sh Tainted: G
D O 3.8.13-557-generic #1382000791 Bochs Bochs
kernel: [ 600.336022] RIP: 0010:[<ffffffff8117ee4b>]
[<ffffffff8117ee4b>] kmem_cache_alloc+0x5b/0x170
kernel: [ 600.336022] RSP: 0018:ffff88005ae6f168 EFLAGS: 00010002
kernel: [ 600.336022] RAX: 0000000000000000 RBX: ffffffff81c99300
RCX: 000000000003b9be
kernel: [ 600.336022] RDX: 000000000003b9bd RSI: 0000000000008020
RDI: 0000000000016c60
kernel: [ 600.336022] RBP: ffff88005ae6f1b8 R08: ffff88007fd96c60
R09: 0000000000000001
kernel: [ 600.336022] R10: 0000000000849400 R11: 000000000000002f
R12: ffff88007d001900
kernel: [ 600.336022] R13: 0000000000010001 R14: 0000000000008020
R15: ffffffff8146e0e3
kernel: [ 600.336022] FS: 00007ffacca11700(0000)
GS:ffff88007fd80000(0000) knlGS:0000000000000000
kernel: [ 600.336022] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
kernel: [ 600.336022] CR2: 0000000000010001 CR3: 0000000069fab000
CR4: 00000000000006e0
kernel: [ 600.336022] DR0: 0000000000000000 DR1: 0000000000000000
DR2: 0000000000000000
kernel: [ 600.336022] DR3: 0000000000000000 DR6: 00000000ffff0ff0
DR7: 0000000000000400
kernel: [ 600.336022] Process udev-logger.sh (pid: 26391, threadinfo
ffff88005ae6e000, task ffff88005e94dc00)
kernel: [ 600.336022] Stack:
kernel: [ 600.336022] ffff880036e25400 ffff880048482a48
ffff88005ae6f198 ffffffff8132ca4e
kernel: [ 600.336022] ffff880036598000 ffffffff81c99300
ffff880036f68800 0000000000000020
kernel: [ 600.336022] 0000000000000020 ffff880036e26000
ffff88005ae6f1e8 ffffffff8146e0e3
kernel: [ 600.336022] Call Trace:
kernel: [ 600.336022] [<ffffffff8132ca4e>] ? cfq_remove_request+0x3e/0xf0
kernel: [ 600.336022] [<ffffffff8146e0e3>] scsi_pool_alloc_command+0x33/0x80
kernel: [ 600.336022] [<ffffffff8146ec68>] scsi_host_alloc_command+0x28/0x90
kernel: [ 600.336022] [<ffffffff8146ecf5>] __scsi_get_command+0x25/0xf0
kernel: [ 600.336022] [<ffffffff8146ee05>] scsi_get_command+0x45/0xc0
kernel: [ 600.336022] [<ffffffff81475dfa>] scsi_get_cmd_from_req+0x5a/0x70
kernel: [ 600.336022] [<ffffffff81476b05>]
scsi_setup_fs_cmnd.part.33+0x35/0x90
kernel: [ 600.336022] [<ffffffff81476b8f>] scsi_setup_fs_cmnd+0x2f/0x40
kernel: [ 600.336022] [<ffffffff814949cf>] sd_prep_fn+0xff/0xb00
kernel: [ 600.336022] [<ffffffff8131347a>] blk_peek_request+0xba/0x220
kernel: [ 600.336022] [<ffffffff81476d6f>] scsi_request_fn+0x4f/0x450
kernel: [ 600.336022] [<ffffffff8130c8b7>] __blk_run_queue+0x37/0x50
kernel: [ 600.336022] [<ffffffff8130ef21>] queue_unplugged+0x41/0xd0
kernel: [ 600.336022] [<ffffffff813139b2>] blk_flush_plug_list+0x1d2/0x230
kernel: [ 600.336022] [<ffffffff81313a28>] blk_finish_plug+0x18/0x50
kernel: [ 600.336022] [<ffffffff81231dfe>] ext4_da_writepages+0x3de/0x620
kernel: [ 600.336022] [<ffffffff8118f9a3>] ? __sb_end_write+0x33/0x70
kernel: [ 600.336022] [<ffffffff8113aec0>] do_writepages+0x20/0x40
kernel: [ 600.336022] [<ffffffff81130df1>]
__filemap_fdatawrite_range+0x51/0x60
kernel: [ 600.336022] [<ffffffff81130e52>]
filemap_write_and_wait_range+0x52/0x80
kernel: [ 600.336022] [<ffffffff812275ef>] ext4_sync_file+0x6f/0x2b0
kernel: [ 600.336022] [<ffffffff8105c988>] ? kmsg_dump_get_line+0x68/0x90
kernel: [ 600.336022] [<ffffffff811bda1b>] vfs_fsync+0x2b/0x40
kernel: [ 600.336022] [<ffffffffa02f9d76>]
dm_iostat_kmsg_write35+0xb6/0xf0 [dm_iostat]
kernel: [ 600.336022] [<ffffffffa02fa0d3>]
dm_iostat_kmsg_dump+0x73/0x220 [dm_iostat]
kernel: [ 600.336022] [<ffffffff8168b992>] ? printk+0x61/0x63
kernel: [ 600.336022] [<ffffffff8105c83c>] kmsg_dump+0x9c/0xc0
kernel: [ 600.336022] [<ffffffff81058ff9>] oops_exit+0x29/0x30
kernel: [ 600.336022] [<ffffffff816a3f33>] oops_end+0x83/0xf0
kernel: [ 600.336022] [<ffffffff8168ab25>] no_context+0x1ab/0x1ba
kernel: [ 600.336022] [<ffffffff8168ad07>] __bad_area_nosemaphore+0x1d3/0x1f2
kernel: [ 600.336022] [<ffffffff810bb842>] ? __module_text_address+0x12/0x70
kernel: [ 600.336022] [<ffffffff8168ad80>] bad_area+0x45/0x4d
kernel: [ 600.336022] [<ffffffff816a6fb8>] __do_page_fault+0x558/0x560
kernel: [ 600.336022] [<ffffffff812269c9>] ? ext4_file_open+0x79/0x250
kernel: [ 600.336022] [<ffffffff81137695>] ? prep_new_page+0x145/0x1e0
kernel: [ 600.336022] [<ffffffff81137917>] ?
get_page_from_freelist+0x1e7/0x5a0
kernel: [ 600.336022] [<ffffffff816a6fce>] do_page_fault+0xe/0x10
kernel: [ 600.336022] [<ffffffff816a66a5>] do_async_page_fault+0x35/0x90
kernel: [ 600.336022] [<ffffffff816a33c8>] async_page_fault+0x28/0x30
kernel: [ 600.336022] [<ffffffff812f4011>] ? aa_path_name+0x91/0xe0
kernel: [ 600.336022] [<ffffffff8117d823>] ? __kmalloc+0x83/0x1c0
kernel: [ 600.336022] [<ffffffff8117d7f5>] ? __kmalloc+0x55/0x1c0
kernel: [ 600.336022] [<ffffffff812f4011>] ? aa_path_name+0x91/0xe0
kernel: [ 600.336022] [<ffffffff812be0cb>] ? get_file_caps+0x17b/0x1a0
kernel: [ 600.336022] [<ffffffff812f4011>] aa_path_name+0x91/0xe0
kernel: [ 600.336022] [<ffffffff812f46e7>]
apparmor_bprm_set_creds+0x107/0x780
kernel: [ 600.336022] [<ffffffff8115d6ef>] ? vma_link+0xcf/0xe0
kernel: [ 600.336022] [<ffffffff812bf203>] security_bprm_set_creds+0x13/0x20
kernel: [ 600.336022] [<ffffffff81193698>] prepare_binprm+0xb8/0x1e0
kernel: [ 600.336022] [<ffffffff811953a2>]
do_execve_common.isra.24+0x232/0x330
kernel: [ 600.336022] [<ffffffff811954b8>] do_execve+0x18/0x20
kernel: [ 600.336022] [<ffffffff8119575d>] sys_execve+0x3d/0x60
kernel: [ 600.336022] [<ffffffff816abe89>] stub_execve+0x69/0xc0
kernel: [ 600.336022] Code: 00 4d 8b 04 24 65 4c 03 04 25 08 dc 00
00 49 8b 50 08 4d 8b 28 4d 85 ed 0f 84 03 01 00 00 49 63 44 24 20 49
8b 3c 24 48 8d 4a 01 <49> 8b 5c 05 00 4c 89 e8 65 48 0f c7 0f 0f 94 c0
84 c0 74 c2 49
kernel: [ 600.336022] RIP [<ffffffff8117ee4b>] kmem_cache_alloc+0x5b/0x170
kernel: [ 600.336022] RSP <ffff88005ae6f168>
kernel: [ 600.336022] CR2: 0000000000010001
kernel: [ 600.336022] ---[ end trace ded81b9e3753ac4f ]---
kernel: [ 600.683610] BUG: unable to handle kernel paging request at
0000000000010001
kernel: [ 600.685003] IP: [<ffffffff8117ee4b>] kmem_cache_alloc+0x5b/0x170
kernel: [ 600.686176] PGD 78bc3067 PUD 6bb27067 PMD 0
kernel: [ 600.687073] Oops: 0000 [#3] SMP
kernel: [ 600.687472] Modules linked in: xfs raid1 xfrm_user
xfrm4_tunnel tunnel4 ipcomp xfrm_ipcomp esp4 ah4 scst_vdisk(O)
iscsi_scst(O) scst(O) dm_zcache(O) dm_btrfs(O) btrfs(O) libcrc32c
dm_iostat(O) md_mod deflate zlib_deflate ctr twofish_generic
twofish_x86_64_3way glue_helper lrw xts gf128mul twofish_x86_64
twofish_common camellia_generic serpent_generic blowfish_generic
blowfish_x86_64 blowfish_common cast5_generic cast_common des_generic
xcbc rmd160 sha512_generic crypto_null af_key xfrm_algo dm_round_robin
kvm vfat fat ppdev nfsd(O) nfs_acl microcode dm_multipath psmouse
serio_raw parport_pc i2c_piix4 mac_hid lp nfsv4 auth_rpcgss parport
nfs fscache lockd sunrpc floppy
kernel: [ 600.687472] CPU 3
kernel: [ 600.687472] Pid: 15622, comm: zadara_vac Tainted: G D
O 3.8.13-557-generic #1382000791 Bochs Bochs
kernel: [ 600.687472] RIP: 0010:[<ffffffff8117ee4b>]
[<ffffffff8117ee4b>] kmem_cache_alloc+0x5b/0x170
kernel: [ 600.687472] RSP: 0018:ffff88006bae5d28 EFLAGS: 00010202
kernel: [ 600.687472] RAX: 0000000000000000 RBX: 0000000000000000
RCX: 000000000003b9be
kernel: [ 600.687472] RDX: 000000000003b9bd RSI: 00000000000080d0
RDI: 0000000000016c60
kernel: [ 600.687472] RBP: ffff88006bae5d78 R08: ffff88007fd96c60
R09: 0000000000000000
kernel: [ 600.687472] R10: 8080808080808080 R11: 0000000000000000
R12: ffff88007d001900
kernel: [ 600.687472] R13: 0000000000010001 R14: 00000000000080d0
R15: ffffffff8118f2c8
kernel: [ 600.687472] FS: 00007f920b7fe700(0000)
GS:ffff88007fd80000(0000) knlGS:0000000000000000
kernel: [ 600.687472] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
kernel: [ 600.687472] CR2: 0000000000010001 CR3: 000000006baf9000
CR4: 00000000000006e0
kernel: [ 600.687472] DR0: 0000000000000000 DR1: 0000000000000000
DR2: 0000000000000000
kernel: [ 600.687472] DR3: 0000000000000000 DR6: 00000000ffff0ff0
DR7: 0000000000000400
kernel: [ 600.687472] Process zadara_vac (pid: 15622, threadinfo
ffff88006bae4000, task ffff8800567aae00)
kernel: [ 600.687472] Stack:
kernel: [ 600.687472] ffff88006bae5d98 ffff88006bae4000
00000003679c1065 ffff88007fd83fc0
kernel: [ 600.687472] ffff880000000029 0000000000000000
ffff88007b27f0c0 ffff88006bae5f28
kernel: [ 600.687472] ffff880057bf3000 00000000ffffff9c
ffff88006bae5d98 ffffffff8118f2c8
kernel: [ 600.687472] Call Trace:
kernel: [ 600.687472] [<ffffffff8118f2c8>] get_empty_filp+0x58/0x180
kernel: [ 600.687472] [<ffffffff8119cb18>] path_openat+0x48/0x4d0
kernel: [ 600.687472] [<ffffffff81189e0b>] ? create_object+0x22b/0x2e0
kernel: [ 600.687472] [<ffffffff8119d742>] do_filp_open+0x42/0xa0
kernel: [ 600.687472] [<ffffffff816a2ace>] ? _raw_spin_lock+0xe/0x20
kernel: [ 600.687472] [<ffffffff811aace5>] ? __alloc_fd+0xe5/0x170
kernel: [ 600.687472] [<ffffffff8118d37e>] do_sys_open+0xfe/0x1e0
kernel: [ 600.687472] [<ffffffff816a6fce>] ? do_page_fault+0xe/0x10
kernel: [ 600.687472] [<ffffffff8118d481>] sys_open+0x21/0x30
kernel: [ 600.687472] [<ffffffff816ab919>] system_call_fastpath+0x16/0x1b
kernel: [ 600.687472] Code: 00 4d 8b 04 24 65 4c 03 04 25 08 dc 00
00 49 8b 50 08 4d 8b 28 4d 85 ed 0f 84 03 01 00 00 49 63 44 24 20 49
8b 3c 24 48 8d 4a 01 <49> 8b 5c 05 00 4c 89 e8 65 48 0f c7 0f 0f 94 c0
84 c0 74 c2 49
kernel: [ 600.687472] RIP [<ffffffff8117ee4b>] kmem_cache_alloc+0x5b/0x170
kernel: [ 600.687472] RSP <ffff88006bae5d28>
kernel: [ 600.687472] CR2: 0000000000010001
kernel: [ 600.745966] ---[ end trace ded81b9e3753ac50 ]---
kernel: [ 600.769821] BUG: unable to handle kernel paging request at
0000000000010001
kernel: [ 600.771366] IP: [<ffffffff8117ee4b>] kmem_cache_alloc+0x5b/0x170
kernel: [ 600.772674] PGD 78bc3067 PUD 6bb27067 PMD 0
kernel: [ 600.773697] Oops: 0000 [#4] SMP
kernel: [ 600.773708] Modules linked in: xfs raid1 xfrm_user
xfrm4_tunnel tunnel4 ipcomp xfrm_ipcomp esp4 ah4 scst_vdisk(O)
iscsi_scst(O) scst(O) dm_zcache(O) dm_btrfs(O) btrfs(O) libcrc32c
dm_iostat(O) md_mod deflate zlib_deflate ctr twofish_generic
twofish_x86_64_3way glue_helper lrw xts gf128mul twofish_x86_64
twofish_common camellia_generic serpent_generic blowfish_generic
blowfish_x86_64 blowfish_common cast5_generic cast_common des_generic
xcbc rmd160 sha512_generic crypto_null af_key xfrm_algo dm_round_robin
kvm vfat fat ppdev nfsd(O) nfs_acl microcode dm_multipath psmouse
serio_raw parport_pc i2c_piix4 mac_hid lp nfsv4 auth_rpcgss parport
nfs fscache lockd sunrpc floppy
kernel: [ 600.773708] CPU 3
kernel: [ 600.773708] Pid: 15571, comm: md1_raid1 Tainted: G D
O 3.8.13-557-generic #1382000791 Bochs Bochs
kernel: [ 600.773708] RIP: 0010:[<ffffffff8117ee4b>]
[<ffffffff8117ee4b>] kmem_cache_alloc+0x5b/0x170
kernel: [ 600.773708] RSP: 0018:ffff880069fa1868 EFLAGS: 00010202
kernel: [ 600.773708] RAX: 0000000000000000 RBX: ffff88005d28cd80
RCX: 000000000003b9be
kernel: [ 600.773708] RDX: 000000000003b9bd RSI: 0000000000011200
RDI: 0000000000016c60
kernel: [ 600.773708] RBP: ffff880069fa18b8 R08: ffff88007fd96c60
R09: 0000000000000001
kernel: [ 600.773708] R10: 0000000000000000 R11: 0000000000000001
R12: ffff88007d001900
kernel: [ 600.773708] R13: 0000000000010001 R14: 0000000000011200
R15: ffffffff81131ed5
kernel: [ 600.773708] FS: 0000000000000000(0000)
GS:ffff88007fd80000(0000) knlGS:0000000000000000
kernel: [ 600.773708] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
kernel: [ 600.773708] CR2: 0000000000010001 CR3: 000000006baf9000
CR4: 00000000000006e0
kernel: [ 600.773708] DR0: 0000000000000000 DR1: 0000000000000000
DR2: 0000000000000000
kernel: [ 600.773708] DR3: 0000000000000000 DR6: 00000000ffff0ff0
DR7: 0000000000000400
kernel: [ 600.773708] Process md1_raid1 (pid: 15571, threadinfo
ffff880069fa0000, task ffff880054758000)
kernel: [ 600.773708] Stack:
kernel: [ 600.773708] ffffffffa02e4ae8 ffff880069fa1888
ffffffff810bb842 0000000000000000
kernel: [ 600.773708] ffff880069fa1898 ffff88005d28cd80
0000000000011210 ffff88005d28cdb0
kernel: [ 600.773708] ffff880069fa1908 ffff880054758000
ffff880069fa18c8 ffffffff81131ed5
kernel: [ 600.773708] Call Trace:
kernel: [ 600.773708] [<ffffffff810bb842>] ? __module_text_address+0x12/0x70
kernel: [ 600.773708] [<ffffffff81131ed5>] mempool_alloc_slab+0x15/0x20
kernel: [ 600.773708] [<ffffffff8113222e>] mempool_alloc+0x5e/0x160
kernel: [ 600.773708] [<ffffffff811c4f13>] bio_alloc_bioset+0x43/0x110
kernel: [ 600.773708] [<ffffffff81131ed5>] ? mempool_alloc_slab+0x15/0x20
kernel: [ 600.773708] [<ffffffff8153d5de>] alloc_tio.isra.15+0x2e/0x60
kernel: [ 600.773708] [<ffffffff8153dab8>] __clone_and_map+0x248/0x2d0
kernel: [ 600.773708] [<ffffffff81189e0b>] ? create_object+0x22b/0x2e0
kernel: [ 600.773708] [<ffffffff8153e378>]
__split_and_process_bio+0x178/0x1b0
kernel: [ 600.773708] [<ffffffff8153e4ac>] _dm_request.isra.24+0xfc/0x150
kernel: [ 600.773708] [<ffffffff8153e528>] dm_request+0x28/0x40
kernel: [ 600.773708] [<ffffffff813103c7>]
generic_make_request.part.55+0x77/0xb0
kernel: [ 600.773708] [<ffffffff81310958>] generic_make_request+0x68/0x70
kernel: [ 600.773708] [<ffffffff813109db>] submit_bio+0x7b/0x160
kernel: [ 600.773708] [<ffffffffa02d77b6>] md_super_write+0x86/0xb0 [md_mod]
kernel: [ 600.773708] [<ffffffffa02dba33>] write_sb_page+0x73/0x1f0 [md_mod]
kernel: [ 600.773708] [<ffffffffa02dd166>] write_page+0x116/0x130 [md_mod]
kernel: [ 600.773708] [<ffffffffa02d5924>] ?
super_1_sync+0x3f4/0x4e0 [md_mod]
kernel: [ 600.773708] [<ffffffffa02dd26d>]
bitmap_update_sb.part.19+0xed/0xf0 [md_mod]
kernel: [ 600.773708] [<ffffffffa02dcf1f>]
bitmap_update_sb+0x2f/0x40 [md_mod]
kernel: [ 600.773708] [<ffffffffa02d7a20>] md_update_sb+0x1b0/0x4f0 [md_mod]
kernel: [ 600.773708] [<ffffffffa02de6b5>] ?
bitmap_daemon_work+0x215/0x340 [md_mod]
kernel: [ 600.773708] [<ffffffffa02d8baa>]
md_check_recovery.part.50+0x1da/0x530 [md_mod]
kernel: [ 600.773708] [<ffffffff8106a8ba>] ? del_timer_sync+0x5a/0x70
kernel: [ 600.773708] [<ffffffffa02d8f15>]
md_check_recovery+0x15/0x20 [md_mod]
kernel: [ 600.773708] [<ffffffffa055eed2>] raid1d+0x22/0x180 [raid1]
kernel: [ 600.773708] [<ffffffff81045cd9>] ? default_spin_lock_flags+0x9/0x10
kernel: [ 600.773708] [<ffffffff81045cd9>] ? default_spin_lock_flags+0x9/0x10
kernel: [ 600.773708] [<ffffffffa02d084d>] md_thread+0x10d/0x140 [md_mod]
kernel: [ 600.773708] [<ffffffff8107ead0>] ? add_wait_queue+0x60/0x60
kernel: [ 600.773708] [<ffffffffa02d0740>] ?
md_rdev_init+0x140/0x140 [md_mod]
kernel: [ 600.773708] [<ffffffff8107df10>] kthread+0xc0/0xd0
kernel: [ 600.773708] [<ffffffff8107de50>] ? flush_kthread_worker+0xb0/0xb0
kernel: [ 600.773708] [<ffffffff816ab86c>] ret_from_fork+0x7c/0xb0
kernel: [ 600.773708] [<ffffffff8107de50>] ? flush_kthread_worker+0xb0/0xb0
kernel: [ 600.773708] Code: 00 4d 8b 04 24 65 4c 03 04 25 08 dc 00
00 49 8b 50 08 4d 8b 28 4d 85 ed 0f 84 03 01 00 00 49 63 44 24 20 49
8b 3c 24 48 8d 4a 01 <49> 8b 5c 05 00 4c 89 e8 65 48 0f c7 0f 0f 94 c0
84 c0 74 c2 49
kernel: [ 600.773708] RIP [<ffffffff8117ee4b>] kmem_cache_alloc+0x5b/0x170
kernel: [ 600.773708] RSP <ffff880069fa1868>
kernel: [ 600.773708] CR2: 0000000000010001
kernel: [ 600.869584] ---[ end trace ded81b9e3753ac51 ]---
kernel: [ 603.924718] BUG: unable to handle kernel paging request at
0000000000010001
kernel: [ 603.926316] IP: [<ffffffff8117ee4b>] kmem_cache_alloc+0x5b/0x170
kernel: [ 603.927624] PGD 79735067 PUD 79734067 PMD 0
kernel: [ 603.928456] Oops: 0000 [#5] SMP
kernel: [ 603.928456] Modules linked in: xfs raid1 xfrm_user
xfrm4_tunnel tunnel4 ipcomp xfrm_ipcomp esp4 ah4 scst_vdisk(O)
iscsi_scst(O) scst(O) dm_zcache(O) dm_btrfs(O) btrfs(O) libcrc32c
dm_iostat(O) md_mod deflate zlib_deflate ctr twofish_generic
twofish_x86_64_3way glue_helper lrw xts gf128mul twofish_x86_64
twofish_common camellia_generic serpent_generic blowfish_generic
blowfish_x86_64 blowfish_common cast5_generic cast_common des_generic
xcbc rmd160 sha512_generic crypto_null af_key xfrm_algo dm_round_robin
kvm vfat fat ppdev nfsd(O) nfs_acl microcode dm_multipath psmouse
serio_raw parport_pc i2c_piix4 mac_hid lp nfsv4 auth_rpcgss parport
nfs fscache lockd sunrpc floppy
kernel: [ 603.928456] CPU 3
kernel: [ 603.928456] Pid: 4922, comm: PassengerHelper Tainted: G
D O 3.8.13-557-generic #1382000791 Bochs Bochs
kernel: [ 603.928456] RIP: 0010:[<ffffffff8117ee4b>]
[<ffffffff8117ee4b>] kmem_cache_alloc+0x5b/0x170
kernel: [ 603.928456] RSP: 0018:ffff88006b8a3e08 EFLAGS: 00010202
kernel: [ 603.928456] RAX: 0000000000000000 RBX: 0000000000000000
RCX: 000000000003b9be
kernel: [ 603.928456] RDX: 000000000003b9bd RSI: 00000000000080d0
RDI: 0000000000016c60
kernel: [ 603.928456] RBP: ffff88006b8a3e58 R08: ffff88007fd96c60
R09: 0000000000000000
kernel: [ 603.928456] R10: 000000000000c6e0 R11: ffff88007fd80000
R12: ffff88007d001900
kernel: [ 603.928456] R13: 0000000000010001 R14: 00000000000080d0
R15: ffffffff8118f2c8
kernel: [ 603.928456] FS: 00007fb3c2462700(0000)
GS:ffff88007fd80000(0000) knlGS:0000000000000000
kernel: [ 603.928456] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
kernel: [ 603.928456] CR2: 0000000000010001 CR3: 000000007af80000
CR4: 00000000000006e0
kernel: [ 603.928456] DR0: 0000000000000000 DR1: 0000000000000000
DR2: 0000000000000000
kernel: [ 603.928456] DR3: 0000000000000000 DR6: 00000000ffff0ff0
DR7: 0000000000000400
kernel: [ 603.928456] Process PassengerHelper (pid: 4922, threadinfo
ffff88006b8a2000, task ffff88007a085c00)
kernel: [ 603.928456] Stack:
kernel: [ 603.928456] ffff88006b8a3e68 ffffffff8117ee9b
ffff88007c9ac500 ffffffff811a55e4
kernel: [ 603.928456] 00000000000000c0 0000000000000000
ffff8800366a59c0 0000000000000002
kernel: [ 603.928456] ffffffff8181bca0 0000000000000003
ffff88006b8a3e78 ffffffff8118f2c8
kernel: [ 603.928456] Call Trace:
kernel: [ 603.928456] [<ffffffff8117ee9b>] ? kmem_cache_alloc+0xab/0x170
kernel: [ 603.928456] [<ffffffff811a55e4>] ? __d_alloc+0x34/0x180
kernel: [ 603.928456] [<ffffffff8118f2c8>] get_empty_filp+0x58/0x180
kernel: [ 603.928456] [<ffffffff8118f41b>] alloc_file+0x2b/0xd0
kernel: [ 603.928456] [<ffffffff81197129>] create_pipe_files+0x89/0x130
kernel: [ 603.928456] [<ffffffff81197212>] __do_pipe_flags+0x42/0xf0
kernel: [ 603.928456] [<ffffffff81197330>] sys_pipe2+0x20/0xa0
kernel: [ 603.928456] [<ffffffff816a3df5>] ?
do_device_not_available+0x15/0x20
kernel: [ 603.928456] [<ffffffff816aca4e>] ? device_not_available+0x1e/0x30
kernel: [ 603.928456] [<ffffffff811973c0>] sys_pipe+0x10/0x20
kernel: [ 603.928456] [<ffffffff816ab919>] system_call_fastpath+0x16/0x1b
kernel: [ 603.928456] Code: 00 4d 8b 04 24 65 4c 03 04 25 08 dc 00
00 49 8b 50 08 4d 8b 28 4d 85 ed 0f 84 03 01 00 00 49 63 44 24 20 49
8b 3c 24 48 8d 4a 01 <49> 8b 5c 05 00 4c 89 e8 65 48 0f c7 0f 0f 94 c0
84 c0 74 c2 49
kernel: [ 603.928456] RIP [<ffffffff8117ee4b>] kmem_cache_alloc+0x5b/0x170
kernel: [ 603.928456] RSP <ffff88006b8a3e08>
kernel: [ 603.928456] CR2: 0000000000010001
kernel: [ 603.994540] ---[ end trace ded81b9e3753ac52 ]---
kernel: [ 645.662431] BUG: unable to handle kernel paging request at
0000000000010001
kernel: [ 645.663218] IP: [<ffffffff8117dd0f>]
kmem_cache_alloc_trace+0x5f/0x160
kernel: [ 645.663824] PGD 5eeda067 PUD 5b684067 PMD 0
kernel: [ 645.664272] Oops: 0000 [#6] SMP
kernel: [ 645.664603] Modules linked in: xfs raid1 xfrm_user
xfrm4_tunnel tunnel4 ipcomp xfrm_ipcomp esp4 ah4 scst_vdisk(O)
iscsi_scst(O) scst(O) dm_zcache(O) dm_btrfs(O) btrfs(O) libcrc32c
dm_iostat(O) md_mod deflate zlib_deflate ctr twofish_generic
twofish_x86_64_3way glue_helper lrw xts gf128mul twofish_x86_64
twofish_common camellia_generic serpent_generic blowfish_generic
blowfish_x86_64 blowfish_common cast5_generic cast_common des_generic
xcbc rmd160 sha512_generic crypto_null af_key xfrm_algo dm_round_robin
kvm vfat fat ppdev nfsd(O) nfs_acl microcode dm_multipath psmouse
serio_raw parport_pc i2c_piix4 mac_hid lp nfsv4 auth_rpcgss parport
nfs fscache lockd sunrpc floppy
kernel: [ 645.666264] CPU 3
kernel: [ 645.666264] Pid: 26392, comm: bash Tainted: G D O
3.8.13-557-generic #1382000791 Bochs Bochs
kernel: [ 645.666264] RIP: 0010:[<ffffffff8117dd0f>]
[<ffffffff8117dd0f>] kmem_cache_alloc_trace+0x5f/0x160
kernel: [ 645.666264] RSP: 0018:ffff88005bddbe58 EFLAGS: 00010202
kernel: [ 645.666264] RAX: 0000000000000000 RBX: ffff880057bf2020
RCX: 000000000003b9be
kernel: [ 645.666264] RDX: 000000000003b9bd RSI: 00000000000080d0
RDI: 0000000000016c60
kernel: [ 645.666264] RBP: ffff88005bddbea8 R08: ffff88007fd96c60
R09: fffff2ff8f908bd0
kernel: [ 645.666264] R10: 8080808080808080 R11: 0000000000000000
R12: ffff88007d001900
kernel: [ 645.666264] R13: 0000000000010001 R14: 00000000000080d0
R15: ffffffff81195237
kernel: [ 645.666264] FS: 00007fb039494700(0000)
GS:ffff88007fd80000(0000) knlGS:0000000000000000
kernel: [ 645.666264] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
kernel: [ 645.666264] CR2: 0000000000010001 CR3: 0000000054649000
CR4: 00000000000006e0
kernel: [ 645.666264] DR0: 0000000000000000 DR1: 0000000000000000
DR2: 0000000000000000
kernel: [ 645.666264] DR3: 0000000000000000 DR6: 00000000ffff0ff0
DR7: 0000000000000400
kernel: [ 645.666264] Process bash (pid: 26392, threadinfo
ffff88005bdda000, task ffff88005dbf0000)
kernel: [ 645.666264] Stack:
kernel: [ 645.666264] ffff880057bf2020 ffff88005bddbed0
0000000000000100 ffffffff8105866b
kernel: [ 645.666264] 0000000000000000 ffff880057bf2020
0000000000000000 000000000151e008
kernel: [ 645.666264] 000000000151e008 0000000001399448
ffff88005bddbf08 ffffffff81195237
kernel: [ 645.666264] Call Trace:
kernel: [ 645.666264] [<ffffffff8105866b>] ? unshare_files+0x3b/0xb0
kernel: [ 645.666264] [<ffffffff81195237>]
do_execve_common.isra.24+0xc7/0x330
kernel: [ 645.666264] [<ffffffff811954b8>] do_execve+0x18/0x20
kernel: [ 645.666264] [<ffffffff8119575d>] sys_execve+0x3d/0x60
kernel: [ 645.666264] [<ffffffff816abe89>] stub_execve+0x69/0xc0
kernel: [ 645.666264] Code: 00 4d 8b 04 24 65 4c 03 04 25 08 dc 00
00 49 8b 50 08 4d 8b 28 4d 85 ed 0f 84 ef 00 00 00 49 63 44 24 20 49
8b 3c 24 48 8d 4a 01 <49> 8b 5c 05 00 4c 89 e8 65 48 0f c7 0f 0f 94 c0
84 c0 74 c2 49
kernel: [ 645.666264] RIP [<ffffffff8117dd0f>]
kmem_cache_alloc_trace+0x5f/0x160
kernel: [ 645.666264] RSP <ffff88005bddbe58>
kernel: [ 645.666264] CR2: 0000000000010001
kernel: [ 645.689292] ---[ end trace ded81b9e3753ac53 ]---
kernel: [ 661.264133] BUG: unable to handle kernel paging request at
0000000000010001
kernel: [ 661.264803] IP: [<ffffffff8117db95>]
kmem_cache_alloc_node+0x95/0x1b0
kernel: [ 661.265405] PGD 0
kernel: [ 661.265610] Oops: 0000 [#7] SMP
kernel: [ 661.265935] Modules linked in: xfs raid1 xfrm_user
xfrm4_tunnel tunnel4 ipcomp xfrm_ipcomp esp4 ah4 scst_vdisk(O)
iscsi_scst(O) scst(O) dm_zcache(O) dm_btrfs(O) btrfs(O) libcrc32c
dm_iostat(O) md_mod deflate zlib_deflate ctr twofish_generic
twofish_x86_64_3way glue_helper lrw xts gf128mul twofish_x86_64
twofish_common camellia_generic serpent_generic blowfish_generic
blowfish_x86_64 blowfish_common cast5_generic cast_common des_generic
xcbc rmd160 sha512_generic crypto_null af_key xfrm_algo dm_round_robin
kvm vfat fat ppdev nfsd(O) nfs_acl microcode dm_multipath psmouse
serio_raw parport_pc i2c_piix4 mac_hid lp nfsv4 auth_rpcgss parport
nfs fscache lockd sunrpc floppy
kernel: [ 661.268028] CPU 3
kernel: [ 661.268028] Pid: 6570, comm: udevd Tainted: G D O
3.8.13-557-generic #1382000791 Bochs Bochs
kernel: [ 661.268028] RIP: 0010:[<ffffffff8117db95>]
[<ffffffff8117db95>] kmem_cache_alloc_node+0x95/0x1b0
kernel: [ 661.268028] RSP: 0018:ffff88005b08dae8 EFLAGS: 00010246
kernel: [ 661.268028] RAX: 0000000000000000 RBX: ffff880069dcb740
RCX: 000000000003b9be
kernel: [ 661.268028] RDX: 000000000003b9bd RSI: 00000000000004d0
RDI: 0000000000016c60
kernel: [ 661.268028] RBP: ffff88005b08db58 R08: ffff88007fd96c60
R09: ffffffff81588007
kernel: [ 661.268028] R10: 0000000000004000 R11: 0000000000000246
R12: ffff88007d001900
kernel: [ 661.268028] R13: 0000000000010001 R14: 00000000ffffffff
R15: 00000000000004d0
kernel: [ 661.268028] FS: 00007f33c38f97c0(0000)
GS:ffff88007fd80000(0000) knlGS:0000000000000000
kernel: [ 661.268028] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
kernel: [ 661.268028] CR2: 0000000000010001 CR3: 000000005b082000
CR4: 00000000000006e0
kernel: [ 661.268028] DR0: 0000000000000000 DR1: 0000000000000000
DR2: 0000000000000000
kernel: [ 661.268028] DR3: 0000000000000000 DR6: 00000000ffff0ff0
DR7: 0000000000000400
kernel: [ 661.268028] Process udevd (pid: 6570, threadinfo
ffff88005b08c000, task ffff8800684fdc00)
kernel: [ 661.268028] Stack:
kernel: [ 661.268028] 0000000000000286 ffff88005b7ac1d0
ffff88005b08db18 ffffffff81588007
kernel: [ 661.268028] 0000000000000001 ffff88005b08dbc8
ffff88005b08db68 ffffffff811a07f7
kernel: [ 661.268028] 00000099f2d3c9cb ffff880069dcb740
00000000000004d0 00000000ffffffff
kernel: [ 661.268028] Call Trace:
kernel: [ 661.268028] [<ffffffff81588007>] ? __alloc_skb+0x57/0x2a0
kernel: [ 661.268028] [<ffffffff811a07f7>] ? poll_freewait+0x57/0xe0
kernel: [ 661.268028] [<ffffffff81588007>] __alloc_skb+0x57/0x2a0
kernel: [ 661.268028] [<ffffffff8158065a>] sock_alloc_send_pskb+0x11a/0x2b0
kernel: [ 661.268028] [<ffffffff816268ad>] unix_dgram_sendmsg+0x1bd/0x660
kernel: [ 661.268028] [<ffffffff81334a9d>] ? cpumask_any_but+0x2d/0x40
kernel: [ 661.268028] [<ffffffff8104e469>] ? flush_tlb_page+0x49/0xb0
kernel: [ 661.268028] [<ffffffff8157b0a2>] sock_sendmsg+0xd2/0xf0
kernel: [ 661.268028] [<ffffffff816a6d08>] ? __do_page_fault+0x2a8/0x560
kernel: [ 661.268028] [<ffffffff8157e50d>] sys_sendto+0x13d/0x190
kernel: [ 661.268028] [<ffffffff81090d6a>] ? wake_up_new_task+0xfa/0x190
kernel: [ 661.268028] [<ffffffff810ac71c>] ? ktime_get_ts+0x4c/0xe0
kernel: [ 661.268028] [<ffffffff816a6fce>] ? do_page_fault+0xe/0x10
kernel: [ 661.268028] [<ffffffff816a66a5>] ? do_async_page_fault+0x35/0x90
kernel: [ 661.268028] [<ffffffff816ab919>] system_call_fastpath+0x16/0x1b
kernel: [ 661.268028] Code: 1d 4c 89 c9 44 89 f2 44 89 fe 4c 89 e7
e8 ca 29 51 00 49 89 c5 eb 31 0f 1f 80 00 00 00 00 49 63 44 24 20 49
8b 3c 24 48 8d 4a 01 <49> 8b 5c 05 00 4c 89 e8 65 48 0f c7 0f 0f 94 c0
84 c0 74 93 49
kernel: [ 661.268028] RIP [<ffffffff8117db95>]
kmem_cache_alloc_node+0x95/0x1b0
kernel: [ 661.268028] RSP <ffff88005b08dae8>
kernel: [ 661.268028] CR2: 0000000000010001
kernel: [ 661.295443] ---[ end trace ded81b9e3753ac54 ]---
kernel: [ 705.043870] BUG: unable to handle kernel paging request at
0000000000010001
kernel: [ 705.044572] IP: [<ffffffff8117ee4b>] kmem_cache_alloc+0x5b/0x170
kernel: [ 705.045145] PGD 55bbe067 PUD 57794067 PMD 0
kernel: [ 705.045603] Oops: 0000 [#8] SMP
kernel: [ 705.045938] Modules linked in: xfs raid1 xfrm_user
xfrm4_tunnel tunnel4 ipcomp xfrm_ipcomp esp4 ah4 scst_vdisk(O)
iscsi_scst(O) scst(O) dm_zcache(O) dm_btrfs(O) btrfs(O) libcrc32c
dm_iostat(O) md_mod deflate zlib_deflate ctr twofish_generic
twofish_x86_64_3way glue_helper lrw xts gf128mul twofish_x86_64
twofish_common camellia_generic serpent_generic blowfish_generic
blowfish_x86_64 blowfish_common cast5_generic cast_common des_generic
xcbc rmd160 sha512_generic crypto_null af_key xfrm_algo dm_round_robin
kvm vfat fat ppdev nfsd(O) nfs_acl microcode dm_multipath psmouse
serio_raw parport_pc i2c_piix4 mac_hid lp nfsv4 auth_rpcgss parport
nfs fscache lockd sunrpc floppy
kernel: [ 705.047420] CPU 3
kernel: [ 705.047420] Pid: 8072, comm: bash Tainted: G D O
3.8.13-557-generic #1382000791 Bochs Bochs
kernel: [ 705.047420] RIP: 0010:[<ffffffff8117ee4b>]
[<ffffffff8117ee4b>] kmem_cache_alloc+0x5b/0x170
kernel: [ 705.047420] RSP: 0018:ffff880055909e08 EFLAGS: 00010202
kernel: [ 705.047420] RAX: 0000000000000000 RBX: 0000000000000000
RCX: 000000000003b9be
kernel: [ 705.047420] RDX: 000000000003b9bd RSI: 00000000000080d0
RDI: 0000000000016c60
kernel: [ 705.047420] RBP: ffff880055909e58 R08: ffff88007fd96c60
R09: 0000000000000000
kernel: [ 705.047420] R10: 000000000000c6e0 R11: ffff88007fd80000
R12: ffff88007d001900
kernel: [ 705.047420] R13: 0000000000010001 R14: 00000000000080d0
R15: ffffffff8118f2c8
kernel: [ 705.047420] FS: 00007f5bfcd56700(0000)
GS:ffff88007fd80000(0000) knlGS:0000000000000000
kernel: [ 705.047420] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
kernel: [ 705.047420] CR2: 0000000000010001 CR3: 0000000057785000
CR4: 00000000000006e0
kernel: [ 705.047420] DR0: 0000000000000000 DR1: 0000000000000000
DR2: 0000000000000000
kernel: [ 705.047420] DR3: 0000000000000000 DR6: 00000000ffff0ff0
DR7: 0000000000000400
kernel: [ 705.047420] Process bash (pid: 8072, threadinfo
ffff880055908000, task ffff88005ad45c00)
kernel: [ 705.047420] Stack:
kernel: [ 705.047420] ffff880055909e68 ffffffff8117ee9b
ffff880055909e28 ffffffff811a55e4
kernel: [ 705.047420] 00000000000000c0 0000000000000000
ffff88007936d240 0000000000000002
kernel: [ 705.047420] ffffffff8181bca0 00000000ffffffff
ffff880055909e78 ffffffff8118f2c8
kernel: [ 705.047420] Call Trace:
kernel: [ 705.047420] [<ffffffff8117ee9b>] ? kmem_cache_alloc+0xab/0x170
kernel: [ 705.047420] [<ffffffff811a55e4>] ? __d_alloc+0x34/0x180
kernel: [ 705.047420] [<ffffffff8118f2c8>] get_empty_filp+0x58/0x180
kernel: [ 705.047420] [<ffffffff8118f41b>] alloc_file+0x2b/0xd0
kernel: [ 705.047420] [<ffffffff81197129>] create_pipe_files+0x89/0x130
kernel: [ 705.047420] [<ffffffff81197212>] __do_pipe_flags+0x42/0xf0
kernel: [ 705.047420] [<ffffffff81197330>] sys_pipe2+0x20/0xa0
kernel: [ 705.047420] [<ffffffff8106fa78>] ? sys_rt_sigprocmask+0xa8/0xd0
kernel: [ 705.047420] [<ffffffff811973c0>] sys_pipe+0x10/0x20
kernel: [ 705.047420] [<ffffffff816ab919>] system_call_fastpath+0x16/0x1b
kernel: [ 705.047420] Code: 00 4d 8b 04 24 65 4c 03 04 25 08 dc 00
00 49 8b 50 08 4d 8b 28 4d 85 ed 0f 84 03 01 00 00 49 63 44 24 20 49
8b 3c 24 48 8d 4a 01 <49> 8b 5c 05 00 4c 89 e8 65 48 0f c7 0f 0f 94 c0
84 c0 74 c2 49
kernel: [ 705.047420] RIP [<ffffffff8117ee4b>] kmem_cache_alloc+0x5b/0x170
kernel: [ 705.047420] RSP <ffff880055909e08>
kernel: [ 705.047420] CR2: 0000000000010001
[2]
kernel: [ 381.277746] XFS (dm-22): metadata I/O error: block
0x7600028 ("xlog_iodone") error 125 numblks 64
kernel: [ 381.279335] XFS (dm-22): xfs_do_force_shutdown(0x2) called
from line 1115 of file
/mnt/compile/linux-stable/source/fs/xfs/xfs_log.c. Return address =
0xffffffffa05d5fa1
kernel: [ 381.279392] XFS (dm-22): Log I/O Error Detected. Shutting
down filesystem
kernel: [ 381.279488] XFS (dm-22): Unable to update superblock
counters. Freespace may not be correct on next mount.
kernel: [ 381.279495] XFS (dm-22): xfs_log_force: error 5 returned.
kernel: [ 381.281015] XFS (dm-22): Please umount the filesystem and
rectify the problem(s)
kernel: [ 381.294706] [29579]dm_iostat [dm_iostat_dtr:157] Removing
iostats (253:22) from 253:21
kernel: [ 381.508073] kmemleak: Cannot insert 0xffff88007947bb00
into the object search tree (overlaps existing)
kernel: [ 381.508105] Pid: 16141, comm: udevd Tainted: G O
3.8.13-557-generic #1382000791
kernel: [ 381.508109] Call Trace:
kernel: [ 381.508140] [<ffffffff81189e66>] create_object+0x286/0x2e0
kernel: [ 381.508149] [<ffffffff8118f2c8>] ? get_empty_filp+0x58/0x180
kernel: [ 381.508172] [<ffffffff816798e6>] kmemleak_alloc+0x26/0x50
kernel: [ 381.508181] [<ffffffff8117ee9b>] kmem_cache_alloc+0xab/0x170
kernel: [ 381.508187] [<ffffffff8118f2c8>] get_empty_filp+0x58/0x180
kernel: [ 381.508196] [<ffffffff8119cb18>] path_openat+0x48/0x4d0
kernel: [ 381.508201] [<ffffffff81189e0b>] ? create_object+0x22b/0x2e0
kernel: [ 381.508206] [<ffffffff8119d742>] do_filp_open+0x42/0xa0
kernel: [ 381.508214] [<ffffffff816a2ace>] ? _raw_spin_lock+0xe/0x20
kernel: [ 381.508238] [<ffffffff811aace5>] ? __alloc_fd+0xe5/0x170
kernel: [ 381.508244] [<ffffffff8118d37e>] do_sys_open+0xfe/0x1e0
kernel: [ 381.508256] [<ffffffff81014a24>] ? do_notify_resume+0x74/0x80
kernel: [ 381.508262] [<ffffffff8118d4a4>] sys_openat+0x14/0x20
kernel: [ 381.508269] [<ffffffff816ab919>] system_call_fastpath+0x16/0x1b
kernel: [ 381.508282] kmemleak: Kernel memory leak detector disabled
kernel: [ 381.508287] kmemleak: Object 0xffff88007947ba40 (size 448):
kernel: [ 381.508290] kmemleak: comm "modprobe", pid 3708, jiffies
4294921709
kernel: [ 381.508292] kmemleak: min_count = 1
kernel: [ 381.508294] kmemleak: count = 0
kernel: [ 381.508296] kmemleak: flags = 0x1
kernel: [ 381.508298] kmemleak: checksum = 0
kernel: [ 381.508300] kmemleak: backtrace:
kernel: [ 381.508302] [<ffffffff816798e6>] kmemleak_alloc+0x26/0x50
kernel: [ 381.508309] [<ffffffff8117ee9b>] kmem_cache_alloc+0xab/0x170
kernel: [ 381.508314] [<ffffffff81131ed5>] mempool_alloc_slab+0x15/0x20
kernel: [ 381.508328] [<ffffffff81132131>] mempool_create_node+0xe1/0x160
kernel: [ 381.508334] [<ffffffff811321ca>] mempool_create+0x1a/0x20
kernel: [ 381.508339] [<ffffffff811c4d04>] bioset_create+0x74/0xe0
kernel: [ 381.508346] [<ffffffffa047b5b2>]
init_module+0x122/0x240 [dm_zcache]
kernel: [ 381.508368] [<ffffffff8100206f>] do_one_initcall+0x3f/0x170
kernel: [ 381.508375] [<ffffffff8168d5de>] do_init_module+0x80/0x1d1
kernel: [ 381.508382] [<ffffffff810c12ca>] load_module+0x3aa/0x530
kernel: [ 381.508399] [<ffffffff810c14cb>] sys_init_module+0x7b/0x80
kernel: [ 381.508405] [<ffffffff816ab919>] system_call_fastpath+0x16/0x1b
kernel: [ 381.508409] [<ffffffffffffffff>] 0xffffffffffffffff
kernel: [ 381.508832] kmemleak: Automatic memory scanning thread ended
kernel: [ 381.508556] BUG: unable to handle kernel paging request at
000000170000000a
kernel: [ 381.511347] IP: [<ffffffff8117ee4b>] kmem_cache_alloc+0x5b/0x170
kernel: [ 381.512031] PGD 0
kernel: [ 381.512031] Oops: 0000 [#1] SMP
kernel: [ 381.512031] Modules linked in: xfs raid1 xfrm_user
xfrm4_tunnel tunnel4 ipcomp xfrm_ipcomp esp4 ah4 scst_vdisk(O)
iscsi_scst(O) scst(O) dm_zcache(O) dm_btrfs(O) btrfs(O) libcrc32c
dm_iostat(O) md_mod deflate zlib_deflate ctr twofish_generic
twofish_x86_64_3way glue_helper lrw xts gf128mul twofish_x86_64
twofish_common camellia_generic serpent_generic blowfish_generic
blowfish_x86_64 blowfish_common cast5_generic cast_common des_generic
xcbc rmd160 sha512_generic dm_round_robin crypto_null af_key xfrm_algo
kvm ppdev vfat fat dm_multipath nfsd(O) nfs_acl psmouse serio_raw
parport_pc i2c_piix4 microcode mac_hid nfsv4 auth_rpcgss nfs fscache
lockd lp sunrpc parport floppy
kernel: [ 381.524857] CPU 3
kernel: [ 381.524857] Pid: 16141, comm: udevd Tainted: G O
3.8.13-557-generic #1382000791 Bochs Bochs
kernel: [ 381.524857] RIP: 0010:[<ffffffff8117ee4b>]
[<ffffffff8117ee4b>] kmem_cache_alloc+0x5b/0x170
kernel: [ 381.524857] RSP: 0018:ffff88005c617d28 EFLAGS: 00010206
kernel: [ 381.524857] RAX: 0000000000000000 RBX: 0000000000000000
RCX: 00000000000427ee
kernel: [ 381.524857] RDX: 00000000000427ed RSI: 00000000000080d0
RDI: 0000000000016c60
kernel: [ 381.524857] RBP: ffff88005c617d78 R08: ffff88007fd96c60
R09: 0000000000000000
kernel: [ 381.524857] R10: 8080808080808080 R11: 0000000000000000
R12: ffff88007d001900
kernel: [ 381.524857] R13: 000000170000000a R14: 00000000000080d0
R15: ffffffff8118f2c8
kernel: [ 381.524857] FS: 00007f77822c37c0(0000)
GS:ffff88007fd80000(0000) knlGS:0000000000000000
kernel: [ 381.524857] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
kernel: [ 381.544074] CR2: 000000170000000a CR3: 000000003687c000
CR4: 00000000000006e0
kernel: [ 381.544074] DR0: 0000000000000000 DR1: 0000000000000000
DR2: 0000000000000000
kernel: [ 381.544074] DR3: 0000000000000000 DR6: 00000000ffff0ff0
DR7: 0000000000000400
kernel: [ 381.548728] Process udevd (pid: 16141, threadinfo
ffff88005c616000, task ffff88005385c500)
kernel: [ 381.548728] Stack:
kernel: [ 381.548728] 0000000000000001 00007fff2038ac28
0000000000000000 ffff880062543480
kernel: [ 381.552737] ffff88005c617d88 0000000000000000
ffff880057ade540 ffff88005c617f28
kernel: [ 381.554800] ffff880067564000 00000000ffffff9c
ffff88005c617d98 ffffffff8118f2c8
kernel: [ 381.554800] Call Trace:
kernel: [ 381.554800] [<ffffffff8118f2c8>] get_empty_filp+0x58/0x180
kernel: [ 381.554800] [<ffffffff8119cb18>] path_openat+0x48/0x4d0
kernel: [ 381.554800] [<ffffffff811993e9>] ? putname+0x29/0x40
kernel: [ 381.554800] [<ffffffff8119d6a3>] ? user_path_at_empty+0x63/0xa0
kernel: [ 381.554800] [<ffffffff816a068d>] ? mutex_lock+0x1d/0x50
kernel: [ 381.554800] [<ffffffff8119d742>] do_filp_open+0x42/0xa0
kernel: [ 381.554800] [<ffffffff816a2ace>] ? _raw_spin_lock+0xe/0x20
kernel: [ 381.554800] [<ffffffff811aace5>] ? __alloc_fd+0xe5/0x170
kernel: [ 381.554800] [<ffffffff8118d37e>] do_sys_open+0xfe/0x1e0
kernel: [ 381.554800] [<ffffffff8118d481>] sys_open+0x21/0x30
kernel: [ 381.554800] [<ffffffff816ab919>] system_call_fastpath+0x16/0x1b
kernel: [ 381.554800] Code: 00 4d 8b 04 24 65 4c 03 04 25 08 dc 00
00 49 8b 50 08 4d 8b 28 4d 85 ed 0f 84 03 01 00 00 49 63 44 24 20 49
8b 3c 24 48 8d 4a 01 <49> 8b 5c 05 00 4c 89 e8 65 48 0f c7 0f 0f 94 c0
84 c0 74 c2 49
kernel: [ 381.554800] RIP [<ffffffff8117ee4b>] kmem_cache_alloc+0x5b/0x170
kernel: [ 381.554800] RSP <ffff88005c617d28>
kernel: [ 381.554800] CR2: 000000170000000a
kernel: [ 381.579253] ---[ end trace 7b5bb1322a28df6d ]---
kernel: [ 381.580826] BUG: unable to handle kernel paging request at
000000170000000a
kernel: [ 381.582404] IP: [<ffffffff8117ee4b>] kmem_cache_alloc+0x5b/0x170
kernel: [ 381.584100] PGD 0
kernel: [ 381.584267] Oops: 0000 [#2] SMP
kernel: [ 381.585161] Modules linked in: xfs raid1 xfrm_user
xfrm4_tunnel tunnel4 ipcomp xfrm_ipcomp esp4 ah4 scst_vdisk(O)
iscsi_scst(O) scst(O) dm_zcache(O) dm_btrfs(O) btrfs(O) libcrc32c
dm_iostat(O) md_mod deflate zlib_deflate ctr twofish_generic
twofish_x86_64_3way glue_helper lrw xts gf128mul twofish_x86_64
twofish_common camellia_generic serpent_generic blowfish_generic
blowfish_x86_64 blowfish_common cast5_generic cast_common des_generic
xcbc rmd160 sha512_generic dm_round_robin crypto_null af_key xfrm_algo
kvm ppdev vfat fat dm_multipath nfsd(O) nfs_acl psmouse serio_raw
parport_pc i2c_piix4 microcode mac_hid nfsv4 auth_rpcgss nfs fscache
lockd lp sunrpc parport floppy
kernel: [ 381.595808] CPU 3
kernel: [ 381.595808] Pid: 16112, comm: udevd Tainted: G D O
3.8.13-557-generic #1382000791 Bochs Bochs
kernel: [ 381.595808] RIP: 0010:[<ffffffff8117ee4b>]
[<ffffffff8117ee4b>] kmem_cache_alloc+0x5b/0x170
kernel: [ 381.604262] RSP: 0018:ffff880067469d28 EFLAGS: 00010206
kernel: [ 381.604262] RAX: 0000000000000000 RBX: 0000000000000000
RCX: 00000000000427ee
kernel: [ 381.604262] RDX: 00000000000427ed RSI: 00000000000080d0
RDI: 0000000000016c60
kernel: [ 381.604262] RBP: ffff880067469d78 R08: ffff88007fd96c60
R09: 0000000000000000
kernel: [ 381.604262] R10: 8080808080808080 R11: 0000000000000000
R12: ffff88007d001900
kernel: [ 381.604262] R13: 000000170000000a R14: 00000000000080d0
R15: ffffffff8118f2c8
kernel: [ 381.604262] FS: 00007f77822c37c0(0000)
GS:ffff88007fd80000(0000) knlGS:0000000000000000
kernel: [ 381.604262] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
kernel: [ 381.604262] CR2: 000000170000000a CR3: 00000000797c1000
CR4: 00000000000006e0
kernel: [ 381.604262] DR0: 0000000000000000 DR1: 0000000000000000
DR2: 0000000000000000
kernel: [ 381.604262] DR3: 0000000000000000 DR6: 00000000ffff0ff0
DR7: 0000000000000400
kernel: [ 381.604262] Process udevd (pid: 16112, threadinfo
ffff880067468000, task ffff88007aed8000)
kernel: [ 381.604262] Stack:
kernel: [ 381.604262] ffff880067469df8 0000000000001000
00000000528e1772 00000000261c9b68
kernel: [ 381.604262] 0000000000000000 0000000000000000
ffff880078e5ce40 ffff880067469f28
kernel: [ 381.604262] ffff880067560000 00000000ffffff9c
ffff880067469d98 ffffffff8118f2c8
kernel: [ 381.604262] Call Trace:
kernel: [ 381.604262] [<ffffffff8118f2c8>] get_empty_filp+0x58/0x180
kernel: [ 381.604262] [<ffffffff8119cb18>] path_openat+0x48/0x4d0
kernel: [ 381.604262] [<ffffffff811a9619>] ? notify_change+0x289/0x390
kernel: [ 381.604262] [<ffffffff8118f9a3>] ? __sb_end_write+0x33/0x70
kernel: [ 381.604262] [<ffffffff8119d742>] do_filp_open+0x42/0xa0
kernel: [ 381.604262] [<ffffffff816a2ace>] ? _raw_spin_lock+0xe/0x20
kernel: [ 381.604262] [<ffffffff811aace5>] ? __alloc_fd+0xe5/0x170
kernel: [ 381.604262] [<ffffffff8118d37e>] do_sys_open+0xfe/0x1e0
kernel: [ 381.604262] [<ffffffff811be489>] ? sys_utimensat+0x69/0xa0
kernel: [ 381.604262] [<ffffffff8118d4a4>] sys_openat+0x14/0x20
kernel: [ 381.604262] [<ffffffff816ab919>] system_call_fastpath+0x16/0x1b
kernel: [ 381.604262] Code: 00 4d 8b 04 24 65 4c 03 04 25 08 dc 00
00 49 8b 50 08 4d 8b 28 4d 85 ed 0f 84 03 01 00 00 49 63 44 24 20 49
8b 3c 24 48 8d 4a 01 <49> 8b 5c 05 00 4c 89 e8 65 48 0f c7 0f 0f 94 c0
84 c0 74 c2 49
kernel: [ 381.604262] RIP [<ffffffff8117ee4b>] kmem_cache_alloc+0x5b/0x170
--
To unsubscribe from this list: send the line "unsubscribe linux-xfs" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
