Greetings, I am using stock XFS from kernel 3.8.13, compiled with kmemleak enabled. I am testing a particular scenario, in which the underlying block device returns IO errors during XFS umount. Almost in all cases this results in kernel crashes in various places, and sometimes kmemleak complains, and sometimes CPU soft lockup happens. It always happens after XFS messages like: kernel: [ 600.190509] XFS (dm-22): metadata I/O error: block 0x7600030 ("xlog_iodone") error 125 numblks 64 kernel: [ 600.192267] XFS (dm-22): xfs_do_force_shutdown(0x2) called from line 1115 of file /mnt/compile/linux-stable/source/fs/xfs/xfs_log.c. Return address = 0xffffffffa05cffa1 kernel: [ 600.192319] XFS (dm-22): Log I/O Error Detected. Shutting down filesystem kernel: [ 600.192392] XFS (dm-22): Unable to update superblock counters. Freespace may not be correct on next mount. kernel: [ 600.192398] XFS (dm-22): xfs_log_force: error 5 returned. kernel: [ 600.193687] XFS (?º.Z): Please umount the filesystem and rectify the problem(s) you can see here the garbage that XFS prints instead of the block device name. In [1] and [2] I am attaching more kernel log from two such crashes. Can anybody please advise on this? Thanks, Alex. [1] kernel: [ 600.190509] XFS (dm-22): metadata I/O error: block 0x7600030 ("xlog_iodone") error 125 numblks 64 kernel: [ 600.192267] XFS (dm-22): xfs_do_force_shutdown(0x2) called from line 1115 of file /mnt/compile/linux-stable/source/fs/xfs/xfs_log.c. Return address = 0xffffffffa05cffa1 kernel: [ 600.192319] XFS (dm-22): Log I/O Error Detected. Shutting down filesystem kernel: [ 600.192392] XFS (dm-22): Unable to update superblock counters. Freespace may not be correct on next mount. kernel: [ 600.192398] XFS (dm-22): xfs_log_force: error 5 returned. kernel: [ 600.193687] XFS (?º.Z): Please umount the filesystem and rectify the problem(s) kernel: [ 600.223974] BUG: unable to handle kernel paging request at 0000000000010001 kernel: [ 600.225347] IP: [<ffffffff8117d823>] __kmalloc+0x83/0x1c0 kernel: [ 600.226381] PGD 486e9067 PUD 5eff6067 PMD 0 kernel: [ 600.227256] Oops: 0000 [#1] SMP kernel: [ 600.227881] Modules linked in: xfs raid1 xfrm_user xfrm4_tunnel tunnel4 ipcomp xfrm_ipcomp esp4 ah4 scst_vdisk(O) iscsi_scst(O) scst(O) dm_zcache(O) dm_btrfs(O) btrfs(O) libcrc32c dm_iostat(O) md_mod deflate zlib_deflate ctr twofish_generic twofish_x86_64_3way glue_helper lrw xts gf128mul twofish_x86_64 twofish_common camellia_generic serpent_generic blowfish_generic blowfish_x86_64 blowfish_common cast5_generic cast_common des_generic xcbc rmd160 sha512_generic crypto_null af_key xfrm_algo dm_round_robin kvm vfat fat ppdev nfsd(O) nfs_acl microcode dm_multipath psmouse serio_raw parport_pc i2c_piix4 mac_hid lp nfsv4 auth_rpcgss parport nfs fscache lockd sunrpc floppy kernel: [ 600.227881] CPU 3 kernel: [ 600.227881] Pid: 26391, comm: udev-logger.sh Tainted: G O 3.8.13-557-generic #1382000791 Bochs Bochs kernel: [ 600.227881] RIP: 0010:[<ffffffff8117d823>] [<ffffffff8117d823>] __kmalloc+0x83/0x1c0 kernel: [ 600.227881] RSP: 0018:ffff88005ae6fcd8 EFLAGS: 00010202 kernel: [ 600.227881] RAX: 0000000000000000 RBX: 0000000000000100 RCX: 000000000003b9be kernel: [ 600.227881] RDX: 000000000003b9bd RSI: 0000000000000000 RDI: 0000000000016c60 kernel: [ 600.227881] RBP: ffff88005ae6fd28 R08: ffff88007fd96c60 R09: ffffffff812f4011 kernel: [ 600.227881] R10: 000000000000c6e0 R11: ffff88007fd80000 R12: ffff88007d001900 kernel: [ 600.227881] R13: 0000000000010001 R14: 00000000000000d0 R15: 0000000000000100 kernel: [ 600.227881] FS: 00007ffacca11700(0000) GS:ffff88007fd80000(0000) knlGS:0000000000000000 kernel: [ 600.227881] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b kernel: [ 600.227881] CR2: 0000000000010001 CR3: 0000000069fab000 CR4: 00000000000006e0 kernel: [ 600.227881] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 kernel: [ 600.227881] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 kernel: [ 600.227881] Process udev-logger.sh (pid: 26391, threadinfo ffff88005ae6e000, task ffff88005e94dc00) kernel: [ 600.227881] Stack: kernel: [ 600.227881] ffffffff812f4011 0000000000000000 ffff88005ae6fd58 ffffffff812be0cb kernel: [ 600.227881] 00000000000112d0 0000000000000100 ffff880054536500 ffff88005ae6fe38 kernel: [ 600.227881] ffff880078c4bf10 0000000000000000 ffff88005ae6fd98 ffffffff812f4011 kernel: [ 600.227881] Call Trace: kernel: [ 600.227881] [<ffffffff812f4011>] ? aa_path_name+0x91/0xe0 kernel: [ 600.227881] [<ffffffff812be0cb>] ? get_file_caps+0x17b/0x1a0 kernel: [ 600.227881] [<ffffffff812f4011>] aa_path_name+0x91/0xe0 kernel: [ 600.227881] [<ffffffff812f46e7>] apparmor_bprm_set_creds+0x107/0x780 kernel: [ 600.227881] [<ffffffff8115d6ef>] ? vma_link+0xcf/0xe0 kernel: [ 600.227881] [<ffffffff812bf203>] security_bprm_set_creds+0x13/0x20 kernel: [ 600.227881] [<ffffffff81193698>] prepare_binprm+0xb8/0x1e0 kernel: [ 600.227881] [<ffffffff811953a2>] do_execve_common.isra.24+0x232/0x330 kernel: [ 600.227881] [<ffffffff811954b8>] do_execve+0x18/0x20 kernel: [ 600.227881] [<ffffffff8119575d>] sys_execve+0x3d/0x60 kernel: [ 600.227881] [<ffffffff816abe89>] stub_execve+0x69/0xc0 kernel: [ 600.227881] Code: b0 4d 8b 04 24 65 4c 03 04 25 08 dc 00 00 49 8b 50 08 4d 8b 28 4d 85 ed 0f 84 03 01 00 00 49 63 44 24 20 49 8b 3c 24 48 8d 4a 01 <49> 8b 5c 05 00 4c 89 e8 65 48 0f c7 0f 0f 94 c0 84 c0 74 c2 49 kernel: [ 600.227881] RIP [<ffffffff8117d823>] __kmalloc+0x83/0x1c0 kernel: [ 600.227881] RSP <ffff88005ae6fcd8> kernel: [ 600.227881] CR2: 0000000000010001 kernel: [ 600.283792] ---[ end trace ded81b9e3753ac4e ]--- kernel: [ 600.333036] BUG: unable to handle kernel paging request at 0000000000010001 kernel: [ 600.334395] IP: [<ffffffff8117ee4b>] kmem_cache_alloc+0x5b/0x170 kernel: [ 600.335518] PGD 486e9067 PUD 5eff6067 PMD 0 kernel: [ 600.336022] Oops: 0000 [#2] SMP kernel: [ 600.336022] Modules linked in: xfs raid1 xfrm_user xfrm4_tunnel tunnel4 ipcomp xfrm_ipcomp esp4 ah4 scst_vdisk(O) iscsi_scst(O) scst(O) dm_zcache(O) dm_btrfs(O) btrfs(O) libcrc32c dm_iostat(O) md_mod deflate zlib_deflate ctr twofish_generic twofish_x86_64_3way glue_helper lrw xts gf128mul twofish_x86_64 twofish_common camellia_generic serpent_generic blowfish_generic blowfish_x86_64 blowfish_common cast5_generic cast_common des_generic xcbc rmd160 sha512_generic crypto_null af_key xfrm_algo dm_round_robin kvm vfat fat ppdev nfsd(O) nfs_acl microcode dm_multipath psmouse serio_raw parport_pc i2c_piix4 mac_hid lp nfsv4 auth_rpcgss parport nfs fscache lockd sunrpc floppy kernel: [ 600.336022] CPU 3 kernel: [ 600.336022] Pid: 26391, comm: udev-logger.sh Tainted: G D O 3.8.13-557-generic #1382000791 Bochs Bochs kernel: [ 600.336022] RIP: 0010:[<ffffffff8117ee4b>] [<ffffffff8117ee4b>] kmem_cache_alloc+0x5b/0x170 kernel: [ 600.336022] RSP: 0018:ffff88005ae6f168 EFLAGS: 00010002 kernel: [ 600.336022] RAX: 0000000000000000 RBX: ffffffff81c99300 RCX: 000000000003b9be kernel: [ 600.336022] RDX: 000000000003b9bd RSI: 0000000000008020 RDI: 0000000000016c60 kernel: [ 600.336022] RBP: ffff88005ae6f1b8 R08: ffff88007fd96c60 R09: 0000000000000001 kernel: [ 600.336022] R10: 0000000000849400 R11: 000000000000002f R12: ffff88007d001900 kernel: [ 600.336022] R13: 0000000000010001 R14: 0000000000008020 R15: ffffffff8146e0e3 kernel: [ 600.336022] FS: 00007ffacca11700(0000) GS:ffff88007fd80000(0000) knlGS:0000000000000000 kernel: [ 600.336022] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b kernel: [ 600.336022] CR2: 0000000000010001 CR3: 0000000069fab000 CR4: 00000000000006e0 kernel: [ 600.336022] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 kernel: [ 600.336022] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 kernel: [ 600.336022] Process udev-logger.sh (pid: 26391, threadinfo ffff88005ae6e000, task ffff88005e94dc00) kernel: [ 600.336022] Stack: kernel: [ 600.336022] ffff880036e25400 ffff880048482a48 ffff88005ae6f198 ffffffff8132ca4e kernel: [ 600.336022] ffff880036598000 ffffffff81c99300 ffff880036f68800 0000000000000020 kernel: [ 600.336022] 0000000000000020 ffff880036e26000 ffff88005ae6f1e8 ffffffff8146e0e3 kernel: [ 600.336022] Call Trace: kernel: [ 600.336022] [<ffffffff8132ca4e>] ? cfq_remove_request+0x3e/0xf0 kernel: [ 600.336022] [<ffffffff8146e0e3>] scsi_pool_alloc_command+0x33/0x80 kernel: [ 600.336022] [<ffffffff8146ec68>] scsi_host_alloc_command+0x28/0x90 kernel: [ 600.336022] [<ffffffff8146ecf5>] __scsi_get_command+0x25/0xf0 kernel: [ 600.336022] [<ffffffff8146ee05>] scsi_get_command+0x45/0xc0 kernel: [ 600.336022] [<ffffffff81475dfa>] scsi_get_cmd_from_req+0x5a/0x70 kernel: [ 600.336022] [<ffffffff81476b05>] scsi_setup_fs_cmnd.part.33+0x35/0x90 kernel: [ 600.336022] [<ffffffff81476b8f>] scsi_setup_fs_cmnd+0x2f/0x40 kernel: [ 600.336022] [<ffffffff814949cf>] sd_prep_fn+0xff/0xb00 kernel: [ 600.336022] [<ffffffff8131347a>] blk_peek_request+0xba/0x220 kernel: [ 600.336022] [<ffffffff81476d6f>] scsi_request_fn+0x4f/0x450 kernel: [ 600.336022] [<ffffffff8130c8b7>] __blk_run_queue+0x37/0x50 kernel: [ 600.336022] [<ffffffff8130ef21>] queue_unplugged+0x41/0xd0 kernel: [ 600.336022] [<ffffffff813139b2>] blk_flush_plug_list+0x1d2/0x230 kernel: [ 600.336022] [<ffffffff81313a28>] blk_finish_plug+0x18/0x50 kernel: [ 600.336022] [<ffffffff81231dfe>] ext4_da_writepages+0x3de/0x620 kernel: [ 600.336022] [<ffffffff8118f9a3>] ? __sb_end_write+0x33/0x70 kernel: [ 600.336022] [<ffffffff8113aec0>] do_writepages+0x20/0x40 kernel: [ 600.336022] [<ffffffff81130df1>] __filemap_fdatawrite_range+0x51/0x60 kernel: [ 600.336022] [<ffffffff81130e52>] filemap_write_and_wait_range+0x52/0x80 kernel: [ 600.336022] [<ffffffff812275ef>] ext4_sync_file+0x6f/0x2b0 kernel: [ 600.336022] [<ffffffff8105c988>] ? kmsg_dump_get_line+0x68/0x90 kernel: [ 600.336022] [<ffffffff811bda1b>] vfs_fsync+0x2b/0x40 kernel: [ 600.336022] [<ffffffffa02f9d76>] dm_iostat_kmsg_write35+0xb6/0xf0 [dm_iostat] kernel: [ 600.336022] [<ffffffffa02fa0d3>] dm_iostat_kmsg_dump+0x73/0x220 [dm_iostat] kernel: [ 600.336022] [<ffffffff8168b992>] ? printk+0x61/0x63 kernel: [ 600.336022] [<ffffffff8105c83c>] kmsg_dump+0x9c/0xc0 kernel: [ 600.336022] [<ffffffff81058ff9>] oops_exit+0x29/0x30 kernel: [ 600.336022] [<ffffffff816a3f33>] oops_end+0x83/0xf0 kernel: [ 600.336022] [<ffffffff8168ab25>] no_context+0x1ab/0x1ba kernel: [ 600.336022] [<ffffffff8168ad07>] __bad_area_nosemaphore+0x1d3/0x1f2 kernel: [ 600.336022] [<ffffffff810bb842>] ? __module_text_address+0x12/0x70 kernel: [ 600.336022] [<ffffffff8168ad80>] bad_area+0x45/0x4d kernel: [ 600.336022] [<ffffffff816a6fb8>] __do_page_fault+0x558/0x560 kernel: [ 600.336022] [<ffffffff812269c9>] ? ext4_file_open+0x79/0x250 kernel: [ 600.336022] [<ffffffff81137695>] ? prep_new_page+0x145/0x1e0 kernel: [ 600.336022] [<ffffffff81137917>] ? get_page_from_freelist+0x1e7/0x5a0 kernel: [ 600.336022] [<ffffffff816a6fce>] do_page_fault+0xe/0x10 kernel: [ 600.336022] [<ffffffff816a66a5>] do_async_page_fault+0x35/0x90 kernel: [ 600.336022] [<ffffffff816a33c8>] async_page_fault+0x28/0x30 kernel: [ 600.336022] [<ffffffff812f4011>] ? aa_path_name+0x91/0xe0 kernel: [ 600.336022] [<ffffffff8117d823>] ? __kmalloc+0x83/0x1c0 kernel: [ 600.336022] [<ffffffff8117d7f5>] ? __kmalloc+0x55/0x1c0 kernel: [ 600.336022] [<ffffffff812f4011>] ? aa_path_name+0x91/0xe0 kernel: [ 600.336022] [<ffffffff812be0cb>] ? get_file_caps+0x17b/0x1a0 kernel: [ 600.336022] [<ffffffff812f4011>] aa_path_name+0x91/0xe0 kernel: [ 600.336022] [<ffffffff812f46e7>] apparmor_bprm_set_creds+0x107/0x780 kernel: [ 600.336022] [<ffffffff8115d6ef>] ? vma_link+0xcf/0xe0 kernel: [ 600.336022] [<ffffffff812bf203>] security_bprm_set_creds+0x13/0x20 kernel: [ 600.336022] [<ffffffff81193698>] prepare_binprm+0xb8/0x1e0 kernel: [ 600.336022] [<ffffffff811953a2>] do_execve_common.isra.24+0x232/0x330 kernel: [ 600.336022] [<ffffffff811954b8>] do_execve+0x18/0x20 kernel: [ 600.336022] [<ffffffff8119575d>] sys_execve+0x3d/0x60 kernel: [ 600.336022] [<ffffffff816abe89>] stub_execve+0x69/0xc0 kernel: [ 600.336022] Code: 00 4d 8b 04 24 65 4c 03 04 25 08 dc 00 00 49 8b 50 08 4d 8b 28 4d 85 ed 0f 84 03 01 00 00 49 63 44 24 20 49 8b 3c 24 48 8d 4a 01 <49> 8b 5c 05 00 4c 89 e8 65 48 0f c7 0f 0f 94 c0 84 c0 74 c2 49 kernel: [ 600.336022] RIP [<ffffffff8117ee4b>] kmem_cache_alloc+0x5b/0x170 kernel: [ 600.336022] RSP <ffff88005ae6f168> kernel: [ 600.336022] CR2: 0000000000010001 kernel: [ 600.336022] ---[ end trace ded81b9e3753ac4f ]--- kernel: [ 600.683610] BUG: unable to handle kernel paging request at 0000000000010001 kernel: [ 600.685003] IP: [<ffffffff8117ee4b>] kmem_cache_alloc+0x5b/0x170 kernel: [ 600.686176] PGD 78bc3067 PUD 6bb27067 PMD 0 kernel: [ 600.687073] Oops: 0000 [#3] SMP kernel: [ 600.687472] Modules linked in: xfs raid1 xfrm_user xfrm4_tunnel tunnel4 ipcomp xfrm_ipcomp esp4 ah4 scst_vdisk(O) iscsi_scst(O) scst(O) dm_zcache(O) dm_btrfs(O) btrfs(O) libcrc32c dm_iostat(O) md_mod deflate zlib_deflate ctr twofish_generic twofish_x86_64_3way glue_helper lrw xts gf128mul twofish_x86_64 twofish_common camellia_generic serpent_generic blowfish_generic blowfish_x86_64 blowfish_common cast5_generic cast_common des_generic xcbc rmd160 sha512_generic crypto_null af_key xfrm_algo dm_round_robin kvm vfat fat ppdev nfsd(O) nfs_acl microcode dm_multipath psmouse serio_raw parport_pc i2c_piix4 mac_hid lp nfsv4 auth_rpcgss parport nfs fscache lockd sunrpc floppy kernel: [ 600.687472] CPU 3 kernel: [ 600.687472] Pid: 15622, comm: zadara_vac Tainted: G D O 3.8.13-557-generic #1382000791 Bochs Bochs kernel: [ 600.687472] RIP: 0010:[<ffffffff8117ee4b>] [<ffffffff8117ee4b>] kmem_cache_alloc+0x5b/0x170 kernel: [ 600.687472] RSP: 0018:ffff88006bae5d28 EFLAGS: 00010202 kernel: [ 600.687472] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 000000000003b9be kernel: [ 600.687472] RDX: 000000000003b9bd RSI: 00000000000080d0 RDI: 0000000000016c60 kernel: [ 600.687472] RBP: ffff88006bae5d78 R08: ffff88007fd96c60 R09: 0000000000000000 kernel: [ 600.687472] R10: 8080808080808080 R11: 0000000000000000 R12: ffff88007d001900 kernel: [ 600.687472] R13: 0000000000010001 R14: 00000000000080d0 R15: ffffffff8118f2c8 kernel: [ 600.687472] FS: 00007f920b7fe700(0000) GS:ffff88007fd80000(0000) knlGS:0000000000000000 kernel: [ 600.687472] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 kernel: [ 600.687472] CR2: 0000000000010001 CR3: 000000006baf9000 CR4: 00000000000006e0 kernel: [ 600.687472] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 kernel: [ 600.687472] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 kernel: [ 600.687472] Process zadara_vac (pid: 15622, threadinfo ffff88006bae4000, task ffff8800567aae00) kernel: [ 600.687472] Stack: kernel: [ 600.687472] ffff88006bae5d98 ffff88006bae4000 00000003679c1065 ffff88007fd83fc0 kernel: [ 600.687472] ffff880000000029 0000000000000000 ffff88007b27f0c0 ffff88006bae5f28 kernel: [ 600.687472] ffff880057bf3000 00000000ffffff9c ffff88006bae5d98 ffffffff8118f2c8 kernel: [ 600.687472] Call Trace: kernel: [ 600.687472] [<ffffffff8118f2c8>] get_empty_filp+0x58/0x180 kernel: [ 600.687472] [<ffffffff8119cb18>] path_openat+0x48/0x4d0 kernel: [ 600.687472] [<ffffffff81189e0b>] ? create_object+0x22b/0x2e0 kernel: [ 600.687472] [<ffffffff8119d742>] do_filp_open+0x42/0xa0 kernel: [ 600.687472] [<ffffffff816a2ace>] ? _raw_spin_lock+0xe/0x20 kernel: [ 600.687472] [<ffffffff811aace5>] ? __alloc_fd+0xe5/0x170 kernel: [ 600.687472] [<ffffffff8118d37e>] do_sys_open+0xfe/0x1e0 kernel: [ 600.687472] [<ffffffff816a6fce>] ? do_page_fault+0xe/0x10 kernel: [ 600.687472] [<ffffffff8118d481>] sys_open+0x21/0x30 kernel: [ 600.687472] [<ffffffff816ab919>] system_call_fastpath+0x16/0x1b kernel: [ 600.687472] Code: 00 4d 8b 04 24 65 4c 03 04 25 08 dc 00 00 49 8b 50 08 4d 8b 28 4d 85 ed 0f 84 03 01 00 00 49 63 44 24 20 49 8b 3c 24 48 8d 4a 01 <49> 8b 5c 05 00 4c 89 e8 65 48 0f c7 0f 0f 94 c0 84 c0 74 c2 49 kernel: [ 600.687472] RIP [<ffffffff8117ee4b>] kmem_cache_alloc+0x5b/0x170 kernel: [ 600.687472] RSP <ffff88006bae5d28> kernel: [ 600.687472] CR2: 0000000000010001 kernel: [ 600.745966] ---[ end trace ded81b9e3753ac50 ]--- kernel: [ 600.769821] BUG: unable to handle kernel paging request at 0000000000010001 kernel: [ 600.771366] IP: [<ffffffff8117ee4b>] kmem_cache_alloc+0x5b/0x170 kernel: [ 600.772674] PGD 78bc3067 PUD 6bb27067 PMD 0 kernel: [ 600.773697] Oops: 0000 [#4] SMP kernel: [ 600.773708] Modules linked in: xfs raid1 xfrm_user xfrm4_tunnel tunnel4 ipcomp xfrm_ipcomp esp4 ah4 scst_vdisk(O) iscsi_scst(O) scst(O) dm_zcache(O) dm_btrfs(O) btrfs(O) libcrc32c dm_iostat(O) md_mod deflate zlib_deflate ctr twofish_generic twofish_x86_64_3way glue_helper lrw xts gf128mul twofish_x86_64 twofish_common camellia_generic serpent_generic blowfish_generic blowfish_x86_64 blowfish_common cast5_generic cast_common des_generic xcbc rmd160 sha512_generic crypto_null af_key xfrm_algo dm_round_robin kvm vfat fat ppdev nfsd(O) nfs_acl microcode dm_multipath psmouse serio_raw parport_pc i2c_piix4 mac_hid lp nfsv4 auth_rpcgss parport nfs fscache lockd sunrpc floppy kernel: [ 600.773708] CPU 3 kernel: [ 600.773708] Pid: 15571, comm: md1_raid1 Tainted: G D O 3.8.13-557-generic #1382000791 Bochs Bochs kernel: [ 600.773708] RIP: 0010:[<ffffffff8117ee4b>] [<ffffffff8117ee4b>] kmem_cache_alloc+0x5b/0x170 kernel: [ 600.773708] RSP: 0018:ffff880069fa1868 EFLAGS: 00010202 kernel: [ 600.773708] RAX: 0000000000000000 RBX: ffff88005d28cd80 RCX: 000000000003b9be kernel: [ 600.773708] RDX: 000000000003b9bd RSI: 0000000000011200 RDI: 0000000000016c60 kernel: [ 600.773708] RBP: ffff880069fa18b8 R08: ffff88007fd96c60 R09: 0000000000000001 kernel: [ 600.773708] R10: 0000000000000000 R11: 0000000000000001 R12: ffff88007d001900 kernel: [ 600.773708] R13: 0000000000010001 R14: 0000000000011200 R15: ffffffff81131ed5 kernel: [ 600.773708] FS: 0000000000000000(0000) GS:ffff88007fd80000(0000) knlGS:0000000000000000 kernel: [ 600.773708] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b kernel: [ 600.773708] CR2: 0000000000010001 CR3: 000000006baf9000 CR4: 00000000000006e0 kernel: [ 600.773708] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 kernel: [ 600.773708] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 kernel: [ 600.773708] Process md1_raid1 (pid: 15571, threadinfo ffff880069fa0000, task ffff880054758000) kernel: [ 600.773708] Stack: kernel: [ 600.773708] ffffffffa02e4ae8 ffff880069fa1888 ffffffff810bb842 0000000000000000 kernel: [ 600.773708] ffff880069fa1898 ffff88005d28cd80 0000000000011210 ffff88005d28cdb0 kernel: [ 600.773708] ffff880069fa1908 ffff880054758000 ffff880069fa18c8 ffffffff81131ed5 kernel: [ 600.773708] Call Trace: kernel: [ 600.773708] [<ffffffff810bb842>] ? __module_text_address+0x12/0x70 kernel: [ 600.773708] [<ffffffff81131ed5>] mempool_alloc_slab+0x15/0x20 kernel: [ 600.773708] [<ffffffff8113222e>] mempool_alloc+0x5e/0x160 kernel: [ 600.773708] [<ffffffff811c4f13>] bio_alloc_bioset+0x43/0x110 kernel: [ 600.773708] [<ffffffff81131ed5>] ? mempool_alloc_slab+0x15/0x20 kernel: [ 600.773708] [<ffffffff8153d5de>] alloc_tio.isra.15+0x2e/0x60 kernel: [ 600.773708] [<ffffffff8153dab8>] __clone_and_map+0x248/0x2d0 kernel: [ 600.773708] [<ffffffff81189e0b>] ? create_object+0x22b/0x2e0 kernel: [ 600.773708] [<ffffffff8153e378>] __split_and_process_bio+0x178/0x1b0 kernel: [ 600.773708] [<ffffffff8153e4ac>] _dm_request.isra.24+0xfc/0x150 kernel: [ 600.773708] [<ffffffff8153e528>] dm_request+0x28/0x40 kernel: [ 600.773708] [<ffffffff813103c7>] generic_make_request.part.55+0x77/0xb0 kernel: [ 600.773708] [<ffffffff81310958>] generic_make_request+0x68/0x70 kernel: [ 600.773708] [<ffffffff813109db>] submit_bio+0x7b/0x160 kernel: [ 600.773708] [<ffffffffa02d77b6>] md_super_write+0x86/0xb0 [md_mod] kernel: [ 600.773708] [<ffffffffa02dba33>] write_sb_page+0x73/0x1f0 [md_mod] kernel: [ 600.773708] [<ffffffffa02dd166>] write_page+0x116/0x130 [md_mod] kernel: [ 600.773708] [<ffffffffa02d5924>] ? super_1_sync+0x3f4/0x4e0 [md_mod] kernel: [ 600.773708] [<ffffffffa02dd26d>] bitmap_update_sb.part.19+0xed/0xf0 [md_mod] kernel: [ 600.773708] [<ffffffffa02dcf1f>] bitmap_update_sb+0x2f/0x40 [md_mod] kernel: [ 600.773708] [<ffffffffa02d7a20>] md_update_sb+0x1b0/0x4f0 [md_mod] kernel: [ 600.773708] [<ffffffffa02de6b5>] ? bitmap_daemon_work+0x215/0x340 [md_mod] kernel: [ 600.773708] [<ffffffffa02d8baa>] md_check_recovery.part.50+0x1da/0x530 [md_mod] kernel: [ 600.773708] [<ffffffff8106a8ba>] ? del_timer_sync+0x5a/0x70 kernel: [ 600.773708] [<ffffffffa02d8f15>] md_check_recovery+0x15/0x20 [md_mod] kernel: [ 600.773708] [<ffffffffa055eed2>] raid1d+0x22/0x180 [raid1] kernel: [ 600.773708] [<ffffffff81045cd9>] ? default_spin_lock_flags+0x9/0x10 kernel: [ 600.773708] [<ffffffff81045cd9>] ? default_spin_lock_flags+0x9/0x10 kernel: [ 600.773708] [<ffffffffa02d084d>] md_thread+0x10d/0x140 [md_mod] kernel: [ 600.773708] [<ffffffff8107ead0>] ? add_wait_queue+0x60/0x60 kernel: [ 600.773708] [<ffffffffa02d0740>] ? md_rdev_init+0x140/0x140 [md_mod] kernel: [ 600.773708] [<ffffffff8107df10>] kthread+0xc0/0xd0 kernel: [ 600.773708] [<ffffffff8107de50>] ? flush_kthread_worker+0xb0/0xb0 kernel: [ 600.773708] [<ffffffff816ab86c>] ret_from_fork+0x7c/0xb0 kernel: [ 600.773708] [<ffffffff8107de50>] ? flush_kthread_worker+0xb0/0xb0 kernel: [ 600.773708] Code: 00 4d 8b 04 24 65 4c 03 04 25 08 dc 00 00 49 8b 50 08 4d 8b 28 4d 85 ed 0f 84 03 01 00 00 49 63 44 24 20 49 8b 3c 24 48 8d 4a 01 <49> 8b 5c 05 00 4c 89 e8 65 48 0f c7 0f 0f 94 c0 84 c0 74 c2 49 kernel: [ 600.773708] RIP [<ffffffff8117ee4b>] kmem_cache_alloc+0x5b/0x170 kernel: [ 600.773708] RSP <ffff880069fa1868> kernel: [ 600.773708] CR2: 0000000000010001 kernel: [ 600.869584] ---[ end trace ded81b9e3753ac51 ]--- kernel: [ 603.924718] BUG: unable to handle kernel paging request at 0000000000010001 kernel: [ 603.926316] IP: [<ffffffff8117ee4b>] kmem_cache_alloc+0x5b/0x170 kernel: [ 603.927624] PGD 79735067 PUD 79734067 PMD 0 kernel: [ 603.928456] Oops: 0000 [#5] SMP kernel: [ 603.928456] Modules linked in: xfs raid1 xfrm_user xfrm4_tunnel tunnel4 ipcomp xfrm_ipcomp esp4 ah4 scst_vdisk(O) iscsi_scst(O) scst(O) dm_zcache(O) dm_btrfs(O) btrfs(O) libcrc32c dm_iostat(O) md_mod deflate zlib_deflate ctr twofish_generic twofish_x86_64_3way glue_helper lrw xts gf128mul twofish_x86_64 twofish_common camellia_generic serpent_generic blowfish_generic blowfish_x86_64 blowfish_common cast5_generic cast_common des_generic xcbc rmd160 sha512_generic crypto_null af_key xfrm_algo dm_round_robin kvm vfat fat ppdev nfsd(O) nfs_acl microcode dm_multipath psmouse serio_raw parport_pc i2c_piix4 mac_hid lp nfsv4 auth_rpcgss parport nfs fscache lockd sunrpc floppy kernel: [ 603.928456] CPU 3 kernel: [ 603.928456] Pid: 4922, comm: PassengerHelper Tainted: G D O 3.8.13-557-generic #1382000791 Bochs Bochs kernel: [ 603.928456] RIP: 0010:[<ffffffff8117ee4b>] [<ffffffff8117ee4b>] kmem_cache_alloc+0x5b/0x170 kernel: [ 603.928456] RSP: 0018:ffff88006b8a3e08 EFLAGS: 00010202 kernel: [ 603.928456] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 000000000003b9be kernel: [ 603.928456] RDX: 000000000003b9bd RSI: 00000000000080d0 RDI: 0000000000016c60 kernel: [ 603.928456] RBP: ffff88006b8a3e58 R08: ffff88007fd96c60 R09: 0000000000000000 kernel: [ 603.928456] R10: 000000000000c6e0 R11: ffff88007fd80000 R12: ffff88007d001900 kernel: [ 603.928456] R13: 0000000000010001 R14: 00000000000080d0 R15: ffffffff8118f2c8 kernel: [ 603.928456] FS: 00007fb3c2462700(0000) GS:ffff88007fd80000(0000) knlGS:0000000000000000 kernel: [ 603.928456] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 kernel: [ 603.928456] CR2: 0000000000010001 CR3: 000000007af80000 CR4: 00000000000006e0 kernel: [ 603.928456] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 kernel: [ 603.928456] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 kernel: [ 603.928456] Process PassengerHelper (pid: 4922, threadinfo ffff88006b8a2000, task ffff88007a085c00) kernel: [ 603.928456] Stack: kernel: [ 603.928456] ffff88006b8a3e68 ffffffff8117ee9b ffff88007c9ac500 ffffffff811a55e4 kernel: [ 603.928456] 00000000000000c0 0000000000000000 ffff8800366a59c0 0000000000000002 kernel: [ 603.928456] ffffffff8181bca0 0000000000000003 ffff88006b8a3e78 ffffffff8118f2c8 kernel: [ 603.928456] Call Trace: kernel: [ 603.928456] [<ffffffff8117ee9b>] ? kmem_cache_alloc+0xab/0x170 kernel: [ 603.928456] [<ffffffff811a55e4>] ? __d_alloc+0x34/0x180 kernel: [ 603.928456] [<ffffffff8118f2c8>] get_empty_filp+0x58/0x180 kernel: [ 603.928456] [<ffffffff8118f41b>] alloc_file+0x2b/0xd0 kernel: [ 603.928456] [<ffffffff81197129>] create_pipe_files+0x89/0x130 kernel: [ 603.928456] [<ffffffff81197212>] __do_pipe_flags+0x42/0xf0 kernel: [ 603.928456] [<ffffffff81197330>] sys_pipe2+0x20/0xa0 kernel: [ 603.928456] [<ffffffff816a3df5>] ? do_device_not_available+0x15/0x20 kernel: [ 603.928456] [<ffffffff816aca4e>] ? device_not_available+0x1e/0x30 kernel: [ 603.928456] [<ffffffff811973c0>] sys_pipe+0x10/0x20 kernel: [ 603.928456] [<ffffffff816ab919>] system_call_fastpath+0x16/0x1b kernel: [ 603.928456] Code: 00 4d 8b 04 24 65 4c 03 04 25 08 dc 00 00 49 8b 50 08 4d 8b 28 4d 85 ed 0f 84 03 01 00 00 49 63 44 24 20 49 8b 3c 24 48 8d 4a 01 <49> 8b 5c 05 00 4c 89 e8 65 48 0f c7 0f 0f 94 c0 84 c0 74 c2 49 kernel: [ 603.928456] RIP [<ffffffff8117ee4b>] kmem_cache_alloc+0x5b/0x170 kernel: [ 603.928456] RSP <ffff88006b8a3e08> kernel: [ 603.928456] CR2: 0000000000010001 kernel: [ 603.994540] ---[ end trace ded81b9e3753ac52 ]--- kernel: [ 645.662431] BUG: unable to handle kernel paging request at 0000000000010001 kernel: [ 645.663218] IP: [<ffffffff8117dd0f>] kmem_cache_alloc_trace+0x5f/0x160 kernel: [ 645.663824] PGD 5eeda067 PUD 5b684067 PMD 0 kernel: [ 645.664272] Oops: 0000 [#6] SMP kernel: [ 645.664603] Modules linked in: xfs raid1 xfrm_user xfrm4_tunnel tunnel4 ipcomp xfrm_ipcomp esp4 ah4 scst_vdisk(O) iscsi_scst(O) scst(O) dm_zcache(O) dm_btrfs(O) btrfs(O) libcrc32c dm_iostat(O) md_mod deflate zlib_deflate ctr twofish_generic twofish_x86_64_3way glue_helper lrw xts gf128mul twofish_x86_64 twofish_common camellia_generic serpent_generic blowfish_generic blowfish_x86_64 blowfish_common cast5_generic cast_common des_generic xcbc rmd160 sha512_generic crypto_null af_key xfrm_algo dm_round_robin kvm vfat fat ppdev nfsd(O) nfs_acl microcode dm_multipath psmouse serio_raw parport_pc i2c_piix4 mac_hid lp nfsv4 auth_rpcgss parport nfs fscache lockd sunrpc floppy kernel: [ 645.666264] CPU 3 kernel: [ 645.666264] Pid: 26392, comm: bash Tainted: G D O 3.8.13-557-generic #1382000791 Bochs Bochs kernel: [ 645.666264] RIP: 0010:[<ffffffff8117dd0f>] [<ffffffff8117dd0f>] kmem_cache_alloc_trace+0x5f/0x160 kernel: [ 645.666264] RSP: 0018:ffff88005bddbe58 EFLAGS: 00010202 kernel: [ 645.666264] RAX: 0000000000000000 RBX: ffff880057bf2020 RCX: 000000000003b9be kernel: [ 645.666264] RDX: 000000000003b9bd RSI: 00000000000080d0 RDI: 0000000000016c60 kernel: [ 645.666264] RBP: ffff88005bddbea8 R08: ffff88007fd96c60 R09: fffff2ff8f908bd0 kernel: [ 645.666264] R10: 8080808080808080 R11: 0000000000000000 R12: ffff88007d001900 kernel: [ 645.666264] R13: 0000000000010001 R14: 00000000000080d0 R15: ffffffff81195237 kernel: [ 645.666264] FS: 00007fb039494700(0000) GS:ffff88007fd80000(0000) knlGS:0000000000000000 kernel: [ 645.666264] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b kernel: [ 645.666264] CR2: 0000000000010001 CR3: 0000000054649000 CR4: 00000000000006e0 kernel: [ 645.666264] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 kernel: [ 645.666264] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 kernel: [ 645.666264] Process bash (pid: 26392, threadinfo ffff88005bdda000, task ffff88005dbf0000) kernel: [ 645.666264] Stack: kernel: [ 645.666264] ffff880057bf2020 ffff88005bddbed0 0000000000000100 ffffffff8105866b kernel: [ 645.666264] 0000000000000000 ffff880057bf2020 0000000000000000 000000000151e008 kernel: [ 645.666264] 000000000151e008 0000000001399448 ffff88005bddbf08 ffffffff81195237 kernel: [ 645.666264] Call Trace: kernel: [ 645.666264] [<ffffffff8105866b>] ? unshare_files+0x3b/0xb0 kernel: [ 645.666264] [<ffffffff81195237>] do_execve_common.isra.24+0xc7/0x330 kernel: [ 645.666264] [<ffffffff811954b8>] do_execve+0x18/0x20 kernel: [ 645.666264] [<ffffffff8119575d>] sys_execve+0x3d/0x60 kernel: [ 645.666264] [<ffffffff816abe89>] stub_execve+0x69/0xc0 kernel: [ 645.666264] Code: 00 4d 8b 04 24 65 4c 03 04 25 08 dc 00 00 49 8b 50 08 4d 8b 28 4d 85 ed 0f 84 ef 00 00 00 49 63 44 24 20 49 8b 3c 24 48 8d 4a 01 <49> 8b 5c 05 00 4c 89 e8 65 48 0f c7 0f 0f 94 c0 84 c0 74 c2 49 kernel: [ 645.666264] RIP [<ffffffff8117dd0f>] kmem_cache_alloc_trace+0x5f/0x160 kernel: [ 645.666264] RSP <ffff88005bddbe58> kernel: [ 645.666264] CR2: 0000000000010001 kernel: [ 645.689292] ---[ end trace ded81b9e3753ac53 ]--- kernel: [ 661.264133] BUG: unable to handle kernel paging request at 0000000000010001 kernel: [ 661.264803] IP: [<ffffffff8117db95>] kmem_cache_alloc_node+0x95/0x1b0 kernel: [ 661.265405] PGD 0 kernel: [ 661.265610] Oops: 0000 [#7] SMP kernel: [ 661.265935] Modules linked in: xfs raid1 xfrm_user xfrm4_tunnel tunnel4 ipcomp xfrm_ipcomp esp4 ah4 scst_vdisk(O) iscsi_scst(O) scst(O) dm_zcache(O) dm_btrfs(O) btrfs(O) libcrc32c dm_iostat(O) md_mod deflate zlib_deflate ctr twofish_generic twofish_x86_64_3way glue_helper lrw xts gf128mul twofish_x86_64 twofish_common camellia_generic serpent_generic blowfish_generic blowfish_x86_64 blowfish_common cast5_generic cast_common des_generic xcbc rmd160 sha512_generic crypto_null af_key xfrm_algo dm_round_robin kvm vfat fat ppdev nfsd(O) nfs_acl microcode dm_multipath psmouse serio_raw parport_pc i2c_piix4 mac_hid lp nfsv4 auth_rpcgss parport nfs fscache lockd sunrpc floppy kernel: [ 661.268028] CPU 3 kernel: [ 661.268028] Pid: 6570, comm: udevd Tainted: G D O 3.8.13-557-generic #1382000791 Bochs Bochs kernel: [ 661.268028] RIP: 0010:[<ffffffff8117db95>] [<ffffffff8117db95>] kmem_cache_alloc_node+0x95/0x1b0 kernel: [ 661.268028] RSP: 0018:ffff88005b08dae8 EFLAGS: 00010246 kernel: [ 661.268028] RAX: 0000000000000000 RBX: ffff880069dcb740 RCX: 000000000003b9be kernel: [ 661.268028] RDX: 000000000003b9bd RSI: 00000000000004d0 RDI: 0000000000016c60 kernel: [ 661.268028] RBP: ffff88005b08db58 R08: ffff88007fd96c60 R09: ffffffff81588007 kernel: [ 661.268028] R10: 0000000000004000 R11: 0000000000000246 R12: ffff88007d001900 kernel: [ 661.268028] R13: 0000000000010001 R14: 00000000ffffffff R15: 00000000000004d0 kernel: [ 661.268028] FS: 00007f33c38f97c0(0000) GS:ffff88007fd80000(0000) knlGS:0000000000000000 kernel: [ 661.268028] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 kernel: [ 661.268028] CR2: 0000000000010001 CR3: 000000005b082000 CR4: 00000000000006e0 kernel: [ 661.268028] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 kernel: [ 661.268028] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 kernel: [ 661.268028] Process udevd (pid: 6570, threadinfo ffff88005b08c000, task ffff8800684fdc00) kernel: [ 661.268028] Stack: kernel: [ 661.268028] 0000000000000286 ffff88005b7ac1d0 ffff88005b08db18 ffffffff81588007 kernel: [ 661.268028] 0000000000000001 ffff88005b08dbc8 ffff88005b08db68 ffffffff811a07f7 kernel: [ 661.268028] 00000099f2d3c9cb ffff880069dcb740 00000000000004d0 00000000ffffffff kernel: [ 661.268028] Call Trace: kernel: [ 661.268028] [<ffffffff81588007>] ? __alloc_skb+0x57/0x2a0 kernel: [ 661.268028] [<ffffffff811a07f7>] ? poll_freewait+0x57/0xe0 kernel: [ 661.268028] [<ffffffff81588007>] __alloc_skb+0x57/0x2a0 kernel: [ 661.268028] [<ffffffff8158065a>] sock_alloc_send_pskb+0x11a/0x2b0 kernel: [ 661.268028] [<ffffffff816268ad>] unix_dgram_sendmsg+0x1bd/0x660 kernel: [ 661.268028] [<ffffffff81334a9d>] ? cpumask_any_but+0x2d/0x40 kernel: [ 661.268028] [<ffffffff8104e469>] ? flush_tlb_page+0x49/0xb0 kernel: [ 661.268028] [<ffffffff8157b0a2>] sock_sendmsg+0xd2/0xf0 kernel: [ 661.268028] [<ffffffff816a6d08>] ? __do_page_fault+0x2a8/0x560 kernel: [ 661.268028] [<ffffffff8157e50d>] sys_sendto+0x13d/0x190 kernel: [ 661.268028] [<ffffffff81090d6a>] ? wake_up_new_task+0xfa/0x190 kernel: [ 661.268028] [<ffffffff810ac71c>] ? ktime_get_ts+0x4c/0xe0 kernel: [ 661.268028] [<ffffffff816a6fce>] ? do_page_fault+0xe/0x10 kernel: [ 661.268028] [<ffffffff816a66a5>] ? do_async_page_fault+0x35/0x90 kernel: [ 661.268028] [<ffffffff816ab919>] system_call_fastpath+0x16/0x1b kernel: [ 661.268028] Code: 1d 4c 89 c9 44 89 f2 44 89 fe 4c 89 e7 e8 ca 29 51 00 49 89 c5 eb 31 0f 1f 80 00 00 00 00 49 63 44 24 20 49 8b 3c 24 48 8d 4a 01 <49> 8b 5c 05 00 4c 89 e8 65 48 0f c7 0f 0f 94 c0 84 c0 74 93 49 kernel: [ 661.268028] RIP [<ffffffff8117db95>] kmem_cache_alloc_node+0x95/0x1b0 kernel: [ 661.268028] RSP <ffff88005b08dae8> kernel: [ 661.268028] CR2: 0000000000010001 kernel: [ 661.295443] ---[ end trace ded81b9e3753ac54 ]--- kernel: [ 705.043870] BUG: unable to handle kernel paging request at 0000000000010001 kernel: [ 705.044572] IP: [<ffffffff8117ee4b>] kmem_cache_alloc+0x5b/0x170 kernel: [ 705.045145] PGD 55bbe067 PUD 57794067 PMD 0 kernel: [ 705.045603] Oops: 0000 [#8] SMP kernel: [ 705.045938] Modules linked in: xfs raid1 xfrm_user xfrm4_tunnel tunnel4 ipcomp xfrm_ipcomp esp4 ah4 scst_vdisk(O) iscsi_scst(O) scst(O) dm_zcache(O) dm_btrfs(O) btrfs(O) libcrc32c dm_iostat(O) md_mod deflate zlib_deflate ctr twofish_generic twofish_x86_64_3way glue_helper lrw xts gf128mul twofish_x86_64 twofish_common camellia_generic serpent_generic blowfish_generic blowfish_x86_64 blowfish_common cast5_generic cast_common des_generic xcbc rmd160 sha512_generic crypto_null af_key xfrm_algo dm_round_robin kvm vfat fat ppdev nfsd(O) nfs_acl microcode dm_multipath psmouse serio_raw parport_pc i2c_piix4 mac_hid lp nfsv4 auth_rpcgss parport nfs fscache lockd sunrpc floppy kernel: [ 705.047420] CPU 3 kernel: [ 705.047420] Pid: 8072, comm: bash Tainted: G D O 3.8.13-557-generic #1382000791 Bochs Bochs kernel: [ 705.047420] RIP: 0010:[<ffffffff8117ee4b>] [<ffffffff8117ee4b>] kmem_cache_alloc+0x5b/0x170 kernel: [ 705.047420] RSP: 0018:ffff880055909e08 EFLAGS: 00010202 kernel: [ 705.047420] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 000000000003b9be kernel: [ 705.047420] RDX: 000000000003b9bd RSI: 00000000000080d0 RDI: 0000000000016c60 kernel: [ 705.047420] RBP: ffff880055909e58 R08: ffff88007fd96c60 R09: 0000000000000000 kernel: [ 705.047420] R10: 000000000000c6e0 R11: ffff88007fd80000 R12: ffff88007d001900 kernel: [ 705.047420] R13: 0000000000010001 R14: 00000000000080d0 R15: ffffffff8118f2c8 kernel: [ 705.047420] FS: 00007f5bfcd56700(0000) GS:ffff88007fd80000(0000) knlGS:0000000000000000 kernel: [ 705.047420] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 kernel: [ 705.047420] CR2: 0000000000010001 CR3: 0000000057785000 CR4: 00000000000006e0 kernel: [ 705.047420] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 kernel: [ 705.047420] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 kernel: [ 705.047420] Process bash (pid: 8072, threadinfo ffff880055908000, task ffff88005ad45c00) kernel: [ 705.047420] Stack: kernel: [ 705.047420] ffff880055909e68 ffffffff8117ee9b ffff880055909e28 ffffffff811a55e4 kernel: [ 705.047420] 00000000000000c0 0000000000000000 ffff88007936d240 0000000000000002 kernel: [ 705.047420] ffffffff8181bca0 00000000ffffffff ffff880055909e78 ffffffff8118f2c8 kernel: [ 705.047420] Call Trace: kernel: [ 705.047420] [<ffffffff8117ee9b>] ? kmem_cache_alloc+0xab/0x170 kernel: [ 705.047420] [<ffffffff811a55e4>] ? __d_alloc+0x34/0x180 kernel: [ 705.047420] [<ffffffff8118f2c8>] get_empty_filp+0x58/0x180 kernel: [ 705.047420] [<ffffffff8118f41b>] alloc_file+0x2b/0xd0 kernel: [ 705.047420] [<ffffffff81197129>] create_pipe_files+0x89/0x130 kernel: [ 705.047420] [<ffffffff81197212>] __do_pipe_flags+0x42/0xf0 kernel: [ 705.047420] [<ffffffff81197330>] sys_pipe2+0x20/0xa0 kernel: [ 705.047420] [<ffffffff8106fa78>] ? sys_rt_sigprocmask+0xa8/0xd0 kernel: [ 705.047420] [<ffffffff811973c0>] sys_pipe+0x10/0x20 kernel: [ 705.047420] [<ffffffff816ab919>] system_call_fastpath+0x16/0x1b kernel: [ 705.047420] Code: 00 4d 8b 04 24 65 4c 03 04 25 08 dc 00 00 49 8b 50 08 4d 8b 28 4d 85 ed 0f 84 03 01 00 00 49 63 44 24 20 49 8b 3c 24 48 8d 4a 01 <49> 8b 5c 05 00 4c 89 e8 65 48 0f c7 0f 0f 94 c0 84 c0 74 c2 49 kernel: [ 705.047420] RIP [<ffffffff8117ee4b>] kmem_cache_alloc+0x5b/0x170 kernel: [ 705.047420] RSP <ffff880055909e08> kernel: [ 705.047420] CR2: 0000000000010001 [2] kernel: [ 381.277746] XFS (dm-22): metadata I/O error: block 0x7600028 ("xlog_iodone") error 125 numblks 64 kernel: [ 381.279335] XFS (dm-22): xfs_do_force_shutdown(0x2) called from line 1115 of file /mnt/compile/linux-stable/source/fs/xfs/xfs_log.c. Return address = 0xffffffffa05d5fa1 kernel: [ 381.279392] XFS (dm-22): Log I/O Error Detected. Shutting down filesystem kernel: [ 381.279488] XFS (dm-22): Unable to update superblock counters. Freespace may not be correct on next mount. kernel: [ 381.279495] XFS (dm-22): xfs_log_force: error 5 returned. kernel: [ 381.281015] XFS (dm-22): Please umount the filesystem and rectify the problem(s) kernel: [ 381.294706] [29579]dm_iostat [dm_iostat_dtr:157] Removing iostats (253:22) from 253:21 kernel: [ 381.508073] kmemleak: Cannot insert 0xffff88007947bb00 into the object search tree (overlaps existing) kernel: [ 381.508105] Pid: 16141, comm: udevd Tainted: G O 3.8.13-557-generic #1382000791 kernel: [ 381.508109] Call Trace: kernel: [ 381.508140] [<ffffffff81189e66>] create_object+0x286/0x2e0 kernel: [ 381.508149] [<ffffffff8118f2c8>] ? get_empty_filp+0x58/0x180 kernel: [ 381.508172] [<ffffffff816798e6>] kmemleak_alloc+0x26/0x50 kernel: [ 381.508181] [<ffffffff8117ee9b>] kmem_cache_alloc+0xab/0x170 kernel: [ 381.508187] [<ffffffff8118f2c8>] get_empty_filp+0x58/0x180 kernel: [ 381.508196] [<ffffffff8119cb18>] path_openat+0x48/0x4d0 kernel: [ 381.508201] [<ffffffff81189e0b>] ? create_object+0x22b/0x2e0 kernel: [ 381.508206] [<ffffffff8119d742>] do_filp_open+0x42/0xa0 kernel: [ 381.508214] [<ffffffff816a2ace>] ? _raw_spin_lock+0xe/0x20 kernel: [ 381.508238] [<ffffffff811aace5>] ? __alloc_fd+0xe5/0x170 kernel: [ 381.508244] [<ffffffff8118d37e>] do_sys_open+0xfe/0x1e0 kernel: [ 381.508256] [<ffffffff81014a24>] ? do_notify_resume+0x74/0x80 kernel: [ 381.508262] [<ffffffff8118d4a4>] sys_openat+0x14/0x20 kernel: [ 381.508269] [<ffffffff816ab919>] system_call_fastpath+0x16/0x1b kernel: [ 381.508282] kmemleak: Kernel memory leak detector disabled kernel: [ 381.508287] kmemleak: Object 0xffff88007947ba40 (size 448): kernel: [ 381.508290] kmemleak: comm "modprobe", pid 3708, jiffies 4294921709 kernel: [ 381.508292] kmemleak: min_count = 1 kernel: [ 381.508294] kmemleak: count = 0 kernel: [ 381.508296] kmemleak: flags = 0x1 kernel: [ 381.508298] kmemleak: checksum = 0 kernel: [ 381.508300] kmemleak: backtrace: kernel: [ 381.508302] [<ffffffff816798e6>] kmemleak_alloc+0x26/0x50 kernel: [ 381.508309] [<ffffffff8117ee9b>] kmem_cache_alloc+0xab/0x170 kernel: [ 381.508314] [<ffffffff81131ed5>] mempool_alloc_slab+0x15/0x20 kernel: [ 381.508328] [<ffffffff81132131>] mempool_create_node+0xe1/0x160 kernel: [ 381.508334] [<ffffffff811321ca>] mempool_create+0x1a/0x20 kernel: [ 381.508339] [<ffffffff811c4d04>] bioset_create+0x74/0xe0 kernel: [ 381.508346] [<ffffffffa047b5b2>] init_module+0x122/0x240 [dm_zcache] kernel: [ 381.508368] [<ffffffff8100206f>] do_one_initcall+0x3f/0x170 kernel: [ 381.508375] [<ffffffff8168d5de>] do_init_module+0x80/0x1d1 kernel: [ 381.508382] [<ffffffff810c12ca>] load_module+0x3aa/0x530 kernel: [ 381.508399] [<ffffffff810c14cb>] sys_init_module+0x7b/0x80 kernel: [ 381.508405] [<ffffffff816ab919>] system_call_fastpath+0x16/0x1b kernel: [ 381.508409] [<ffffffffffffffff>] 0xffffffffffffffff kernel: [ 381.508832] kmemleak: Automatic memory scanning thread ended kernel: [ 381.508556] BUG: unable to handle kernel paging request at 000000170000000a kernel: [ 381.511347] IP: [<ffffffff8117ee4b>] kmem_cache_alloc+0x5b/0x170 kernel: [ 381.512031] PGD 0 kernel: [ 381.512031] Oops: 0000 [#1] SMP kernel: [ 381.512031] Modules linked in: xfs raid1 xfrm_user xfrm4_tunnel tunnel4 ipcomp xfrm_ipcomp esp4 ah4 scst_vdisk(O) iscsi_scst(O) scst(O) dm_zcache(O) dm_btrfs(O) btrfs(O) libcrc32c dm_iostat(O) md_mod deflate zlib_deflate ctr twofish_generic twofish_x86_64_3way glue_helper lrw xts gf128mul twofish_x86_64 twofish_common camellia_generic serpent_generic blowfish_generic blowfish_x86_64 blowfish_common cast5_generic cast_common des_generic xcbc rmd160 sha512_generic dm_round_robin crypto_null af_key xfrm_algo kvm ppdev vfat fat dm_multipath nfsd(O) nfs_acl psmouse serio_raw parport_pc i2c_piix4 microcode mac_hid nfsv4 auth_rpcgss nfs fscache lockd lp sunrpc parport floppy kernel: [ 381.524857] CPU 3 kernel: [ 381.524857] Pid: 16141, comm: udevd Tainted: G O 3.8.13-557-generic #1382000791 Bochs Bochs kernel: [ 381.524857] RIP: 0010:[<ffffffff8117ee4b>] [<ffffffff8117ee4b>] kmem_cache_alloc+0x5b/0x170 kernel: [ 381.524857] RSP: 0018:ffff88005c617d28 EFLAGS: 00010206 kernel: [ 381.524857] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 00000000000427ee kernel: [ 381.524857] RDX: 00000000000427ed RSI: 00000000000080d0 RDI: 0000000000016c60 kernel: [ 381.524857] RBP: ffff88005c617d78 R08: ffff88007fd96c60 R09: 0000000000000000 kernel: [ 381.524857] R10: 8080808080808080 R11: 0000000000000000 R12: ffff88007d001900 kernel: [ 381.524857] R13: 000000170000000a R14: 00000000000080d0 R15: ffffffff8118f2c8 kernel: [ 381.524857] FS: 00007f77822c37c0(0000) GS:ffff88007fd80000(0000) knlGS:0000000000000000 kernel: [ 381.524857] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 kernel: [ 381.544074] CR2: 000000170000000a CR3: 000000003687c000 CR4: 00000000000006e0 kernel: [ 381.544074] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 kernel: [ 381.544074] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 kernel: [ 381.548728] Process udevd (pid: 16141, threadinfo ffff88005c616000, task ffff88005385c500) kernel: [ 381.548728] Stack: kernel: [ 381.548728] 0000000000000001 00007fff2038ac28 0000000000000000 ffff880062543480 kernel: [ 381.552737] ffff88005c617d88 0000000000000000 ffff880057ade540 ffff88005c617f28 kernel: [ 381.554800] ffff880067564000 00000000ffffff9c ffff88005c617d98 ffffffff8118f2c8 kernel: [ 381.554800] Call Trace: kernel: [ 381.554800] [<ffffffff8118f2c8>] get_empty_filp+0x58/0x180 kernel: [ 381.554800] [<ffffffff8119cb18>] path_openat+0x48/0x4d0 kernel: [ 381.554800] [<ffffffff811993e9>] ? putname+0x29/0x40 kernel: [ 381.554800] [<ffffffff8119d6a3>] ? user_path_at_empty+0x63/0xa0 kernel: [ 381.554800] [<ffffffff816a068d>] ? mutex_lock+0x1d/0x50 kernel: [ 381.554800] [<ffffffff8119d742>] do_filp_open+0x42/0xa0 kernel: [ 381.554800] [<ffffffff816a2ace>] ? _raw_spin_lock+0xe/0x20 kernel: [ 381.554800] [<ffffffff811aace5>] ? __alloc_fd+0xe5/0x170 kernel: [ 381.554800] [<ffffffff8118d37e>] do_sys_open+0xfe/0x1e0 kernel: [ 381.554800] [<ffffffff8118d481>] sys_open+0x21/0x30 kernel: [ 381.554800] [<ffffffff816ab919>] system_call_fastpath+0x16/0x1b kernel: [ 381.554800] Code: 00 4d 8b 04 24 65 4c 03 04 25 08 dc 00 00 49 8b 50 08 4d 8b 28 4d 85 ed 0f 84 03 01 00 00 49 63 44 24 20 49 8b 3c 24 48 8d 4a 01 <49> 8b 5c 05 00 4c 89 e8 65 48 0f c7 0f 0f 94 c0 84 c0 74 c2 49 kernel: [ 381.554800] RIP [<ffffffff8117ee4b>] kmem_cache_alloc+0x5b/0x170 kernel: [ 381.554800] RSP <ffff88005c617d28> kernel: [ 381.554800] CR2: 000000170000000a kernel: [ 381.579253] ---[ end trace 7b5bb1322a28df6d ]--- kernel: [ 381.580826] BUG: unable to handle kernel paging request at 000000170000000a kernel: [ 381.582404] IP: [<ffffffff8117ee4b>] kmem_cache_alloc+0x5b/0x170 kernel: [ 381.584100] PGD 0 kernel: [ 381.584267] Oops: 0000 [#2] SMP kernel: [ 381.585161] Modules linked in: xfs raid1 xfrm_user xfrm4_tunnel tunnel4 ipcomp xfrm_ipcomp esp4 ah4 scst_vdisk(O) iscsi_scst(O) scst(O) dm_zcache(O) dm_btrfs(O) btrfs(O) libcrc32c dm_iostat(O) md_mod deflate zlib_deflate ctr twofish_generic twofish_x86_64_3way glue_helper lrw xts gf128mul twofish_x86_64 twofish_common camellia_generic serpent_generic blowfish_generic blowfish_x86_64 blowfish_common cast5_generic cast_common des_generic xcbc rmd160 sha512_generic dm_round_robin crypto_null af_key xfrm_algo kvm ppdev vfat fat dm_multipath nfsd(O) nfs_acl psmouse serio_raw parport_pc i2c_piix4 microcode mac_hid nfsv4 auth_rpcgss nfs fscache lockd lp sunrpc parport floppy kernel: [ 381.595808] CPU 3 kernel: [ 381.595808] Pid: 16112, comm: udevd Tainted: G D O 3.8.13-557-generic #1382000791 Bochs Bochs kernel: [ 381.595808] RIP: 0010:[<ffffffff8117ee4b>] [<ffffffff8117ee4b>] kmem_cache_alloc+0x5b/0x170 kernel: [ 381.604262] RSP: 0018:ffff880067469d28 EFLAGS: 00010206 kernel: [ 381.604262] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 00000000000427ee kernel: [ 381.604262] RDX: 00000000000427ed RSI: 00000000000080d0 RDI: 0000000000016c60 kernel: [ 381.604262] RBP: ffff880067469d78 R08: ffff88007fd96c60 R09: 0000000000000000 kernel: [ 381.604262] R10: 8080808080808080 R11: 0000000000000000 R12: ffff88007d001900 kernel: [ 381.604262] R13: 000000170000000a R14: 00000000000080d0 R15: ffffffff8118f2c8 kernel: [ 381.604262] FS: 00007f77822c37c0(0000) GS:ffff88007fd80000(0000) knlGS:0000000000000000 kernel: [ 381.604262] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 kernel: [ 381.604262] CR2: 000000170000000a CR3: 00000000797c1000 CR4: 00000000000006e0 kernel: [ 381.604262] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 kernel: [ 381.604262] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 kernel: [ 381.604262] Process udevd (pid: 16112, threadinfo ffff880067468000, task ffff88007aed8000) kernel: [ 381.604262] Stack: kernel: [ 381.604262] ffff880067469df8 0000000000001000 00000000528e1772 00000000261c9b68 kernel: [ 381.604262] 0000000000000000 0000000000000000 ffff880078e5ce40 ffff880067469f28 kernel: [ 381.604262] ffff880067560000 00000000ffffff9c ffff880067469d98 ffffffff8118f2c8 kernel: [ 381.604262] Call Trace: kernel: [ 381.604262] [<ffffffff8118f2c8>] get_empty_filp+0x58/0x180 kernel: [ 381.604262] [<ffffffff8119cb18>] path_openat+0x48/0x4d0 kernel: [ 381.604262] [<ffffffff811a9619>] ? notify_change+0x289/0x390 kernel: [ 381.604262] [<ffffffff8118f9a3>] ? __sb_end_write+0x33/0x70 kernel: [ 381.604262] [<ffffffff8119d742>] do_filp_open+0x42/0xa0 kernel: [ 381.604262] [<ffffffff816a2ace>] ? _raw_spin_lock+0xe/0x20 kernel: [ 381.604262] [<ffffffff811aace5>] ? __alloc_fd+0xe5/0x170 kernel: [ 381.604262] [<ffffffff8118d37e>] do_sys_open+0xfe/0x1e0 kernel: [ 381.604262] [<ffffffff811be489>] ? sys_utimensat+0x69/0xa0 kernel: [ 381.604262] [<ffffffff8118d4a4>] sys_openat+0x14/0x20 kernel: [ 381.604262] [<ffffffff816ab919>] system_call_fastpath+0x16/0x1b kernel: [ 381.604262] Code: 00 4d 8b 04 24 65 4c 03 04 25 08 dc 00 00 49 8b 50 08 4d 8b 28 4d 85 ed 0f 84 03 01 00 00 49 63 44 24 20 49 8b 3c 24 48 8d 4a 01 <49> 8b 5c 05 00 4c 89 e8 65 48 0f c7 0f 0f 94 c0 84 c0 74 c2 49 kernel: [ 381.604262] RIP [<ffffffff8117ee4b>] kmem_cache_alloc+0x5b/0x170 -- To unsubscribe from this list: send the line "unsubscribe linux-xfs" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html