NeilBrown <neilb@xxxxxxx> writes: > On Wed, 03 Jun 2015 16:20:21 -0400 Jes Sorensen <Jes.Sorensen@xxxxxxxxxx> > wrote: > >> Neil, >> >> I was running testing on the current 4.1-rc6 tree (Linus' top of trunk >> 8cd9234c64c584432f6992fe944ca9e46ca8ea76) and I am seeing the following >> OOPS which is reproducible. >> >> It shows up when running the mdadm test suite, 07changelevelintr to be >> specific. >> >> Is this something you have seen? >> >> Cheers, >> Jes >> >> ------------[ cut here ]------------ >> kernel BUG at drivers/md/raid5.c:5391! > > No, I haven't seen that. And I've been running the test suite quite a bit > lately. > > Can you get it to print out the relevant numbers? Include > readpos/writepos/safepos too. This enough? Let me know if you need more. I suspect this started happening with the changes that went in between 4.1-rc5 and 4.1-rc6. I will try to bisect it tomorrow. Cheers, Jes mddev->dev_sectors: 0x9800, reshape_sectors: 0x0200 stripe_addr: fffffffffffffdff, sector_nr 0, readpos 511, writepos -513, safepos 512 ------------[ cut here ]------------ kernel BUG at drivers/md/raid5.c:5394! invalid opcode: 0000 [#1] SMP Modules linked in: raid456 async_raid6_recov async_memcpy async_pq async_xor xor async_tx raid6_pq nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 iptable_filter ip_tables tun bridge stp llc x86_pkg_temp_thermal coretemp kvm_intel kvm xfs crc32c_intel ghash_clmulni_intel aesni_intel aes_x86_64 glue_helper lrw libcrc32c nfsd gf128mul ablk_helper auth_rpcgss oid_registry exportfs nfs_acl lockd cryptd iTCO_wdt grace sunrpc microcode raid0 ppdev iTCO_vendor_support shpchp i2c_i801 i2c_core pcspkr lpc_ich mfd_core parport_pc acpi_cpufreq parport video uinput autofs4 ext4 mbcache jbd2 sd_mod e1000e ptp pps_core ahci r8169 libahci mii dm_mirror dm_region_hash dm_log dm_mod ipv6 CPU: 1 PID: 14648 Comm: md0_resync Not tainted 4.1.0-rc2+ #5 Hardware name: Intel Corporation S1200BTL/S1200BTL, BIOS S1200BT.86B.02.00.0035.030220120927 03/02/2012 task: ffff880234b0f080 ti: ffff88022aadc000 task.ti: ffff88022aadc000 RIP: 0010:[<ffffffffa05db722>] [<ffffffffa05db722>] reshape_request+0x8d2/0x8e0 [raid456] RSP: 0018:ffff88022aadfb68 EFLAGS: 00010296 RAX: 0000000000000087 RBX: ffff88022a8d2400 RCX: 0000000000000000 RDX: 0000000000000001 RSI: ffff88023ee2d368 RDI: ffff88023ee2d368 RBP: ffff88022aadfc28 R08: 0000000000000400 R09: ffffffff81d6c864 R10: 000000000000058f R11: 000000000000058e R12: ffff88022aadfd0c R13: 0000000000000000 R14: ffff88022aadfd0c R15: ffff8800a778c800 FS: 0000000000000000(0000) GS:ffff88023ee20000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007fc8ddece000 CR3: 0000000001a0b000 CR4: 00000000001406e0 Stack: fffffffffffffdff 0000000000000200 0000000000000000 0000000000000200 0000000000000001 0000000000000000 0000000000000001 fffffffffffffdff 0000020000000001 ffffffff815b2d23 ffff880234b0f080 ffff880233748f00 Call Trace: [<ffffffff815b2d23>] ? __schedule+0x383/0x8e0 [<ffffffffa05dba5e>] sync_request+0x32e/0x3a0 [raid456] [<ffffffff81092708>] ? __wake_up+0x48/0x60 [<ffffffff8148b564>] md_do_sync+0x8f4/0xe90 [<ffffffff81487888>] md_thread+0x128/0x140 [<ffffffff81487760>] ? find_pers+0x80/0x80 [<ffffffff81487760>] ? find_pers+0x80/0x80 [<ffffffff81071b49>] kthread+0xc9/0xe0 [<ffffffff810edd76>] ? __audit_syscall_exit+0x1e6/0x280 [<ffffffff81071a80>] ? kthread_create_on_node+0x170/0x170 [<ffffffff815b6b92>] ret_from_fork+0x42/0x70 [<ffffffff81071a80>] ? kthread_create_on_node+0x170/0x170 Code: 48 89 44 24 08 48 8b 85 78 ff ff ff 48 c7 c7 e0 3d 5e a0 4c 8b 85 68 ff ff ff 8b 55 84 48 89 04 24 48 89 c1 31 c0 e8 fe 10 fd e0 <0f> 0b 66 66 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 41 57 41 RIP [<ffffffffa05db722>] reshape_request+0x8d2/0x8e0 [raid456] RSP <ffff88022aadfb68> ---[ end trace 7eefd62cedfb2b13 ]--- BUG: unable to handle kernel NULL pointer dereference at (null) IP: [<ffffffff81073c2f>] exit_creds+0x1f/0x70 PGD 2365b1067 PUD 235280067 PMD 0 Oops: 0000 [#2] SMP Modules linked in: raid456 async_raid6_recov async_memcpy async_pq async_xor xor async_tx raid6_pq nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 iptable_filter ip_tables tun bridge stp llc x86_pkg_temp_thermal coretemp kvm_intel kvm xfs crc32c_intel ghash_clmulni_intel aesni_intel aes_x86_64 glue_helper lrw libcrc32c nfsd gf128mul ablk_helper auth_rpcgss oid_registry exportfs nfs_acl lockd cryptd iTCO_wdt grace sunrpc microcode raid0 ppdev iTCO_vendor_support shpchp i2c_i801 i2c_core pcspkr lpc_ich mfd_core parport_pc acpi_cpufreq parport video uinput autofs4 ext4 mbcache jbd2 sd_mod e1000e ptp pps_core ahci r8169 libahci mii dm_mirror dm_region_hash dm_log dm_mod ipv6 CPU: 1 PID: 14535 Comm: mdadm Tainted: G D 4.1.0-rc2+ #5 Hardware name: Intel Corporation S1200BTL/S1200BTL, BIOS S1200BT.86B.02.00.0035.030220120927 03/02/2012 task: ffff8800a6b67100 ti: ffff880232ae4000 task.ti: ffff880232ae4000 RIP: 0010:[<ffffffff81073c2f>] [<ffffffff81073c2f>] exit_creds+0x1f/0x70 RSP: 0018:ffff880232ae7ca8 EFLAGS: 00010292 RAX: 0000000000000000 RBX: ffff880234b0f080 RCX: ffff880232ae7d00 RDX: 0000000000005a31 RSI: 0000000000000296 RDI: 0000000000000000 RBP: ffff880232ae7cb8 R08: 0000000000000000 R09: 0000000000000000 R10: 000000000000000b R11: 0000000000000246 R12: ffff880234b0f080 R13: 000000000000000b R14: ffff8802338f0000 R15: 0000000000000004 FS: 00007f70bfc97740(0000) GS:ffff88023ee20000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 0000000000000000 CR3: 0000000234966000 CR4: 00000000001406e0 Stack: ffff880232ae7cb8 ffff880234b0f080 ffff880232ae7cd8 ffffffff8105265a 0000000000000000 ffff880234b0f080 ffff880232ae7d08 ffffffff81072058 ffff880200000004 ffff8802361046c0 ffff8800a778c950 0000000000000004 Call Trace: [<ffffffff8105265a>] __put_task_struct+0x4a/0x130 [<ffffffff81072058>] kthread_stop+0x88/0x100 [<ffffffff814878e5>] md_unregister_thread+0x45/0x80 [<ffffffff8148e32d>] md_reap_sync_thread+0x1d/0x1a0 [<ffffffff8148e650>] action_store+0x1a0/0x290 [<ffffffff8105cd5d>] ? ns_capable+0x2d/0x60 [<ffffffff8148bb7b>] md_attr_store+0x7b/0xd0 [<ffffffff8120f45d>] sysfs_kf_write+0x3d/0x50 [<ffffffff8120ebba>] kernfs_fop_write+0x12a/0x180 [<ffffffff81199878>] __vfs_write+0x28/0xf0 [<ffffffff8119c479>] ? __sb_start_write+0x49/0xf0 [<ffffffff81230873>] ? security_file_permission+0x23/0xa0 [<ffffffff81199f69>] vfs_write+0xa9/0x1b0 [<ffffffff8119ad36>] SyS_write+0x46/0xb0 [<ffffffff810edb34>] ? __audit_syscall_entry+0xb4/0x110 [<ffffffff815b67d7>] system_call_fastpath+0x12/0x6a Code: 0f 84 37 fe ff ff e9 10 fe ff ff 90 0f 1f 44 00 00 55 48 89 e5 53 48 89 fb 48 83 ec 08 48 8b 87 c8 09 00 00 48 8b bf c0 09 00 00 <8b> 00 48 c7 83 c0 09 00 00 00 00 00 00 f0 ff 0f 74 1f 48 8b bb RIP [<ffffffff81073c2f>] exit_creds+0x1f/0x70 RSP <ffff880232ae7ca8> CR2: 0000000000000000 ---[ end trace 7eefd62cedfb2b14 ]--- > > Thanks, > NeilBrown > > >> invalid opcode: 0000 [#1] SMP >> Modules linked in: raid456 async_raid6_recov async_memcpy async_pq >> async_xor xor async_tx raid6_pq nf_conntrack_ipv4 nf_defrag_ipv4 >> xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 iptable_filter >> ip_tables tun bridge stp llc xfs x86_pkg_temp_thermal coretemp >> kvm_intel kvm crc32c_intel ghash_clmulni_intel libcrc32c aesni_intel >> aes_x86_64 glue_helper lrw gf128mul ablk_helper cryptd ppdev raid0 >> microcode pcspkr iTCO_wdt iTCO_vendor_support parport_pc i2c_i801 >> i2c_core parport shpchp lpc_ich mfd_core video acpi_cpufreq nfsd >> auth_rpcgss oid_registry exportfs nfs_acl lockd grace sunrpc uinput >> autofs4 ext4 mbcache jbd2 sd_mod e1000e ptp pps_core ahci r8169 mii >> libahci dm_mirror dm_region_hash dm_log dm_mod ipv6 >> CPU: 0 PID: 15142 Comm: md0_resync Tainted: G W 4.1.0-rc6+ #2 >> Hardware name: Intel Corporation S1200BTL/S1200BTL, BIOS >> S1200BT.86B.02.00.0035.030220120927 03/02/2012 >> task: ffff88022b2642d0 ti: ffff8802327e0000 task.ti: ffff8802327e0000 >> RIP: 0010:[<ffffffffa046d6f7>] [<ffffffffa046d6f7>] >> reshape_request+0x8a7/0x8b0 [raid456] >> RSP: 0018:ffff8802327e3b78 EFLAGS: 00010297 >> RAX: 0000000000000200 RBX: ffff88022b130400 RCX: 0000000000000200 >> RDX: 0000000000009801 RSI: 0000000000000200 RDI: 0000000000000001 >> RBP: ffff8802327e3c28 R08: 00000000000001ff R09: 0000000000000000 >> R10: fffffffffffffdff R11: 0000000000000001 R12: ffff8802327e3d0c >> R13: 0000000000000000 R14: ffff8802327e3d0c R15: ffff880234577000 >> FS: 0000000000000000(0000) GS:ffff88023ee00000(0000) knlGS:0000000000000000 >> CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 >> CR2: 00007f985b677140 CR3: 0000000001a0b000 CR4: 00000000001406f0 >> Stack: >> 0000000000000000 0000000000000200 0000000000000001 0000000000000000 >> 0000000000000000 fffffffffffffdff 0000020000000001 ffffffff815b34d3 >> ffff88022b2642d0 ffff880235f2f080 ffff8802327e3c08 ffff8802327e4000 >> Call Trace: >> [<ffffffff815b34d3>] ? __schedule+0x383/0x8e0 >> [<ffffffffa046da2e>] sync_request+0x32e/0x3a0 [raid456] >> [<ffffffff81092738>] ? __wake_up+0x48/0x60 >> [<ffffffff8148b974>] md_do_sync+0x8f4/0xe90 >> [<ffffffff810779bc>] ? update_rq_clock.part.89+0x1c/0x40 >> [<ffffffff81487c98>] md_thread+0x128/0x140 >> [<ffffffff81487b70>] ? find_pers+0x80/0x80 >> [<ffffffff81487b70>] ? find_pers+0x80/0x80 >> [<ffffffff81071b39>] kthread+0xc9/0xe0 >> [<ffffffff810eddd6>] ? __audit_syscall_exit+0x1e6/0x280 >> [<ffffffff81071a70>] ? kthread_create_on_node+0x170/0x170 >> [<ffffffff815b7352>] ret_from_fork+0x42/0x70 >> [<ffffffff81071a70>] ? kthread_create_on_node+0x170/0x170 >> Code: 00 00 be 03 00 00 00 e8 18 50 c2 e0 49 8d 7f 50 48 c7 c2 39 55 >> 47 a0 31 f6 e8 56 21 da e0 48 8b 45 88 e9 ae fb ff ff 0f 0b 0f 0b >> <0f> 0b 0f 1f 80 00 00 00 00 55 48 89 e5 41 57 41 56 41 55 41 54 >> RIP [<ffffffffa046d6f7>] reshape_request+0x8a7/0x8b0 [raid456] >> RSP <ffff8802327e3b78> >> ---[ end trace 1d72a07e246912ea ]--- >> BUG: unable to handle kernel NULL pointer dereference at (null) >> IP: [<ffffffff81073c1f>] exit_creds+0x1f/0x70 >> PGD bdc0c067 PUD a6697067 PMD 0 >> Oops: 0000 [#2] SMP >> Modules linked in: raid456 async_raid6_recov async_memcpy async_pq >> async_xor xor async_tx raid6_pq nf_conntrack_ipv4 nf_defrag_ipv4 >> xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 iptable_filter >> ip_tables tun bridge stp llc xfs x86_pkg_temp_thermal coretemp >> kvm_intel kvm crc32c_intel ghash_clmulni_intel libcrc32c aesni_intel >> aes_x86_64 glue_helper lrw gf128mul ablk_helper cryptd ppdev raid0 >> microcode pcspkr iTCO_wdt iTCO_vendor_support parport_pc i2c_i801 >> i2c_core parport shpchp lpc_ich mfd_core video acpi_cpufreq nfsd >> auth_rpcgss oid_registry exportfs nfs_acl lockd grace sunrpc uinput >> autofs4 ext4 mbcache jbd2 sd_mod e1000e ptp pps_core ahci r8169 mii >> libahci dm_mirror dm_region_hash dm_log dm_mod ipv6 >> CPU: 1 PID: 15030 Comm: mdadm Tainted: G D W 4.1.0-rc6+ #2 >> Hardware name: Intel Corporation S1200BTL/S1200BTL, BIOS >> S1200BT.86B.02.00.0035.030220120927 03/02/2012 >> task: ffff880233550110 ti: ffff8800a6cbc000 task.ti: ffff8800a6cbc000 >> RIP: 0010:[<ffffffff81073c1f>] [<ffffffff81073c1f>] exit_creds+0x1f/0x70 >> RSP: 0018:ffff8800a6cbfca8 EFLAGS: 00010292 >> RAX: 0000000000000000 RBX: ffff88022b2642d0 RCX: ffff8800a6cbfd00 >> RDX: 0000000000005110 RSI: 0000000000000296 RDI: 0000000000000000 >> RBP: ffff8800a6cbfcb8 R08: 0000000000000000 R09: 0000000000000000 >> R10: 000000000000000b R11: 0000000000000246 R12: ffff88022b2642d0 >> R13: 000000000000ff00 R14: 0000000000000004 R15: 0000000000000004 >> FS: 00007f70ff26f740(0000) GS:ffff88023ee20000(0000) knlGS:0000000000000000 >> CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 >> CR2: 0000000000000000 CR3: 00000000a66cf000 CR4: 00000000001406e0 >> Stack: >> ffff8800a6cbfcb8 ffff88022b2642d0 ffff8800a6cbfcd8 ffffffff8105264a >> 0000000000000000 ffff88022b2642d0 ffff8800a6cbfd08 ffffffff81072048 >> ffff880000000001 ffff880233cb9140 ffff880234577150 ffffffffa0477320 >> Call Trace: >> [<ffffffff8105264a>] __put_task_struct+0x4a/0x130 >> [<ffffffff81072048>] kthread_stop+0x88/0x100 >> [<ffffffff81487cf5>] md_unregister_thread+0x45/0x80 >> [<ffffffff8148e73d>] md_reap_sync_thread+0x1d/0x1a0 >> [<ffffffff8148ea68>] action_store+0x1a8/0x2b0 >> [<ffffffff8105cd4d>] ? ns_capable+0x2d/0x60 >> [<ffffffff8148bf8b>] md_attr_store+0x7b/0xd0 >> [<ffffffff8120f77d>] sysfs_kf_write+0x3d/0x50 >> [<ffffffff8120eeda>] kernfs_fop_write+0x12a/0x180 >> [<ffffffff81199ba8>] __vfs_write+0x28/0xf0 >> [<ffffffff8119c7a9>] ? __sb_start_write+0x49/0xf0 >> [<ffffffff81230b93>] ? security_file_permission+0x23/0xa0 >> [<ffffffff8119a299>] vfs_write+0xa9/0x1b0 >> [<ffffffff8119b066>] SyS_write+0x46/0xb0 >> [<ffffffff810edb94>] ? __audit_syscall_entry+0xb4/0x110 >> [<ffffffff815b6f97>] system_call_fastpath+0x12/0x6a >> Code: 0f 84 37 fe ff ff e9 10 fe ff ff 90 0f 1f 44 00 00 55 48 89 e5 >> 53 48 89 fb 48 83 ec 08 48 8b 87 c8 09 00 00 48 8b bf c0 09 00 00 >> <8b> 00 48 c7 83 c0 09 00 00 00 00 00 00 f0 ff 0f 74 1f 48 8b bb >> RIP [<ffffffff81073c1f>] exit_creds+0x1f/0x70 >> RSP <ffff8800a6cbfca8> >> CR2: 0000000000000000 >> ---[ end trace 1d72a07e246912eb ]--- >> -- >> To unsubscribe from this list: send the line "unsubscribe linux-raid" in >> the body of a message to majordomo@xxxxxxxxxxxxxxx >> More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
