On Mon, Dec 12, 2011 at 8:24 AM, NeilBrown <neilb@xxxxxxx> wrote:
> On Mon, 12 Dec 2011 08:02:33 +0200 Anssi Hannula <anssi.hannula@xxxxxx> wrote:
>
>> On Mon, Dec 12, 2011 at 7:42 AM, NeilBrown <neilb@xxxxxxx> wrote:
>> > On Mon, 12 Dec 2011 07:22:17 +0200 Anssi Hannula <anssi.hannula@xxxxxx> wrote:
>> >
>> >> On Mon, Dec 12, 2011 at 5:01 AM, NeilBrown <neilb@xxxxxxx> wrote:
>> >> > On Sun, 11 Dec 2011 09:03:14 +0200 Anssi Hannula <anssi.hannula@xxxxxx> wrote:
>> >> >
>> >> >> Hi!
>> >> >>
>> >> >> After I rebooted during a raid6 rebuild, the rebuild didn't start again.
>> >> >> Instead, there is a flood of "RAID conf printout"s that seemingly happen
>> >> >> on array activity.
>> >> >>
>> >> >> All the devices show up properly in --detail and two devices are marked
>> >> >> as "spare rebuilding", and I can access the contents of the array just
>> >> >> fine, but the rebuild doesn't actually start. Is this a bug or am I
>> >> >> missing something? :)
>> >> >>
>> >> >> I was initially on 2.6.38.8, but also tried 3.1.4 which seems to have
>> >> >> the same issue. mdadm is 3.1.5.
>> >> >>
>> >> >> I'm not using start_ro and writing to the array doesn't trigger a
>> >> >> rebuild either.
>> >> >>
>> >> >> Attached are --examine outputs before assembly, kernel log output on
>> >> >> assembly, /proc/mdstat and --detail after assembly (on 3.1.4).
>> >> >>
>> >> >
>> >> > Thank you for the very detailed problem report.
>> >>
>> >> Thanks for the quick response :)
>> >>
>> >> > Unfortunately it is a complete mystery to me what is happening.
>> >> >
>> >> > The repeated "RAID conf printout" messages are almost certainly coming from
>> >> > the end of raid5_remove_disk.
>> >> > It is being called from remove_and_add_spares for each of the two devices
>> >> > that are being rebuilt. raid5_remove_disk declines to remove them because it
>> >> > can keep rebuilding them.
>> >> >
>> >> > remove_and_add_spares then counts them and notes there are 2.
>> >> > md_check_recovery notes that this is > 0, so it should create a thread to run
>> >> > md_do_sync.
>> >> >
>> >> > md_do_sync should then print out a message like
>> >> > md: recovery of RAID array md0
>> >> >
>> >> > but it doesn't. So something went wrong.
>> >> > There are three reasons that md_do_sync might not print a message:
>> >> >
>> >> > 1/ MD_RECOVERY_DONE is set. As only md_do_sync ever sets it, that is
>> >> > unlikely, and in any case md_check_recovery clears it.
>> >> > 2/ mddev->ro != 0. It is only ever set to 0, 1, or 2. If it is 1 or 2
>> >> > then we would be able to see that in /proc/mdstat as a "(readonly)"
>> >> > status. But we don't.
>> >> > 3/ MD_RECOVERY_INTR is set. Again, md_check_recovery clears this. It does
>> >> > get set if kthread_should_stop() returns 'true', but that should only
>> >> > happen if kthread_stop() was called. That is only called by
>> >> > md_unregister_thread and I cannot see any way that could be call.
>> >> >
>> >> > So. No idea.
>> >> >
>> >> > Are you compiling these kernels yourself?
>> >>
>> >> Nope (used Mageia kernels), but I did now (3.1.5).
>> >>
>> >> > If so, could you:
>> >> > - put a printk in the top of md_do_sync to report the values of
>> >> > mddev->recovery and mddev->ro
>> >> > - print a message whenever md_unregister_thread is called
>> >> > - in md_check_recovery, in the
>> >> > if (mddev->ro) {
>> >> > /* Only thing we do on a ro array is remove
>> >> > * failed devices.
>> >> > */
>> >> > mdk_rdev_t *rdev;
>> >> >
>> >> > in statement, print the value of mddev->ro.
>> >> >
>> >> > Then see which of those printk's fire, and what they tell us.
>> >>
>> >> Only the last one does, and mddev->ro == 0.
>> >>
>> >> For reference, attached is the used patch and resulting log output.
>> >>
>> >
>> > Thanks.
>> >
>> > So it isn't running md_do_sync at all. Odd.
>> >
>> > Could please add:
>> > - call "WARN_ON(1);" in print_raid5_conf() so we get a stack trace and can
>> > see who is calling it.
>> > - print the value that remove_and_add_spares is going to return.
>>
>> Attached. As you can see, remove_and_add_spare returns 0.
>>
>> --
>> Anssi Hannula
>
>
> Please add:
>
> diff --git a/drivers/md/md.c b/drivers/md/md.c
> index 5c95ccb..fa56ac5 100644
> --- a/drivers/md/md.c
> +++ b/drivers/md/md.c
> @@ -7328,8 +7328,10 @@ static int remove_and_add_spares(mddev_t *mddev)
> }
> }
>
> + printk("degraded=%d\n", mddev->degraded);
> if (mddev->degraded) {
> list_for_each_entry(rdev, &mddev->disks, same_set) {
> + printk("raid_disk=%d flags=%x\n", rdev->raid_disk, rdev->flags);
> if (rdev->raid_disk >= 0 &&
> !test_bit(In_sync, &rdev->flags) &&
> !test_bit(Faulty, &rdev->flags))
>
>
> 'degraded' must be 2 as dmesg contains
>
> [ 45.544806] md/raid:md0: raid level 6 active with 8 out of 10 devices, algorithm 2
>
> and 'degraded' is exactly the difference between '8' and '10' there.
>
> raid disks 3 and 7 must have In_sync and Faulty clear as both of them just
> show "spare rebuilding" in the 'detail' output.
>
> so remove_and_add_spares "must" return 2.
>
> Hopefully the above patch will help me understand which of those is wrong.
The output is:
[ 47.389379] md0: degraded=2
[ 47.389380] md0: raid_disk=0 flags=4
[ 47.389381] md0: raid_disk=-1 flags=0
Full assemble log attached.
--
Anssi Hannula
[ 45.225768] md: md0 stopped.
[ 45.238528] md: bind<sdq1>
[ 45.243922] md: bind<sdo1>
[ 45.244059] md: bind<sdh1>
[ 45.246554] md: bind<sdk1>
[ 45.246987] md: bind<sdp1>
[ 45.247224] md: bind<sdn1>
[ 45.247357] md: bind<sdf1>
[ 45.247596] md: bind<sdm1>
[ 45.247738] md: bind<sdl1>
[ 45.247860] md: bind<sdi1>
[ 45.247986] md: bind<sdg1>
[ 45.248100] md: bind<sdd1>
[ 45.248408] md: bind<sdj1>
[ 45.251243] async_tx: api initialized (async)
[ 45.433646] raid6: int64x1 2754 MB/s
[ 45.603511] raid6: int64x2 3535 MB/s
[ 45.773372] raid6: int64x4 2375 MB/s
[ 45.943221] raid6: int64x8 2489 MB/s
[ 46.113076] raid6: sse2x1 7372 MB/s
[ 46.282914] raid6: sse2x2 8653 MB/s
[ 46.452774] raid6: sse2x4 9642 MB/s
[ 46.452776] raid6: using algorithm sse2x4 (9642 MB/s)
[ 46.460643] xor: automatically using best checksumming function: generic_sse
[ 46.502736] generic_sse: 12817.200 MB/sec
[ 46.502738] xor: using function: generic_sse (12817.200 MB/sec)
[ 46.514000] md: raid6 personality registered for level 6
[ 46.514002] md: raid5 personality registered for level 5
[ 46.514003] md: raid4 personality registered for level 4
[ 46.514225] md/raid:md0: device sdj1 operational as raid disk 0
[ 46.514227] md/raid:md0: device sdl1 operational as raid disk 9
[ 46.514228] md/raid:md0: device sdm1 operational as raid disk 8
[ 46.514230] md/raid:md0: device sdn1 operational as raid disk 6
[ 46.514231] md/raid:md0: device sdp1 operational as raid disk 5
[ 46.514232] md/raid:md0: device sdk1 operational as raid disk 4
[ 46.514233] md/raid:md0: device sdo1 operational as raid disk 2
[ 46.514234] md/raid:md0: device sdq1 operational as raid disk 1
[ 46.514822] md/raid:md0: allocated 10592kB
[ 46.514941] md/raid:md0: raid level 6 active with 8 out of 10 devices, algorithm 2
[ 46.515042] ------------[ cut here ]------------
[ 46.515048] WARNING: at drivers/md/raid5.c:5030 print_raid5_conf+0x37/0x110 [raid456]()
[ 46.515055] Hardware name: System Product Name
[ 46.515057] Modules linked in: raid456 async_pq async_xor xor async_memcpy async_raid6_recov raid6_pq async_tx coretemp cpufreq_ondemand cpufreq_conservative cpufreq_powersave acpi_cpufreq freq_table mperf pcspkr nvram netconsole configfs kvm_intel kvm snd_hda_codec_hdmi tda18271c2dd ir_lirc_codec lirc_dev ir_mce_kbd_decoder ir_sony_decoder ir_jvc_decoder uas ir_rc6_decoder rc_imon_pad ir_rc5_decoder usb_storage joydev imon ir_nec_decoder rc_core snd_hda_codec_realtek snd_hda_intel snd_hda_codec snd_hwdep drxk snd_seq_dummy snd_seq_oss ddbridge dvb_core snd_seq_midi_event snd_seq snd_seq_device cxd2099(C) snd_pcm_oss snd_pcm snd_timer snd_mixer_oss snd evdev e1000e mxm_wmi sr_mod firewire_ohci serio_raw firewire_core i2c_i801 iTCO_wdt i7core_edac i2c_core iTCO_vendor_support soundcore edac_core crc_itu_t snd_page_alloc asus_atk0110 wmi sg xhci_hcd button processor ide_generic ide_core mptsas mptscsih mptbase mpt2sas scsi_transport_sas raid_class ata_generic pata_jmicron cciss pata_amd sata_sil24 sata_sil sata_via dm_snapshot dm_zero dm_mirror dm_region_hash dm_log dm_mod shpchp pci_hotplug ata_piix ahci libahci libata sd_mod scsi_mod crc_t10dif raid1 ext4 jbd2 crc16 uhci_hcd ohci_hcd ehci_hcd usbhid hid usbcore [last unloaded: scsi_wait_scan]
[ 46.515126] Pid: 899, comm: mdadm Tainted: G C 3.1.5 #6
[ 46.515127] Call Trace:
[ 46.515132] [<ffffffff8106487f>] warn_slowpath_common+0x7f/0xc0
[ 46.515134] [<ffffffff810648da>] warn_slowpath_null+0x1a/0x20
[ 46.515136] [<ffffffffa067e607>] print_raid5_conf+0x37/0x110 [raid456]
[ 46.515139] [<ffffffffa06872ba>] run+0x2aa/0x750 [raid456]
[ 46.515142] [<ffffffff811908a1>] ? bioset_create+0x1c1/0x280
[ 46.515144] [<ffffffff8133f42b>] md_run+0x36b/0x8a0
[ 46.515146] [<ffffffff8134064e>] ? add_new_disk+0x2ee/0x550
[ 46.515149] [<ffffffff8104d544>] ? check_preempt_curr+0x84/0xa0
[ 46.515150] [<ffffffff8133f97e>] do_md_run+0x1e/0xd0
[ 46.515152] [<ffffffff81341263>] md_ioctl+0x9b3/0x1230
[ 46.515156] [<ffffffff8121d260>] blkdev_ioctl+0x230/0x740
[ 46.515158] [<ffffffff81191a31>] block_ioctl+0x41/0x50
[ 46.515160] [<ffffffff8116f867>] do_vfs_ioctl+0x97/0x4f0
[ 46.515162] [<ffffffff8116fd51>] sys_ioctl+0x91/0xa0
[ 46.515165] [<ffffffff8143cdd2>] system_call_fastpath+0x16/0x1b
[ 46.515166] ---[ end trace d6c4c3c9ed54b494 ]---
[ 46.515167] RAID conf printout:
[ 46.515168] --- level:6 rd:10 wd:8
[ 46.515169] disk 0, o:1, dev:sdj1
[ 46.515170] disk 1, o:1, dev:sdq1
[ 46.515171] disk 2, o:1, dev:sdo1
[ 46.515173] disk 3, o:1, dev:sdh1
[ 46.515173] disk 4, o:1, dev:sdk1
[ 46.515174] disk 5, o:1, dev:sdp1
[ 46.515175] disk 6, o:1, dev:sdn1
[ 46.515176] disk 7, o:1, dev:sdf1
[ 46.515177] disk 8, o:1, dev:sdm1
[ 46.515178] disk 9, o:1, dev:sdl1
[ 46.515327] created bitmap (15 pages) for device md0
[ 46.930490] md0: bitmap initialized from disk: read 1/1 pages, set 258 of 29809 bits
[ 47.387963] md0: detected capacity change from 0 to 16003169779712
[ 47.388949] md0: in md_check_recovery(), mddev->ro = 0
[ 47.388954] ------------[ cut here ]------------
[ 47.388961] WARNING: at drivers/md/raid5.c:5030 print_raid5_conf+0x37/0x110 [raid456]()
[ 47.388964] Hardware name: System Product Name
[ 47.388966] Modules linked in: raid456 async_pq async_xor xor async_memcpy async_raid6_recov raid6_pq async_tx coretemp cpufreq_ondemand cpufreq_conservative cpufreq_powersave acpi_cpufreq freq_table mperf pcspkr nvram netconsole configfs kvm_intel kvm snd_hda_codec_hdmi tda18271c2dd ir_lirc_codec lirc_dev ir_mce_kbd_decoder ir_sony_decoder ir_jvc_decoder uas ir_rc6_decoder rc_imon_pad ir_rc5_decoder usb_storage joydev imon ir_nec_decoder rc_core snd_hda_codec_realtek snd_hda_intel snd_hda_codec snd_hwdep drxk snd_seq_dummy snd_seq_oss ddbridge dvb_core snd_seq_midi_event snd_seq snd_seq_device cxd2099(C) snd_pcm_oss snd_pcm snd_timer snd_mixer_oss snd evdev e1000e mxm_wmi sr_mod firewire_ohci serio_raw firewire_core i2c_i801 iTCO_wdt i7core_edac i2c_core iTCO_vendor_support soundcore edac_core crc_itu_t snd_page_alloc asus_atk0110 wmi sg xhci_hcd button processor ide_generic ide_core mptsas mptscsih mptbase mpt2sas scsi_transport_sas raid_class ata_generic pata_jmicron cciss pata_amd sata_sil24 sata_sil sata_via dm_snapshot dm_zero dm_mirror dm_region_hash dm_log dm_mod shpchp pci_hotplug ata_piix ahci libahci libata sd_mod scsi_mod crc_t10dif raid1 ext4 jbd2 crc16 uhci_hcd ohci_hcd ehci_hcd usbhid hid usbcore [last unloaded: scsi_wait_scan]
[ 47.389058] Pid: 983, comm: md0_raid6 Tainted: G WC 3.1.5 #6
[ 47.389059] Call Trace:
[ 47.389064] [<ffffffff8106487f>] warn_slowpath_common+0x7f/0xc0
[ 47.389066] [<ffffffff810648da>] warn_slowpath_null+0x1a/0x20
[ 47.389068] [<ffffffffa067e607>] print_raid5_conf+0x37/0x110 [raid456]
[ 47.389070] [<ffffffffa067eff7>] raid5_remove_disk+0x47/0x120 [raid456]
[ 47.389073] [<ffffffff8133dd45>] md_check_recovery+0x3c5/0x6d0
[ 47.389076] [<ffffffffa0684e63>] raid5d+0x43/0x780 [raid456]
[ 47.389079] [<ffffffff810749ea>] ? del_timer_sync+0x3a/0x60
[ 47.389083] [<ffffffff81433097>] ? schedule_timeout+0x177/0x2e0
[ 47.389085] [<ffffffff81073660>] ? cascade+0xa0/0xa0
[ 47.389088] [<ffffffff81338f96>] md_thread+0x116/0x150
[ 47.389091] [<ffffffff81086f40>] ? wake_up_bit+0x40/0x40
[ 47.389093] [<ffffffff81338e80>] ? md_rdev_init+0x130/0x130
[ 47.389094] [<ffffffff810869a6>] kthread+0x96/0xa0
[ 47.389098] [<ffffffff8143ef34>] kernel_thread_helper+0x4/0x10
[ 47.389100] [<ffffffff81086910>] ? kthread_worker_fn+0x1a0/0x1a0
[ 47.389102] [<ffffffff8143ef30>] ? gs_change+0x13/0x13
[ 47.389103] ---[ end trace d6c4c3c9ed54b495 ]---
[ 47.389104] RAID conf printout:
[ 47.389105] --- level:6 rd:10 wd:8
[ 47.389106] disk 0, o:1, dev:sdj1
[ 47.389108] disk 1, o:1, dev:sdq1
[ 47.389109] disk 2, o:1, dev:sdo1
[ 47.389110] disk 3, o:1, dev:sdh1
[ 47.389111] disk 4, o:1, dev:sdk1
[ 47.389113] disk 5, o:1, dev:sdp1
[ 47.389114] disk 6, o:1, dev:sdn1
[ 47.389115] disk 7, o:1, dev:sdf1
[ 47.389116] disk 8, o:1, dev:sdm1
[ 47.389117] disk 9, o:1, dev:sdl1
[ 47.389118] ------------[ cut here ]------------
[ 47.389121] WARNING: at drivers/md/raid5.c:5030 print_raid5_conf+0x37/0x110 [raid456]()
[ 47.389122] Hardware name: System Product Name
[ 47.389123] Modules linked in: raid456 async_pq async_xor xor async_memcpy async_raid6_recov raid6_pq async_tx coretemp cpufreq_ondemand cpufreq_conservative cpufreq_powersave acpi_cpufreq freq_table mperf pcspkr nvram netconsole configfs kvm_intel kvm snd_hda_codec_hdmi tda18271c2dd ir_lirc_codec lirc_dev ir_mce_kbd_decoder ir_sony_decoder ir_jvc_decoder uas ir_rc6_decoder rc_imon_pad ir_rc5_decoder usb_storage joydev imon ir_nec_decoder rc_core snd_hda_codec_realtek snd_hda_intel snd_hda_codec snd_hwdep drxk snd_seq_dummy snd_seq_oss ddbridge dvb_core snd_seq_midi_event snd_seq snd_seq_device cxd2099(C) snd_pcm_oss snd_pcm snd_timer snd_mixer_oss snd evdev e1000e mxm_wmi sr_mod firewire_ohci serio_raw firewire_core i2c_i801 iTCO_wdt i7core_edac i2c_core iTCO_vendor_support soundcore edac_core crc_itu_t snd_page_alloc asus_atk0110 wmi sg xhci_hcd button processor ide_generic ide_core mptsas mptscsih mptbase mpt2sas scsi_transport_sas raid_class ata_generic pata_jmicron cciss pata_amd sata_sil24 sata_sil sata_via dm_snapshot dm_zero dm_mirror dm_region_hash dm_log dm_mod shpchp pci_hotplug ata_piix ahci libahci libata sd_mod scsi_mod crc_t10dif raid1 ext4 jbd2 crc16 uhci_hcd ohci_hcd ehci_hcd usbhid hid usbcore [last unloaded: scsi_wait_scan]
[ 47.389162] Pid: 983, comm: md0_raid6 Tainted: G WC 3.1.5 #6
[ 47.389163] Call Trace:
[ 47.389164] [<ffffffff8106487f>] warn_slowpath_common+0x7f/0xc0
[ 47.389166] [<ffffffff810648da>] warn_slowpath_null+0x1a/0x20
[ 47.389168] [<ffffffffa067e607>] print_raid5_conf+0x37/0x110 [raid456]
[ 47.389171] [<ffffffffa067f00c>] raid5_remove_disk+0x5c/0x120 [raid456]
[ 47.389173] [<ffffffff8133dd45>] md_check_recovery+0x3c5/0x6d0
[ 47.389175] [<ffffffffa0684e63>] raid5d+0x43/0x780 [raid456]
[ 47.389177] [<ffffffff810749ea>] ? del_timer_sync+0x3a/0x60
[ 47.389179] [<ffffffff81433097>] ? schedule_timeout+0x177/0x2e0
[ 47.389181] [<ffffffff81073660>] ? cascade+0xa0/0xa0
[ 47.389183] [<ffffffff81338f96>] md_thread+0x116/0x150
[ 47.389184] [<ffffffff81086f40>] ? wake_up_bit+0x40/0x40
[ 47.389186] [<ffffffff81338e80>] ? md_rdev_init+0x130/0x130
[ 47.389188] [<ffffffff810869a6>] kthread+0x96/0xa0
[ 47.389190] [<ffffffff8143ef34>] kernel_thread_helper+0x4/0x10
[ 47.389191] [<ffffffff81086910>] ? kthread_worker_fn+0x1a0/0x1a0
[ 47.389193] [<ffffffff8143ef30>] ? gs_change+0x13/0x13
[ 47.389194] ---[ end trace d6c4c3c9ed54b496 ]---
[ 47.389195] RAID conf printout:
[ 47.389196] --- level:6 rd:10 wd:8
[ 47.389197] disk 0, o:1, dev:sdj1
[ 47.389198] disk 1, o:1, dev:sdq1
[ 47.389199] disk 2, o:1, dev:sdo1
[ 47.389200] disk 3, o:1, dev:sdh1
[ 47.389201] disk 4, o:1, dev:sdk1
[ 47.389202] disk 5, o:1, dev:sdp1
[ 47.389203] disk 6, o:1, dev:sdn1
[ 47.389204] disk 7, o:1, dev:sdf1
[ 47.389205] disk 8, o:1, dev:sdm1
[ 47.389206] disk 9, o:1, dev:sdl1
[ 47.389206] ------------[ cut here ]------------
[ 47.389209] WARNING: at drivers/md/raid5.c:5030 print_raid5_conf+0x37/0x110 [raid456]()
[ 47.389210] Hardware name: System Product Name
[ 47.389210] Modules linked in: raid456 async_pq async_xor xor async_memcpy async_raid6_recov raid6_pq async_tx coretemp cpufreq_ondemand cpufreq_conservative cpufreq_powersave acpi_cpufreq freq_table mperf pcspkr nvram netconsole configfs kvm_intel kvm snd_hda_codec_hdmi tda18271c2dd ir_lirc_codec lirc_dev ir_mce_kbd_decoder ir_sony_decoder ir_jvc_decoder uas ir_rc6_decoder rc_imon_pad ir_rc5_decoder usb_storage joydev imon ir_nec_decoder rc_core snd_hda_codec_realtek snd_hda_intel snd_hda_codec snd_hwdep drxk snd_seq_dummy snd_seq_oss ddbridge dvb_core snd_seq_midi_event snd_seq snd_seq_device cxd2099(C) snd_pcm_oss snd_pcm snd_timer snd_mixer_oss snd evdev e1000e mxm_wmi sr_mod firewire_ohci serio_raw firewire_core i2c_i801 iTCO_wdt i7core_edac i2c_core iTCO_vendor_support soundcore edac_core crc_itu_t snd_page_alloc asus_atk0110 wmi sg xhci_hcd button processor ide_generic ide_core mptsas mptscsih mptbase mpt2sas scsi_transport_sas raid_class ata_generic pata_jmicron cciss pata_amd sata_sil24 sata_sil sata_via dm_snapshot dm_zero dm_mirror dm_region_hash dm_log dm_mod shpchp pci_hotplug ata_piix ahci libahci libata sd_mod scsi_mod crc_t10dif raid1 ext4 jbd2 crc16 uhci_hcd ohci_hcd ehci_hcd usbhid hid usbcore [last unloaded: scsi_wait_scan]
[ 47.389248] Pid: 983, comm: md0_raid6 Tainted: G WC 3.1.5 #6
[ 47.389249] Call Trace:
[ 47.389251] [<ffffffff8106487f>] warn_slowpath_common+0x7f/0xc0
[ 47.389253] [<ffffffff810648da>] warn_slowpath_null+0x1a/0x20
[ 47.389255] [<ffffffffa067e607>] print_raid5_conf+0x37/0x110 [raid456]
[ 47.389257] [<ffffffffa067eff7>] raid5_remove_disk+0x47/0x120 [raid456]
[ 47.389259] [<ffffffff8133dd45>] md_check_recovery+0x3c5/0x6d0
[ 47.389261] [<ffffffffa0684e63>] raid5d+0x43/0x780 [raid456]
[ 47.389263] [<ffffffff810749ea>] ? del_timer_sync+0x3a/0x60
[ 47.389265] [<ffffffff81433097>] ? schedule_timeout+0x177/0x2e0
[ 47.389267] [<ffffffff81073660>] ? cascade+0xa0/0xa0
[ 47.389269] [<ffffffff81338f96>] md_thread+0x116/0x150
[ 47.389271] [<ffffffff81086f40>] ? wake_up_bit+0x40/0x40
[ 47.389273] [<ffffffff81338e80>] ? md_rdev_init+0x130/0x130
[ 47.389274] [<ffffffff810869a6>] kthread+0x96/0xa0
[ 47.389276] [<ffffffff8143ef34>] kernel_thread_helper+0x4/0x10
[ 47.389278] [<ffffffff81086910>] ? kthread_worker_fn+0x1a0/0x1a0
[ 47.389280] [<ffffffff8143ef30>] ? gs_change+0x13/0x13
[ 47.389281] ---[ end trace d6c4c3c9ed54b497 ]---
[ 47.389282] RAID conf printout:
[ 47.389283] --- level:6 rd:10 wd:8
[ 47.389284] disk 0, o:1, dev:sdj1
[ 47.389285] disk 1, o:1, dev:sdq1
[ 47.389285] disk 2, o:1, dev:sdo1
[ 47.389286] disk 3, o:1, dev:sdh1
[ 47.389287] disk 4, o:1, dev:sdk1
[ 47.389288] disk 5, o:1, dev:sdp1
[ 47.389289] disk 6, o:1, dev:sdn1
[ 47.389290] disk 7, o:1, dev:sdf1
[ 47.389291] disk 8, o:1, dev:sdm1
[ 47.389292] disk 9, o:1, dev:sdl1
[ 47.389293] ------------[ cut here ]------------
[ 47.389295] WARNING: at drivers/md/raid5.c:5030 print_raid5_conf+0x37/0x110 [raid456]()
[ 47.389296] Hardware name: System Product Name
[ 47.389297] Modules linked in: raid456 async_pq async_xor xor async_memcpy async_raid6_recov raid6_pq async_tx coretemp cpufreq_ondemand cpufreq_conservative cpufreq_powersave acpi_cpufreq freq_table mperf pcspkr nvram netconsole configfs kvm_intel kvm snd_hda_codec_hdmi tda18271c2dd ir_lirc_codec lirc_dev ir_mce_kbd_decoder ir_sony_decoder ir_jvc_decoder uas ir_rc6_decoder rc_imon_pad ir_rc5_decoder usb_storage joydev imon ir_nec_decoder rc_core snd_hda_codec_realtek snd_hda_intel snd_hda_codec snd_hwdep drxk snd_seq_dummy snd_seq_oss ddbridge dvb_core snd_seq_midi_event snd_seq snd_seq_device cxd2099(C) snd_pcm_oss snd_pcm snd_timer snd_mixer_oss snd evdev e1000e mxm_wmi sr_mod firewire_ohci serio_raw firewire_core i2c_i801 iTCO_wdt i7core_edac i2c_core iTCO_vendor_support soundcore edac_core crc_itu_t snd_page_alloc asus_atk0110 wmi sg xhci_hcd button processor ide_generic ide_core mptsas mptscsih mptbase mpt2sas scsi_transport_sas raid_class ata_generic pata_jmicron cciss pata_amd sata_sil24 sata_sil sata_via dm_snapshot dm_zero dm_mirror dm_region_hash dm_log dm_mod shpchp pci_hotplug ata_piix ahci libahci libata sd_mod scsi_mod crc_t10dif raid1 ext4 jbd2 crc16 uhci_hcd ohci_hcd ehci_hcd usbhid hid usbcore [last unloaded: scsi_wait_scan]
[ 47.389334] Pid: 983, comm: md0_raid6 Tainted: G WC 3.1.5 #6
[ 47.389335] Call Trace:
[ 47.389337] [<ffffffff8106487f>] warn_slowpath_common+0x7f/0xc0
[ 47.389339] [<ffffffff810648da>] warn_slowpath_null+0x1a/0x20
[ 47.389341] [<ffffffffa067e607>] print_raid5_conf+0x37/0x110 [raid456]
[ 47.389343] [<ffffffffa067f00c>] raid5_remove_disk+0x5c/0x120 [raid456]
[ 47.389345] [<ffffffff8133dd45>] md_check_recovery+0x3c5/0x6d0
[ 47.389347] [<ffffffffa0684e63>] raid5d+0x43/0x780 [raid456]
[ 47.389349] [<ffffffff810749ea>] ? del_timer_sync+0x3a/0x60
[ 47.389351] [<ffffffff81433097>] ? schedule_timeout+0x177/0x2e0
[ 47.389353] [<ffffffff81073660>] ? cascade+0xa0/0xa0
[ 47.389355] [<ffffffff81338f96>] md_thread+0x116/0x150
[ 47.389357] [<ffffffff81086f40>] ? wake_up_bit+0x40/0x40
[ 47.389359] [<ffffffff81338e80>] ? md_rdev_init+0x130/0x130
[ 47.389360] [<ffffffff810869a6>] kthread+0x96/0xa0
[ 47.389362] [<ffffffff8143ef34>] kernel_thread_helper+0x4/0x10
[ 47.389364] [<ffffffff81086910>] ? kthread_worker_fn+0x1a0/0x1a0
[ 47.389366] [<ffffffff8143ef30>] ? gs_change+0x13/0x13
[ 47.389367] ---[ end trace d6c4c3c9ed54b498 ]---
[ 47.389368] RAID conf printout:
[ 47.389368] --- level:6 rd:10 wd:8
[ 47.389369] disk 0, o:1, dev:sdj1
[ 47.389370] disk 1, o:1, dev:sdq1
[ 47.389371] disk 2, o:1, dev:sdo1
[ 47.389372] disk 3, o:1, dev:sdh1
[ 47.389373] disk 4, o:1, dev:sdk1
[ 47.389374] disk 5, o:1, dev:sdp1
[ 47.389375] disk 6, o:1, dev:sdn1
[ 47.389376] disk 7, o:1, dev:sdf1
[ 47.389377] disk 8, o:1, dev:sdm1
[ 47.389378] disk 9, o:1, dev:sdl1
[ 47.389379] md0: degraded=2
[ 47.389380] md0: raid_disk=0 flags=4
[ 47.389381] md0: raid_disk=-1 flags=0
[ 47.389382] ------------[ cut here ]------------
[ 47.389384] WARNING: at drivers/md/raid5.c:5030 print_raid5_conf+0x37/0x110 [raid456]()
[ 47.389385] Hardware name: System Product Name
[ 47.389386] Modules linked in: raid456 async_pq async_xor xor async_memcpy async_raid6_recov raid6_pq async_tx coretemp cpufreq_ondemand cpufreq_conservative cpufreq_powersave acpi_cpufreq freq_table mperf pcspkr nvram netconsole configfs kvm_intel kvm snd_hda_codec_hdmi tda18271c2dd ir_lirc_codec lirc_dev ir_mce_kbd_decoder ir_sony_decoder ir_jvc_decoder uas ir_rc6_decoder rc_imon_pad ir_rc5_decoder usb_storage joydev imon ir_nec_decoder rc_core snd_hda_codec_realtek snd_hda_intel snd_hda_codec snd_hwdep drxk snd_seq_dummy snd_seq_oss ddbridge dvb_core snd_seq_midi_event snd_seq snd_seq_device cxd2099(C) snd_pcm_oss snd_pcm snd_timer snd_mixer_oss snd evdev e1000e mxm_wmi sr_mod firewire_ohci serio_raw firewire_core i2c_i801 iTCO_wdt i7core_edac i2c_core iTCO_vendor_support soundcore edac_core crc_itu_t snd_page_alloc asus_atk0110 wmi sg xhci_hcd button processor ide_generic ide_core mptsas mptscsih mptbase mpt2sas scsi_transport_sas raid_class ata_generic pata_jmicron cciss pata_amd sata_sil24 sata_sil sata_via dm_snapshot dm_zero dm_mirror dm_region_hash dm_log dm_mod shpchp pci_hotplug ata_piix ahci libahci libata sd_mod scsi_mod crc_t10dif raid1 ext4 jbd2 crc16 uhci_hcd ohci_hcd ehci_hcd usbhid hid usbcore [last unloaded: scsi_wait_scan]
[ 47.389423] Pid: 983, comm: md0_raid6 Tainted: G WC 3.1.5 #6
[ 47.389424] Call Trace:
[ 47.389426] [<ffffffff8106487f>] warn_slowpath_common+0x7f/0xc0
[ 47.389428] [<ffffffff810648da>] warn_slowpath_null+0x1a/0x20
[ 47.389430] [<ffffffffa067e607>] print_raid5_conf+0x37/0x110 [raid456]
[ 47.389432] [<ffffffffa067e7b4>] raid5_add_disk+0xd4/0x120 [raid456]
[ 47.389434] [<ffffffff8133ded9>] md_check_recovery+0x559/0x6d0
[ 47.389436] [<ffffffffa0684e63>] raid5d+0x43/0x780 [raid456]
[ 47.389438] [<ffffffff810749ea>] ? del_timer_sync+0x3a/0x60
[ 47.389440] [<ffffffff81433097>] ? schedule_timeout+0x177/0x2e0
[ 47.389442] [<ffffffff81073660>] ? cascade+0xa0/0xa0
[ 47.389444] [<ffffffff81338f96>] md_thread+0x116/0x150
[ 47.389446] [<ffffffff81086f40>] ? wake_up_bit+0x40/0x40
[ 47.389447] [<ffffffff81338e80>] ? md_rdev_init+0x130/0x130
[ 47.389449] [<ffffffff810869a6>] kthread+0x96/0xa0
[ 47.389451] [<ffffffff8143ef34>] kernel_thread_helper+0x4/0x10
[ 47.389453] [<ffffffff81086910>] ? kthread_worker_fn+0x1a0/0x1a0
[ 47.389455] [<ffffffff8143ef30>] ? gs_change+0x13/0x13
[ 47.389456] ---[ end trace d6c4c3c9ed54b499 ]---
[ 47.389456] RAID conf printout:
[ 47.389457] --- level:6 rd:10 wd:8
[ 47.389458] disk 0, o:1, dev:sdj1
[ 47.389459] disk 1, o:1, dev:sdq1
[ 47.389460] disk 2, o:1, dev:sdo1
[ 47.389461] disk 3, o:1, dev:sdh1
[ 47.389462] disk 4, o:1, dev:sdk1
[ 47.389463] disk 5, o:1, dev:sdp1
[ 47.389464] disk 6, o:1, dev:sdn1
[ 47.389465] disk 7, o:1, dev:sdf1
[ 47.389466] disk 8, o:1, dev:sdm1
[ 47.389467] disk 9, o:1, dev:sdl1
[ 47.389468] md0: remove_and_add_spares will return 0
[ 47.397561] md0: unknown partition table
