So, now after manually assembling, mounting and starting nfs-kernel
server on the array:
The following is a verbatim command sequence.
>From my thoughts, firstly, note that this time I did not have
'fail_last_dev' set. You see the effect in LVM, Error reading device
...... This is not exactly the same error I remember as it does not
involve udev, but it is different than when the entire array has
failed at the mdraid level.
Further note that the device mapper still sees an 'open' on the
device, which is still visible. I unmount...successfully no problem,
dev mapper shows the 'open = 0'
Then we see no further errors from LVM. That seems kinda odd, as they
are not in that order in the stack.
If I then manually remove the device mapper entry, I can stop the
array. SUCCESS.
Thoughts?
Aidan
pvdisplay
Error reading device /dev/sdc at 0 length 512.
Error reading device /dev/sdc at 0 length 4096.
Error reading device /dev/sdd at 0 length 512.
Error reading device /dev/sdd at 0 length 4096.
Error reading device /dev/md90 at 0 length 512.
Error reading device /dev/md90 at 0 length 4096.
--- Physical volume ---
PV Name /dev/md1
VG Name storage.mx.vg1
PV Size 111.73 GiB / not usable 3.00 MiB
Allocatable yes (but full)
PE Size 4.00 MiB
Total PE 28603
Free PE 0
Allocated PE 28603
PV UUID 4yDnuz-PEHg-uZqd-djWS-DNnp-Qzuf-fYvGZJ
--- Physical volume ---
PV Name /dev/md0
VG Name ubuntu_server-vg
PV Size <37.22 GiB / not usable 0
Allocatable yes (but full)
PE Size 4.00 MiB
Total PE 9528
Free PE 0
Allocated PE 9528
PV UUID G0bNbO-DOz4-I2nN-rEQq-X00m-PG3a-fPAP3I
dmsetup info -c
Name Maj Min Stat Open Targ Event UUID
storage.mx.vg2-shared_sun_NAS.lv1 253 2 L--w 1 1 0
LVM-Ud9pj6QE4hK1K3xiAFMVCnno3SrXaRyTXJLtTGDOPjBUppJgzr4t0jJowixEOtx7
storage.mx.vg1-shared_sun_users.lv1 253 1 L--w 1 1 0
LVM-ypcHlbNXu36FLRgU0EcUiXBSIvcOlHEP3MHkBKsBeHf6Q68TIuGA9hd5UfCpvOeo
ubuntu_server--vg-ubuntu_server--lv 253 0 L--w 1 1 0
LVM-eGBUJxP1vlW3MfNNeC2r5JfQUiKKWZ73t3U3Jji3lggXe8LPrUf0xRE0YyPzSorO
cat /proc/mounts | grep mapper
/dev/mapper/ubuntu_server--vg-ubuntu_server--lv / ext4
rw,relatime,errors=remount-ro 0 0
/dev/mapper/storage.mx.vg1-shared_sun_users.lv1 /mnt/home ext4 rw,relatime 0 0
/dev/mapper/storage.mx.vg2-shared_sun_NAS.lv1 /mnt/shared_sun_NAS ext4
rw,relatime 0 0
umount /mnt/shared_sun_NAS
cat /proc/mounts | grep mapper
/dev/mapper/ubuntu_server--vg-ubuntu_server--lv / ext4
rw,relatime,errors=remount-ro 0 0
/dev/mapper/storage.mx.vg1-shared_sun_users.lv1 /mnt/home ext4 rw,relatime 0 0
dmsetup info -c
Name Maj Min Stat Open Targ Event UUID
storage.mx.vg2-shared_sun_NAS.lv1 253 2 L--w 0 1 0
LVM-Ud9pj6QE4hK1K3xiAFMVCnno3SrXaRyTXJLtTGDOPjBUppJgzr4t0jJowixEOtx7
storage.mx.vg1-shared_sun_users.lv1 253 1 L--w 1 1 0
LVM-ypcHlbNXu36FLRgU0EcUiXBSIvcOlHEP3MHkBKsBeHf6Q68TIuGA9hd5UfCpvOeo
ubuntu_server--vg-ubuntu_server--lv 253 0 L--w 1 1 0
LVM-eGBUJxP1vlW3MfNNeC2r5JfQUiKKWZ73t3U3Jji3lggXe8LPrUf0xRE0YyPzSorO
pvdisplay
--- Physical volume ---
PV Name /dev/md1
VG Name storage.mx.vg1
PV Size 111.73 GiB / not usable 3.00 MiB
Allocatable yes (but full)
PE Size 4.00 MiB
Total PE 28603
Free PE 0
Allocated PE 28603
PV UUID 4yDnuz-PEHg-uZqd-djWS-DNnp-Qzuf-fYvGZJ
--- Physical volume ---
PV Name /dev/md0
VG Name ubuntu_server-vg
PV Size <37.22 GiB / not usable 0
Allocatable yes (but full)
PE Size 4.00 MiB
Total PE 9528
Free PE 0
Allocated PE 9528
PV UUID G0bNbO-DOz4-I2nN-rEQq-X00m-PG3a-fPAP3I
mdadm -D /dev/md90
/dev/md90:
Version : 1.2
Creation Time : Sat Nov 3 03:09:16 2018
Raid Level : raid1
Array Size : 488253440 (465.63 GiB 499.97 GB)
Used Dev Size : 488253440 (465.63 GiB 499.97 GB)
Raid Devices : 2
Total Devices : 2
Persistence : Superblock is persistent
Intent Bitmap : Internal
Update Time : Thu Jan 13 17:59:38 2022
State : clean, degraded
Active Devices : 1
Working Devices : 1
Failed Devices : 1
Spare Devices : 0
Consistency Policy : bitmap
Number Major Minor RaidDevice State
- 0 0 0 removed
2 8 49 1 active sync /dev/sdd1
0 8 33 - faulty /dev/sdc1
root@mx:~# mdadm --stop /dev/md90
mdadm: Cannot get exclusive access to /dev/md90:Perhaps a running
process, mounted filesystem or active volume group?
dmsetup remove /dev/mapper/storage.mx.vg2-shared_sun_NAS.lv1
root@mx:~# dmsetup info -c
Name Maj Min Stat Open Targ Event UUID
storage.mx.vg1-shared_sun_users.lv1 253 1 L--w 1 1 0
LVM-ypcHlbNXu36FLRgU0EcUiXBSIvcOlHEP3MHkBKsBeHf6Q68TIuGA9hd5UfCpvOeo
ubuntu_server--vg-ubuntu_server--lv 253 0 L--w 1 1 0
LVM-eGBUJxP1vlW3MfNNeC2r5JfQUiKKWZ73t3U3Jji3lggXe8LPrUf0xRE0YyPzSorO
mdadm --stop /dev/md90
mdadm: stopped /dev/md90
On Thu, 13 Jan 2022 at 17:35, Aidan Walton <aidan.walton@xxxxxxxxx> wrote:
>
> Progress of sorts:
> I have tried to get the results as requested. However I was
> experimenting with the patch that you referenced initially and before
> I did this test, I forgot that I had actually run:
> echo 1 > /sys/block/md90/md/fail_last_dev
>
> So the outcome was interesting as the result proves this feature does
> work as expected and the md90 array showed both devices flagged as
> down. Better!
> /dev/md90:
> Version : 1.2
> Creation Time : Sat Nov 3 03:09:16 2018
> Raid Level : raid1
> Array Size : 488253440 (465.63 GiB 499.97 GB)
> Used Dev Size : 488253440 (465.63 GiB 499.97 GB)
> Raid Devices : 2
> Total Devices : 2
> Persistence : Superblock is persistent
>
> Intent Bitmap : Internal
>
> Update Time : Thu Jan 13 17:08:38 2022
> State : clean, FAILED
> Active Devices : 0
> Failed Devices : 2
> Spare Devices : 0
>
> Consistency Policy : bitmap
>
> Number Major Minor RaidDevice State
> - 0 0 0 removed
> - 0 0 1 removed
>
> 0 8 33 - faulty /dev/sdc1
> 2 8 49 - faulty /dev/sdd1
>
> The process that led to this is below:
> from journalctl:
> Jan 13 17:07:05 mx kernel: ata7.00: exception Emask 0x32 SAct 0x0 SErr
> 0x0 action 0xe frozen
> Jan 13 17:07:05 mx kernel: ata7.00: irq_stat 0xffffffff, unknown FIS
> 00000000 00000000 00000000 00000000, host bus
> Jan 13 17:07:05 mx kernel: ata7.00: failed command: READ DMA
> Jan 13 17:07:05 mx kernel: ata7.00: cmd
> c8/00:00:18:2d:ec/00:00:00:00:00/e0 tag 22 dma 131072 in
> Jan 13 17:07:05 mx kernel: ata7.00: status: { DRDY }
> Jan 13 17:07:05 mx kernel: ata7: hard resetting link
> Jan 13 17:07:15 mx kernel: ata7: softreset failed (1st FIS failed)
> Jan 13 17:07:15 mx kernel: ata7: hard resetting link
> Jan 13 17:07:25 mx kernel: ata7: softreset failed (1st FIS failed)
> Jan 13 17:07:25 mx kernel: ata7: hard resetting link
> Jan 13 17:07:36 mx kernel: ata8.00: exception Emask 0x40 SAct 0x0 SErr
> 0x800 action 0x6 frozen
> Jan 13 17:07:36 mx kernel: ata8: SError: { HostInt }
> Jan 13 17:07:36 mx kernel: ata8.00: failed command: READ DMA
> Jan 13 17:07:36 mx kernel: ata8.00: cmd
> c8/00:00:78:53:d9/00:00:00:00:00/e0 tag 10 dma 131072 in
> Jan 13 17:07:36 mx kernel: ata8.00: status: { DRDY }
> Jan 13 17:07:36 mx kernel: ata8: hard resetting link
> Jan 13 17:07:46 mx kernel: ata8: softreset failed (1st FIS failed)
> Jan 13 17:07:46 mx kernel: ata8: hard resetting link
> Jan 13 17:07:56 mx kernel: ata8: softreset failed (1st FIS failed)
> Jan 13 17:07:56 mx kernel: ata8: hard resetting link
> Jan 13 17:08:00 mx kernel: ata7: softreset failed (1st FIS failed)
> Jan 13 17:08:00 mx kernel: ata7: hard resetting link
> Jan 13 17:08:05 mx kernel: ata7: softreset failed (1st FIS failed)
> Jan 13 17:08:05 mx kernel: ata7: reset failed, giving up
> Jan 13 17:08:05 mx kernel: ata7.00: disabled
> Jan 13 17:08:05 mx kernel: ata7: EH complete
> Jan 13 17:08:31 mx kernel: ata8: softreset failed (1st FIS failed)
> Jan 13 17:08:31 mx kernel: ata8: hard resetting link
> Jan 13 17:08:36 mx kernel: ata8: softreset failed (1st FIS failed)
> Jan 13 17:08:36 mx kernel: ata8: reset failed, giving up
> Jan 13 17:08:36 mx kernel: ata8.00: disabled
> Jan 13 17:08:36 mx kernel: ata8: EH complete
>
>
> from udevadm monitor, NOTE, I noticed that udev monitor did not seem
> to spit anything out at the point in time when the first SATA device
> ata7.00: was disabled. The messages below only appeared 30secs later
> when ata8.00: went down: Sorry no timestamping on udevadm monitor, but
> read below from exactly when this occurred: Jan 13 17:08:36 mx kernel:
> ata8.00: disabled
>
> KERNEL[226088.463136] add
> /kernel/slab/kmalloc-192/cgroup/kmalloc-192(549:mdmonitor.service)
> (cgroup)
> KERNEL[226088.463335] add
> /kernel/slab/kmalloc-1k/cgroup/kmalloc-1k(549:mdmonitor.service)
> (cgroup)
> KERNEL[226088.464229] add
> /kernel/slab/task_struct/cgroup/task_struct(549:mdmonitor.service)
> (cgroup)
> KERNEL[226088.464374] add
> /kernel/slab/:A-0000080/cgroup/task_delay_info(549:mdmonitor.service)
> (cgroup)
> KERNEL[226088.464467] add
> /kernel/slab/:A-0000704/cgroup/files_cache(549:mdmonitor.service)
> (cgroup)
> KERNEL[226088.464718] add
> /kernel/slab/sighand_cache/cgroup/sighand_cache(549:mdmonitor.service)
> (cgroup)
> KERNEL[226088.465312] add
> /kernel/slab/:A-0001152/cgroup/signal_cache(549:mdmonitor.service)
> (cgroup)
> UDEV [226088.472483] add
> /kernel/slab/kmalloc-192/cgroup/kmalloc-192(549:mdmonitor.service)
> (cgroup)
> UDEV [226088.473360] add
> /kernel/slab/kmalloc-1k/cgroup/kmalloc-1k(549:mdmonitor.service)
> (cgroup)
> UDEV [226088.480328] add
> /kernel/slab/task_struct/cgroup/task_struct(549:mdmonitor.service)
> (cgroup)
> UDEV [226088.486830] add
> /kernel/slab/:A-0000080/cgroup/task_delay_info(549:mdmonitor.service)
> (cgroup)
> UDEV [226088.487077] add
> /kernel/slab/sighand_cache/cgroup/sighand_cache(549:mdmonitor.service)
> (cgroup)
> UDEV [226088.488851] add
> /kernel/slab/:A-0000704/cgroup/files_cache(549:mdmonitor.service)
> (cgroup)
> UDEV [226088.493066] add
> /kernel/slab/:A-0001152/cgroup/signal_cache(549:mdmonitor.service)
> (cgroup)
> KERNEL[226089.014423] add
> /kernel/slab/:a-0000104/cgroup/buffer_head(549:mdmonitor.service)
> (cgroup)
> UDEV [226089.017837] add
> /kernel/slab/:a-0000104/cgroup/buffer_head(549:mdmonitor.service)
> (cgroup)
> KERNEL[226089.188538] add
> /kernel/slab/radix_tree_node/cgroup/radix_tree_node(549:mdmonitor.service)
> (cgroup)
> KERNEL[226089.189366] add
> /kernel/slab/ext4_inode_cache/cgroup/ext4_inode_cache(549:mdmonitor.service)
> (cgroup)
> UDEV [226089.193043] add
> /kernel/slab/radix_tree_node/cgroup/radix_tree_node(549:mdmonitor.service)
> (cgroup)
> UDEV [226089.194414] add
> /kernel/slab/ext4_inode_cache/cgroup/ext4_inode_cache(549:mdmonitor.service)
> (cgroup)
> KERNEL[226090.421762] add
> /kernel/slab/kmalloc-8/cgroup/kmalloc-8(549:mdmonitor.service)
> (cgroup)
> UDEV [226090.425465] add
> /kernel/slab/kmalloc-8/cgroup/kmalloc-8(549:mdmonitor.service)
> (cgroup)
> KERNEL[226090.442542] add
> /kernel/slab/proc_inode_cache/cgroup/proc_inode_cache(549:mdmonitor.service)
> (cgroup)
> UDEV [226090.445999] add
> /kernel/slab/proc_inode_cache/cgroup/proc_inode_cache(549:mdmonitor.service)
> (cgroup)
> KERNEL[226090.458800] add
> /kernel/slab/:A-0000072/cgroup/eventpoll_pwq(549:mdmonitor.service)
> (cgroup)
> KERNEL[226090.460227] add
> /kernel/slab/kmalloc-16/cgroup/kmalloc-16(549:mdmonitor.service)
> (cgroup)
> UDEV [226090.463193] add
> /kernel/slab/:A-0000072/cgroup/eventpoll_pwq(549:mdmonitor.service)
> (cgroup)
> UDEV [226090.467271] add
> /kernel/slab/kmalloc-16/cgroup/kmalloc-16(549:mdmonitor.service)
> (cgroup)
> KERNEL[226090.880178] add
> /kernel/slab/kmalloc-rcl-64/cgroup/kmalloc-rcl-64(549:mdmonitor.service)
> (cgroup)
> UDEV [226090.883794] add
> /kernel/slab/kmalloc-rcl-64/cgroup/kmalloc-rcl-64(549:mdmonitor.service)
> (cgroup)
>
>
>
> dmsetup info -c
> Name Maj Min Stat Open Targ Event UUID
> storage.mx.vg2-shared_sun_NAS.lv1 253 0 L--w 1 1 0
> LVM-Ud9pj6QE4hK1K3xiAFMVCnno3SrXaRyTXJLtTGDOPjBUppJgzr4t0jJowixEOtx7
> storage.mx.vg1-shared_sun_users.lv1 253 2 L--w 1 1 0
> LVM-ypcHlbNXu36FLRgU0EcUiXBSIvcOlHEP3MHkBKsBeHf6Q68TIuGA9hd5UfCpvOeo
> ubuntu_server--vg-ubuntu_server--lv 253 1 L--w 1 1 0
> LVM-eGBUJxP1vlW3MfNNeC2r5JfQUiKKWZ73t3U3Jji3lggXe8LPrUf0xRE0YyPzSorO
>
> NOTE the 'open' status on the NAS.lv1 device. In fact the device is not mounted:
>
> cat /proc/mounts | grep mapper
> /dev/mapper/ubuntu_server--vg-ubuntu_server--lv / ext4
> rw,relatime,errors=remount-ro 0 0
> /dev/mapper/storage.mx.vg1-shared_sun_users.lv1 /mnt/home ext4 rw,relatime 0 0
>
> pvdisplay
> --- Physical volume ---
> PV Name /dev/md1
> VG Name storage.mx.vg1
> PV Size 111.73 GiB / not usable 3.00 MiB
> Allocatable yes (but full)
> PE Size 4.00 MiB
> Total PE 28603
> Free PE 0
> Allocated PE 28603
> PV UUID 4yDnuz-PEHg-uZqd-djWS-DNnp-Qzuf-fYvGZJ
>
> --- Physical volume ---
> PV Name /dev/md0
> VG Name ubuntu_server-vg
> PV Size <37.22 GiB / not usable 0
> Allocatable yes (but full)
> PE Size 4.00 MiB
> Total PE 9528
> Free PE 0
> Allocated PE 9528
> PV UUID G0bNbO-DOz4-I2nN-rEQq-X00m-PG3a-fPAP3I
>
> So in this case LVM seems to recognise that the md90 device is gone.
>
> Before I changed the 'fail_last_dev' option, When I ran these LVM
> commands, I was experiencing a short delay and then a report in place
> of the failed device saying that LVM had given up waiting for a udev
> entry to become available after 10000mS. Sorry I didn't catch this,
> for the log, but it is from memory.
>
> But now this delay is not happening and LVM seems to have a correct
> and consistent view of the failed mount point. Clearly mdraid has sent
> the failure up the stack.
>
>
> However mdraid will still NOT stop the md90 device:
>
> mdadm --stop /dev/md90
> mdadm: Cannot get exclusive access to /dev/md90:Perhaps a running
> process, mounted filesystem or active volume group?
>
> ATB
> Aidan
>
> On Thu, 13 Jan 2022 at 15:46, Mariusz Tkaczyk
> <mariusz.tkaczyk@xxxxxxxxxxxxxxx> wrote:
> >
> > Hi Aidan,
> >
> > On Wed, 12 Jan 2022 02:29:47 +0100
> > Aidan Walton <aidan.walton@xxxxxxxxx> wrote:
> >
> > > Hi Mariusz,
> > > In my case, the fact that mdraid does not show a 'total failure' is
> > > not the root of the problem. However in my opinion I would say that
> > > not having mdraid more accurately reflect the state of the underlying
> > > hardware can be mis-leading. Initially when I looked at this issue, I
> > > was convinced that only one disk had failed and I was scratching my
> > > head about firstly why I still could not R/W the array while it
> > > appeared to have an active member. Secondly, when I rebooted I noticed
> > > that the array became instantly synchronised with both members active.
> >
> > We have raid1 so first fail should be recorded in metadata. From your
> > description, I understand that nothing like this happened. For me, it
> > seems that the controller lost both drives in the same moment and as a
> > result nothing was saved. After reboot raid is assembled without
> > rebuild because metadata on both members is valid.
> >
> > > This was not what I expected, as normally an array that has had a
> > > single failed disk would require a ra-add and resync. Then when the
> > > problem re-occured I noticed that it was not the same disk that was
> > > flagged faulty, next reboot; the faulty disk flipped back the other
> > > way... and so on. This was what prompted me to look closer at the
> > > kernel. Here I found my answer at the SATA controller. Therefore
> > > although mdraids design approach did not cause me any data loss, it
> > > did have me looking in the wrong direction for the fault, assuming a
> > > disk problem.
> > >
> > > I have still not been able to successfully --stop the array. I think
> > > the issue sits in the LVM domain. Although I can not be 100% sure.
> > > What I have achieved is some level of understanding that some process
> > > that starts a boot time is in some unknown manner holding a lock on
> > > the mdarid - devmapper - LVM combination. I have unmounted the file
> > > system, but LVM refuses to let go of the logical volume. Therefore so
> > > does dev-mapper and of course mdraid. I have systematically stopped or
> > > killed almost every single running process on the system, taking it
> > > back to a skeleton, with not much more than init running, it still
> > > refuses to let go
> > >
> > > However, when I prevent auto-mounting of the raid array at boot, and
> > > then manually assemble the raid array, LVM finds its meta data, builds
> > > the VG and LV and mounts. If I then manually force the exact same SATA
> > > controller failure, which results in the exact same mdraid behaviour,
> > > I am then able to unmount the filesystem and ...... hey presto
> > > deactivate the LVM LV. Which then allows me to --stop the mdraid. Just
> > > as I want. Again it does not solve my SATA hardware issues, but being
> > > at this point does give me options to restart the hardware etc, and
> > > probably, though very messily, get the filesystem up again without a
> > > reboot. The problem being I can not achieve this behaviour without
> > > manually assembling the array after boot. If you have any idea what
> > > could possibly be holding this lock I would be glad to hear.
> > >
> > Could you connect to the udev monitor and analyze events triggered in
> > both cases? This is the one idea I have.
> >
> > Thanks,
> > Mariusz
> >
> > > At this point I'm going to have to try and systematically step through
> > > the boot process and try re-arranging, when the array gets assembled.
> > > My first attempts at this have been to <ignore> the raid array in
> > > mdadm.conf and comment the array out of /etc/fstab. In this way mdraid
> > > inside initramfs does not auto-assemble and LVM does not auto scan for
> > > the VG. Once I am in the real boot sequence, I have created a systemd
> > > mount unit that I can pull in from other systemd units, to change the
> > > point in the boot process when the array is assembled. In this way
> > > hopefully I can influence when other services are interacting with the
> > > array in some way and perhaps find the root cause ...... Work in
> > > progress..but slowly as the fault occurs only very occasionally and I
> > > still need a working server.
> > > All the best.. Aidan
> > >
> > > On Mon, 10 Jan 2022 at 10:47, Mariusz Tkaczyk
> > > <mariusz.tkaczyk@xxxxxxxxxxxxxxx> wrote:
> > > >
> > > > On Fri, 7 Jan 2022 23:30:31 +0100
> > > > Aidan Walton <aidan.walton@xxxxxxxxx> wrote:
> > > >
> > > > > Hi,
> > > > > I have a system running:
> > > > > Ubuntu Server 20.04.3 LTS on a 5.4.0-92-generic kernel.
> > > > >
> > > > > On the motherboard is a:
> > > > > SATA controller: JMicron Technology Corp. JMB363 SATA/IDE
> > > > > Controller (rev 02)
> > > > >
> > > > > Connected to this I have:
> > > > > 2x Western Digital - WDC WD5001AALS-00L3B2, BIOS :01.03B01 500Gb
> > > > > drives
> > > > >
> > > > > Each drive has a single partition and is part of a RAID1 array:
> > > > > /dev/md90:
> > > > > .....
> > > > > Number Major Minor RaidDevice State
> > > > > 0 8 33 0 active sync /dev/sdc1
> > > > > 2 8 49 1 active sync /dev/sdd1
> > > > >
> > > > > On top of this I have a single LVM VG and LV. (Probably not
> > > > > important)
> > > > >
> > > > > I have noticed some strange behaviour and upon investigation I
> > > > > find the md device in the following state:
> > > > > /dev/md90:
> > > > > ....
> > > > >
> > > > > Number Major Minor RaidDevice State
> > > > > 0 8 33 0 active sync /dev/sdc1
> > > > > - 0 0 1 removed
> > > > >
> > > > > 2 8 49 - faulty /dev/sdd1
> > > > >
> > > > >
> > > > > In fact neither /dev/sdc1 or /dev/sdd1 are available. In fact
> > > > > neither are /dev/sdc or /dev/sdd, the physical drives, as they
> > > > > both been disconnected by the kernel:
> > > > > /dev/sdc is attached to ata7:00 and /dev/sdd is attached to
> > > > > ata.8:00 This is the log of the kernel events:
> > > > >
> > > > >
> > > > > Jan 07 22:09:03 mx kernel: ata7.00: exception Emask 0x32 SAct 0x0
> > > > > SErr 0x0 action 0xe frozen
> > > > > Jan 07 22:09:03 mx kernel: ata7.00: irq_stat 0xffffffff, unknown
> > > > > FIS 00000000 00000000 00000000 00000000, host bus
> > > > > Jan 07 22:09:03 mx kernel: ata7.00: failed command: READ DMA
> > > > > Jan 07 22:09:03 mx kernel: ata7.00: cmd
> > > > > c8/00:00:00:cf:26/00:00:00:00:00/e0 tag 18 dma 131072 in
> > > > > Jan 07 22:09:03 mx kernel: ata7.00: status: { DRDY }
> > > > > Jan 07 22:09:03 mx kernel: ata7: hard resetting link
> > > > > Jan 07 22:09:03 mx kernel: ata7: SATA link up 1.5 Gbps (SStatus
> > > > > 113 SControl 310)
> > > > > Jan 07 22:09:09 mx kernel: ata7.00: qc timeout (cmd 0xec)
> > > > > Jan 07 22:09:09 mx kernel: ata7.00: failed to IDENTIFY (I/O error,
> > > > > err_mask=0x4) Jan 07 22:09:09 mx kernel: ata7.00: revalidation
> > > > > failed (errno=-5) Jan 07 22:09:09 mx kernel: ata7: hard resetting
> > > > > link Jan 07 22:09:19 mx kernel: ata7: softreset failed (1st FIS
> > > > > failed) Jan 07 22:09:19 mx kernel: ata7: hard resetting link
> > > > > Jan 07 22:09:29 mx kernel: ata7: softreset failed (1st FIS failed)
> > > > > Jan 07 22:09:29 mx kernel: ata7: hard resetting link
> > > > > Jan 07 22:09:35 mx kernel: ata8.00: exception Emask 0x40 SAct 0x0
> > > > > SErr 0x800 action 0x6 frozen
> > > > > Jan 07 22:09:35 mx kernel: ata8: SError: { HostInt }
> > > > > Jan 07 22:09:35 mx kernel: ata8.00: failed command: READ DMA
> > > > > Jan 07 22:09:35 mx kernel: ata8.00: cmd
> > > > > c8/00:00:00:64:4a/00:00:00:00:00/e0 tag 2 dma 131072 in
> > > > > Jan 07 22:09:35 mx kernel: ata8.00: status: { DRDY }
> > > > > Jan 07 22:09:35 mx kernel: ata8: hard resetting link
> > > > > Jan 07 22:09:45 mx kernel: ata8: softreset failed (1st FIS failed)
> > > > > Jan 07 22:09:45 mx kernel: ata8: hard resetting link
> > > > > Jan 07 22:09:55 mx kernel: ata8: softreset failed (1st FIS failed)
> > > > > Jan 07 22:09:55 mx kernel: ata8: hard resetting link
> > > > > Jan 07 22:10:04 mx kernel: ata7: softreset failed (1st FIS failed)
> > > > > Jan 07 22:10:04 mx kernel: ata7: hard resetting link
> > > > > Jan 07 22:10:09 mx kernel: ata7: softreset failed (1st FIS failed)
> > > > > Jan 07 22:10:09 mx kernel: ata7: reset failed, giving up
> > > > > Jan 07 22:10:09 mx kernel: ata7.00: disabled
> > > > > Jan 07 22:10:09 mx kernel: ata7: EH complete
> > > > > Jan 07 22:10:30 mx kernel: ata8: softreset failed (1st FIS failed)
> > > > > Jan 07 22:10:30 mx kernel: ata8: hard resetting link
> > > > > Jan 07 22:10:35 mx kernel: ata8: softreset failed (1st FIS failed)
> > > > > Jan 07 22:10:35 mx kernel: ata8: reset failed, giving up
> > > > > Jan 07 22:10:35 mx kernel: ata8.00: disabled
> > > > > Jan 07 22:10:35 mx kernel: ata8: EH complete
> > > > >
> > > > > This is happening because of some issue with the SATA controller
> > > > > on the motherboard. This has not been resolved and probably never
> > > > > will be, I see many others through google search complaining of
> > > > > similar issues with the SATA controller.
> > > > > This failure only occurs when the SATA controller is placed under
> > > > > very heavy load, I have minimised the impact of the problem by
> > > > > not using NCQ, this helps, but it still occurs. Ironically the
> > > > > biggest issue I have is that mdadm "checkarray" is running
> > > > > because of a systemd background process every week or so, and
> > > > > this hammers the disk into failure. Most of the normal daily
> > > > > usage never generates the link resets.
> > > > > Naturally I have changed SATA cables and moved drives around onto
> > > > > different controllers, but alas, it does seem to be the hardware
> > > > > on the motherboard.
> > > > > However as a workaround I was hoping to accept the occasional
> > > > > failure and then using some scripting and 'setpci' I can get the
> > > > > kernel to hard reset the chipset and attach the drives again. I
> > > > > have the process working in terms of getting the kernel to
> > > > > re-attach the drives, but.......
> > > > >
> > > > > Unfortunately mdraid will not let go of them, I can not stop the
> > > > > arrays, and therefore can't rebuild them. If I simply allow the
> > > > > kernel to re-attach the drives the kernel names are swapped over,
> > > > > as something (mdraid) is stopping the kernel re-using the same
> > > > > device names. Anyway being dependent on the same kernel device
> > > > > names is not a great plan anyway, so I was simply trying to get
> > > > > mdadm to reassemble the array as soon as the 'workaround' script
> > > > > gets the drives back in contact with libata (kernel).
> > > > >
> > > > > Plan:
> > > > > 1. Detecting the problem. (mdadm state)
> > > > > 2. Stop the array totally (can NOT do it)
> > > > > 3. reset the chipset across the PCI bus.
> > > > > 4. allow kernel to re-attach drives.
> > > > > 5. re-assemble the md device with mdadm
> > > > > 6. restart, if necessary higher layer services...
> > > > >
> > > > > So why is mdraid holding on to the array:
> > > > >
> > > > > # mdadm --stop /dev/md90
> > > > > mdadm: Cannot get exclusive access to /dev/md90:Perhaps a running
> > > > > process, mounted filesystem or active volume group?
> > > > >
> > > > > I can not be 100% sure that something else is using the device,
> > > > > but I can't think of anything that is and I stopped every process
> > > > > I can think of..... Plus why is the array still shown as 'active'
> > > > > when none of its member devices even exist anymore?
> > > > >
> > > > > What I do know is that device mapper (coming down from LVM)
> > > > > still has an entry in /dev/mapper. But then probably no surprise
> > > > > as /dev/md90 the failed array is still an active device node. If
> > > > > you attempt to write to it, I receive I/O errors from the kernel.
> > > > > In fact as far as any higher layer services are concerned md90
> > > > > and the LVM LV on top of it are still active and working when in
> > > > > reality, they are not. It causes very strange NFS errors and such.
> > > > >
> > > > > mdraid does actually attempt to iteratively remove both partitions
> > > > > when the kernel signals the disable state, but only 1 of them
> > > > > succeeds.
> > > > > I did an strace of the same iterative 'fail:remove' process that
> > > > > mdraid attempts when the kernel issues -- kernel: ata7.00:
> > > > > disabled
> > > > >
> > > > > eg:
> > > > > /sbin/mdadm -If sdc1 --path pci-0000:02:00.0-ata-1
> > > > > mdadm: set device faulty failed for sdc1: Device or resource busy
> > > > >
> > > > > The only clue is perhaps this line from the strace:
> > > > > openat(AT_FDCWD, "/sys/block/md90/md/dev-sdc1/block/dev", O_RDWR)
> > > > > = -1 EACCES (Permission denied) What is the mdadm command
> > > > > doing that results in a permission problem?
> > > > >
> > > > > So the only way I can get rid of this md raid array is a reboot.
> > > > > Damn!!!
> > > > >
> > > > >
> > > > > Any help is much appreciated.
> > > > > Aidan
> > > > >
> > > > >
> > > > >
> > > > Hi Aidan,
> > > > This is how it is implemented. Drive is not removed if array failure
> > > > will cause array failed. Please see:
> > > > https://git.kernel.org/pub/scm/linux/kernel/git/song/md.git/commit/?id=9a567843f7ce0037bfd4d5fdc58a09d0a527b28b
> > > >
> > > > For RAID1 you can use solution proposed in patch below but IMO it is
> > > > not your problem. Please stop LVM and then try to stop array. To
> > > > stop array it needs to be "free" (all upper handlers are down).
> > > >
> > > > Thanks,
> > > > Mariusz
> >
