Re: RAID5 Phantom Drive Appeared while Reshaping Four Drive Array (HARDLOCK)

[Yu Kuai <yukuai1@xxxxxxxxxxxxxxx>]

Hi,

在 2023/05/22 14:56, raid 写道:
> Hi,
> Thanks for the guidance as the current state has at least changed somewhat.
>
> BTW Sorry about Life getting in the way of tech. =) Reason for my delayed response.
>
> -sudo mdadm -I /dev/sdc1
> mdadm: /dev/sdc1 attached to /dev/md480, not enough to start (1).
> -sudo mdadm -D /dev/md480
> /dev/md480:
>             Version : 1.2
>          Raid Level : raid0
>       Total Devices : 1
>         Persistence : Superblock is persistent
>
>               State : inactive
>     Working Devices : 1
>
>       Delta Devices : 1, (-1->0)
>           New Level : raid5
>          New Layout : left-symmetric
>       New Chunksize : 512K
>
>                Name : GRANDSLAM:480
>                UUID : 20211025:02005a7a:5a7abeef:cafebabe
>              Events : 78714
>
>      Number   Major   Minor   RaidDevice
>
>         -       8       33        -        /dev/sdc1
> -sudo mdadm -I /dev/sdd1
> mdadm: /dev/sdd1 attached to /dev/md480, not enough to start (2).
> -sudo mdadm -D /dev/md480
> /dev/md480:
>             Version : 1.2
>          Raid Level : raid0
>       Total Devices : 2
>         Persistence : Superblock is persistent
>
>               State : inactive
>     Working Devices : 2
>
>       Delta Devices : 1, (-1->0)
>           New Level : raid5
>          New Layout : left-symmetric
>       New Chunksize : 512K
>
>                Name : GRANDSLAM:480
>                UUID : 20211025:02005a7a:5a7abeef:cafebabe
>              Events : 78714
>
>      Number   Major   Minor   RaidDevice
>
>         -       8       49        -        /dev/sdd1
>         -       8       33        -        /dev/sdc1
> -sudo mdadm -I /dev/sde1
> mdadm: /dev/sde1 attached to /dev/md480, not enough to start (2).
> -sudo mdadm -D /dev/md480
> /dev/md480:
>             Version : 1.2
>          Raid Level : raid0
>       Total Devices : 3
>         Persistence : Superblock is persistent
>
>               State : inactive
>     Working Devices : 3
>
>       Delta Devices : 1, (-1->0)
>           New Level : raid5
>          New Layout : left-symmetric
>       New Chunksize : 512K
>
>                Name : GRANDSLAM:480
>                UUID : 20211025:02005a7a:5a7abeef:cafebabe
>              Events : 78712
>
>      Number   Major   Minor   RaidDevice
>
>         -       8       65        -        /dev/sde1
>         -       8       49        -        /dev/sdd1
>         -       8       33        -        /dev/sdc1
> -sudo mdadm -I /dev/sdf1
> mdadm: /dev/sdf1 attached to /dev/md480, not enough to start (3).
> -sudo mdadm -D /dev/md480
> /dev/md480:
>             Version : 1.2
>          Raid Level : raid0
>       Total Devices : 4
>         Persistence : Superblock is persistent
>
>               State : inactive
>     Working Devices : 4
>
>       Delta Devices : 1, (-1->0)
>           New Level : raid5
>          New Layout : left-symmetric
>       New Chunksize : 512K
>
>                Name : GRANDSLAM:480
>                UUID : 20211025:02005a7a:5a7abeef:cafebabe
>              Events : 78714
>
>      Number   Major   Minor   RaidDevice
>
>         -       8       81        -        /dev/sdf1
>         -       8       65        -        /dev/sde1
>         -       8       49        -        /dev/sdd1
>         -       8       33        -        /dev/sdc1
> -sudo mdadm -R /dev/md480
> mdadm: failed to start array /dev/md480: Input/output error
> ---
> NOTE: Of additional interest...
> ---
> -sudo mdadm -D /dev/md480
> /dev/md480:
>             Version : 1.2
>       Creation Time : Tue Oct 26 14:06:53 2021
>          Raid Level : raid5
>       Used Dev Size : 18446744073709551615
>        Raid Devices : 5
>       Total Devices : 3
>         Persistence : Superblock is persistent
>
>         Update Time : Thu May  4 14:39:03 2023
>               State : active, FAILED, Not Started
>      Active Devices : 3
>     Working Devices : 3
>      Failed Devices : 0
>       Spare Devices : 0
>
>              Layout : left-symmetric
>          Chunk Size : 512K
>
> Consistency Policy : unknown
>
>       Delta Devices : 1, (4->5)
>
>                Name : GRANDSLAM:480
>                UUID : 20211025:02005a7a:5a7abeef:cafebabe
>              Events : 78714
>
>      Number   Major   Minor   RaidDevice State
>         -       0        0        0      removed
>         -       0        0        1      removed
>         -       0        0        2      removed
>         -       0        0        3      removed
>         -       0        0        4      removed
>
>         -       8       81        3      sync   /dev/sdf1
>         -       8       49        1      sync   /dev/sdd1
>         -       8       33        0      sync   /dev/sdc1

So the reason that this array can't start is that /dev/sde1 is not
recognized as RaidDevice 2, and there are two RaidDevice missing for
a raid5.

Sadly I have no idea to workaroud this, sb metadate seems to be broken.

Thanks,
Kuai
> ---
> -watch -c -d -n 1 cat /proc/mdstat
> ---
> Every 1.0s: cat /proc/mdstat                                                     OAK2023: Mon May 22 01:48:24 2023
>
> Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
> md480 : inactive sdf1[4] sdd1[1] sdc1[0]
>        46877239294 blocks super 1.2
>
> unused devices: <none>
> ---
> Hopeful that is some progress towards an array start? It's definately unexpected output to me.
> I/O Error starting md480
>
> Thanks!
> SA
>
> On Thu, 2023-05-18 at 11:15 +0800, Yu Kuai wrote:
>
> > I have no idle why other disk shows that device 2 is missing, and what
> > is device 4.
> >
> > Anyway, can you try the following?
> >
> > mdadm -I /dev/sdc1
> > mdadm -D /dev/mdxxx
> >
> > mdadm -I /dev/sdd1
> > mdadm -D /dev/mdxxx
> >
> > mdadm -I /dev/sde1
> > mdadm -D /dev/mdxxx
> >
> > mdadm -I /dev/sdf1
> > mdadm -D /dev/mdxxx
> >
> > If above works well, you can try:
> >
> > mdadm -R /dev/mdxxx, and see if the array can be started.
> >
> > Thanks,
> > Kuai
>
>
>
>
> .