Re: HDD bad sector, pg inconsistent, no object remapping

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 




Replication does not occur until the OSD is “out.”  This creates a new mapping in the cluster of where the PGs should be and thus data begins to move and/or create sufficient copies.  This scheme lets you control how and when you want the replication to occur.  If you have plenty of space and you aren’t going to replace the drive immediately, just mark the OSD “down" AND “out.".  If you are going to replace the drive immediately, set the “noout” flag.  Take the OSD “down” and replace drive.  Assuming it is mounted in the same place as the bad drive, bring the OSD back up.  This will replicate exactly the same PGs the bad drive held back to the replacement drive.  As was stated before don’t forget to “ceph osd unset noout"

Keep in mind that in the case of a machine that has a hardware failure and takes OSD(s) down there is an automatic timeout which will mark them “out" for unattended operation.  Unless you are monitoring the cluster 24/7 you should have enough disk space available to handle failures.

Related info in:


David Zafman
Senior Developer

On Nov 15, 2013, at 1:58 AM, Mihály Árva-Tóth <mihaly.arva-toth@xxxxxxxxxxxxxxxxxxxxxx> wrote:

Hello,

I think you misunderstood me. I known how can I replace bad HDD, thanks. My problem is the following:

Object replica number is 3. Objects that in 11.15d PG which store data on osd.0 bad sectors place inter alia. Ceph should know objects are in 11.15d on osd.0 is bad (because deep-scrub and repair are both failed), so there is no 3 clean replica number only two. I think Ceph should replicate its to new osd. Or not?

Thank you,
Mihaly

2013/11/13 David Zafman <david.zafman@xxxxxxxxxxx>

Since the disk is failing and you have 2 other copies I would take osd.0 down.  This means that ceph will not attempt to read the bad disk either for clients or to make another copy of the data:

***** Not sure about the syntax of this for the version of ceph you are running
ceph osd down 0

Mark it “out” which will immediately trigger recovery to create more copies of the data with the remaining OSDs.
ceph osd out 0

You can now finish the process of removing the osd by looking at these instructions:

http://ceph.com/docs/master/rados/operations/add-or-rm-osds/#removing-osds-manual

David Zafman
Senior Developer
http://www.inktank.com

On Nov 12, 2013, at 3:16 AM, Mihály Árva-Tóth <mihaly.arva-toth@xxxxxxxxxxxxxxxxxxxxxx> wrote:

> Hello,
>
> I have 3 node, with 3 OSD in each node. I'm using .rgw.buckets pool with 3 replica. One of my HDD (osd.0) has just bad sectors, when I try to read an object from OSD direct, I get Input/output errror. dmesg:
>
> [1214525.670065] mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
> [1214525.670072] mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
> [1214525.670100] sd 0:0:2:0: [sdc] Unhandled sense code
> [1214525.670104] sd 0:0:2:0: [sdc]
> [1214525.670107] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [1214525.670110] sd 0:0:2:0: [sdc]
> [1214525.670112] Sense Key : Medium Error [current]
> [1214525.670117] Info fld=0x60c8f21
> [1214525.670120] sd 0:0:2:0: [sdc]
> [1214525.670123] Add. Sense: Unrecovered read error
> [1214525.670126] sd 0:0:2:0: [sdc] CDB:
> [1214525.670128] Read(16): 88 00 00 00 00 00 06 0c 8f 20 00 00 00 08 00 00
>
> Okay I known need to replace HDD.
>
> Fragment of ceph -s  output:
>   pgmap v922039: 856 pgs: 855 active+clean, 1 active+clean+inconsistent;
>
> ceph pg dump | grep inconsistent
>
> 11.15d  25443   0       0       0       6185091790      3001    3001    active+clean+inconsistent       2013-11-06 02:30:45.23416.....
>
> ceph pg map 11.15d
>
> osdmap e1600 pg 11.15d (11.15d) -> up [0,8,3] acting [0,8,3]
>
> pg repair or deep-scrub can not fix this issue. But if I understand correctly, osd has to known it can not retrieve object from osd.0 and need to be replicate an another osd because there is no 3 working replicas now.
>
> Thank you,
> Mihaly
> _______________________________________________
> ceph-users mailing list
> ceph-users@xxxxxxxxxxxxxx
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux