Hello, On Tue, 29 Mar 2016 14:00:44 +0800 lin zhou wrote: > Hi,Christian. > When I re-add these OSD(0,3,9,12,15),the high latency occur again.the > default reweight of these OSD is 0.0 > That makes no sense, at a crush weight (not reweight) of 0 they should not get used at all. When you deleted the other OSD (6?) because of high latency, was your only reason/data point the "ceph osd perf" output? > root@node-65:~# ceph osd tree > # id weight type name up/down reweight > -1 103.7 root default > -2 8.19 host node-65 > 18 2.73 osd.18 up 1 > 21 0 osd.21 up 1 > 24 2.73 osd.24 up 1 > 27 2.73 osd.27 up 1 > 30 0 osd.30 up 1 > 33 0 osd.33 up 1 > 0 0 osd.0 up 1 > 3 0 osd.3 up 1 > 6 0 osd.6 down 0 > 9 0 osd.9 up 1 > 12 0 osd.12 up 1 > 15 0 osd.15 up 1 > > ceph osd perf: > 0 9825 10211 > 3 9398 9775 > 9 35852 36904 > 12 24716 25626 > 15 18893 19633 > This could very well be old, stale data. Still, these are some seriously bad numbers, if they are real. Do the these perf numbers change at all? My guess would be no. > but iostat of these device is empty. Unsurprising, as they should not be used by Ceph with a weight of 0. atop gives you an even better, complete view. > smartctl say nothing error found in these OSD device. > What exactly are these devices (model please), 3TB SATA drives Ia assume? How are they connect (controller)? Christian > 2016-03-29 13:22 GMT+08:00 lin zhou <hnuzhoulin2@xxxxxxxxx>: > > Thanks.I try this method just like ceph document say. > > But I just test osd.6 in this way,and the leveldb of osd.6 is > > broken.so it can not start. > > > > When I try this for other osd,it works. > > > > 2016-03-29 8:22 GMT+08:00 Christian Balzer <chibi@xxxxxxx>: > >> On Mon, 28 Mar 2016 18:36:14 +0800 lin zhou wrote: > >> > >>> > Hello, > >>> > > >>> > On Sun, 27 Mar 2016 13:41:57 +0800 lin zhou wrote: > >>> > > >>> > > Hi,guys. > >>> > > some days ago,one osd have a large latency seeing in ceph osd > >>> > > perf.and this device make this node a high cpu await. > >>> > The thing to do at that point would have been look at things with > >>> > atop or iostat to verify that it was the device itself that was > >>> > slow and not because it was genuinely busy due to uneven activity > >>> > maybe. As well as a quick glance at SMART of course. > >>> > >>> Thanks.I will follow this when I face this problem next time. > >>> > >>> > > So,I delete this osd ad then check this device. > >>> > If that device (HDD, SSD, which model?) slowed down your cluster, > >>> > you should not have deleted it. > >>> > The best method would have been to set your cluster to noout and > >>> > stop that specific OSD. > >>> > > >>> > When you say "delete", what exact steps did you take? > >>> > Did this include removing it from the crush map? > >>> > >>> Yes,I delete it from crush map.delete its auth,and rm osd. > >>> > >> > >> Google is your friend, if you deleted it like in the link below you > >> should be be able to re-add it the same way: > >> http://lists.ceph.com/pipermail/ceph-users-ceph.com/2015-June/002345.html > >> > >> Christian > >> > >>> > > But nothing error found. > >>> > > > >>> > > And now I want to re-add this device into cluster with it's data. > >>> > > > >>> > All the data was already replicated elsewhere if you > >>> > deleted/removed the OSD, you're likely not going to save much if > >>> > any data movement by re-adding it. > >>> > >>> Yes,the cluster finished rebalance.but I face a problem of one > >>> unfound object. And in the output of pg query in recovery_state > >>> say,this osd is down,but other odds are ok. > >>> So I want to recover this osd to recover this unfound object. > >>> > >>> and mark_unfound_lost revert/delete do not work: > >>> Error EINVAL: pg has 1 unfound objects but we haven't probed all > >>> sources, > >>> > >>> detail see: > >>> http://lists.ceph.com/pipermail/ceph-users-ceph.com/2016-March/008452.html > >>> > >>> Thanks again. > >>> > >>> > > > >>> > > I try to using ceph-osd to add it,but it can not start.log are > >>> > > paste in : > >>> > > https://gist.github.com/hnuzhoulin/836f9e633b90041e89ad > >>> > > > >>> > > so what's the recommend steps. > >>> > That depends on how you deleted it, but at this point your data is > >>> > likely to be mostly stale anyway, so I'd start from scratch. > >>> > >>> > Christian > >>> > -- > >>> > Christian Balzer Network/Systems Engineer > >>> > chibi@xxxxxxx Global OnLine Japan/Rakuten Communications > >>> > http://www.gol.com/ > >>> > > >>> > >> > >> > >> -- > >> Christian Balzer Network/Systems Engineer > >> chibi@xxxxxxx Global OnLine Japan/Rakuten Communications > >> http://www.gol.com/ > -- Christian Balzer Network/Systems Engineer chibi@xxxxxxx Global OnLine Japan/Rakuten Communications http://www.gol.com/ _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |