On Apr 6, 2015, at 1:49 PM, Craig Lewis <clewis@xxxxxxxxxxxxxxxxxx> wrote: > In that case, I'd set the crush weight to the disk's size in TiB, and mark the osd out: > ceph osd crush reweight osd.<OSDID> <weight> > ceph osd out <OSDID> > > Then your tree should look like: > -9 2.72 host ithome > 30 2.72 osd.30 up 0 > > An OSD can be UP and OUT, which causes Ceph to migrate all of it's data away. Thanks, Craig, but that didn’t seem to work. Some *other* PGs moved around in my cluster but not those on osd.30. Craig, both you and Jean-Charles suggested I up the weight of the temporary OSD to get the PGs there to migrate off which I don’t really understand and is counter to the article I read here: https://ceph.com/community/incomplete-pgs-oh-my/ I’m not sure why the OSD would need any weight to have it’s PGs move off of it back into the cluster. As this didn’t work I’m going to try the following unless someone tells me it’s a bad idea: * Move osd.0 from storage1 over to Ithome * Start osd.0 there and then stop it (to cover any upgrade process because storage1 is 0.80.9 and Ithome is 0.87.1) * Use ceph_objectstore_tool to import PGs 3.c7 and 3.102 into osd.0 * If that works: ** Set osd.0 out and have it flush all of its PGs back to the OSDs on storage1. ** Remove osd.0 from the cluster ** Move the drive back to storage1 and re-deploy it as a new 0.80.9 OSD * If the ceph_objectstore_tool fails (because there’s already an empty / bad copy of those PGs) ** Attempt to remove current/3.c7_head and current/3.102_head and try again Just for reference again my ceph osd tree looks like: # id weight type name up/down reweight -1 81.65 root default -2 81.65 host storage1 -3 13.63 journal storage1-journal1 1 2.72 osd.1 up 1 4 2.72 osd.4 up 1 2 2.73 osd.2 up 1 3 2.73 osd.3 up 1 0 2.73 osd.0 up 1 -4 13.61 journal storage1-journal2 5 2.72 osd.5 up 1 6 2.72 osd.6 up 1 8 2.72 osd.8 up 1 9 2.72 osd.9 up 1 7 2.73 osd.7 up 1 -5 13.6 journal storage1-journal3 11 2.72 osd.11 up 1 12 2.72 osd.12 up 1 13 2.72 osd.13 up 1 14 2.72 osd.14 up 1 10 2.72 osd.10 up 1 -6 13.61 journal storage1-journal4 16 2.72 osd.16 up 1 17 2.72 osd.17 up 1 18 2.72 osd.18 up 1 19 2.72 osd.19 up 1 15 2.73 osd.15 up 1 -7 13.6 journal storage1-journal5 20 2.72 osd.20 up 1 21 2.72 osd.21 up 1 22 2.72 osd.22 up 1 23 2.72 osd.23 up 1 24 2.72 osd.24 up 1 -8 13.6 journal storage1-journal6 25 2.72 osd.25 up 1 26 2.72 osd.26 up 1 27 2.72 osd.27 up 1 28 2.72 osd.28 up 1 29 2.72 osd.29 up 1 -9 0 host ithome 30 0 osd.30 up 0 The PGs I lost are currently mapped to osd.0 and osd.15. Those are the two drives that failed at the same time in my double replica cluster. PGs 3.c7 and 3.102 are apparently the only two PGs which were on both of those drives. I was able to extract the data from those PGs from one of the dead drives using ceph_objectstore_tool and that seems to have been successful. I imported those PGs using ceph_objectstore_tool to the temporary OSD on Ithome, osd.30. I just can’t seem to get them to migrate from osd.30 back to 0 and 15. I have min_size = 1 and size = 2. > On Thu, Apr 2, 2015 at 10:20 PM, Chris Kitzmiller <cakDS@xxxxxxxxxxxxx> wrote: >> On Apr 3, 2015, at 12:37 AM, LOPEZ Jean-Charles <jelopez@xxxxxxxxxx> wrote: >>> according to your ceph osd tree capture, although the OSD reweight is set to 1, the OSD CRUSH weight is set to 0 (2nd column). You need to assign the OSD a CRUSH weight so that it can be selected by CRUSH: ceph osd crush reweight osd.30 x.y (where 1.0=1TB) >>> >>> Only when this is done will you see if it joins. >> >> I don't really want osd.30 to join my cluster though. It is a purely temporary device that I restored just those two PGs to. It should still be able to (and be trying to) push out those two PGs with a weight of zero, right? I don't want any of my production data to migrate towards osd.30. _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |