There is a (unfortunately non-optional at the moment) feature in crush where we retry in the same bucket a few times before restarting the descent when hitting an out leaf. The result of this is to localise recovery at the expense of inadequately redistributing data on node failure. We will most likely remove this behaviour in the next crush version. -Sam On Wed, Apr 25, 2012 at 3:40 PM, Jim Schutt <jaschut@xxxxxxxxxx> wrote: > Hi, > > I've been experimenting with failure scenarios to make sure > I understand what happens when an OSD drops out. In particular, > I've been using "ceph osd out <n>" and watching my all my OSD > servers to see where the data from the removed OSD ends up > after recovery. I've been doing this testing with 12 servers, > 24 OSDs/server. > > What I see is that all the data from the removed OSD seem to > end up distributed across the other OSDs on the host holding > the removed OSD. > > It works this way if I use the default CRUSH map generated > for just host buckets and devices, i.e. a map using the > straw algorithm for the root and host buckets. It also > works this way if I generate my own map using the uniform > algorithm for the root and host buckets. > > For example, using my map based on the uniform algorithm, > after successively taking out osd.0 thru osd.9 (all on > host cs32), here's the top 24 OSD data store usage: > > Host 1K-blocks Used Available Use% Mounted on > > cs39: 942180120 33379144 871114904 4% /ram/mnt/ceph/data.osd.174 > cs39: 942180120 33386420 871106604 4% /ram/mnt/ceph/data.osd.186 > cs43: 942180120 33484704 871008448 4% /ram/mnt/ceph/data.osd.270 > cs43: 942180120 33563912 870930616 4% /ram/mnt/ceph/data.osd.267 > cs38: 942180120 33637652 870856876 4% /ram/mnt/ceph/data.osd.162 > cs34: 942180120 33773780 870721740 4% /ram/mnt/ceph/data.osd.67 > cs37: 942180120 33834584 870660136 4% /ram/mnt/ceph/data.osd.123 > cs40: 942180120 33936696 870557928 4% /ram/mnt/ceph/data.osd.203 > cs38: 942180120 34212020 870283404 4% /ram/mnt/ceph/data.osd.165 > cs42: 942180120 34402852 870095036 4% /ram/mnt/ceph/data.osd.259 > cs32: 942180120 45969104 858567088 6% /ram/mnt/ceph/data.osd.17 > cs32: 942180120 49694156 854854196 6% /ram/mnt/ceph/data.osd.16 > cs32: 942180120 50182636 854370356 6% /ram/mnt/ceph/data.osd.15 > cs32: 942180120 50520484 854030460 6% /ram/mnt/ceph/data.osd.12 > cs32: 942180120 50669280 853882848 6% /ram/mnt/ceph/data.osd.10 > cs32: 942180120 51234372 853319452 6% /ram/mnt/ceph/data.osd.14 > cs32: 942180120 51277080 853276808 6% /ram/mnt/ceph/data.osd.22 > cs32: 942180120 51279984 853273392 6% /ram/mnt/ceph/data.osd.13 > cs32: 942180120 52364512 852192000 6% /ram/mnt/ceph/data.osd.23 > cs32: 942180120 52376512 852180384 6% /ram/mnt/ceph/data.osd.21 > cs32: 942180120 53026724 851531228 6% /ram/mnt/ceph/data.osd.18 > cs32: 942180120 53217832 851343128 6% /ram/mnt/ceph/data.osd.20 > cs32: 942180120 53723152 850838768 6% /ram/mnt/ceph/data.osd.19 > cs32: 942180120 56159652 848410460 7% /ram/mnt/ceph/data.osd.11 > > I was thinking that CRUSH would re-replicate considering all > buckets, subject to preventing multiple replicas in the same > bucket, based on this from Sage's thesis, section 5.2.2.1: > > For failed or overloaded devices, CRUSH uniformly > redistributes items across the storage cluster by > restarting the recursion at the beginning of the > select(n,t) (see Algorithm 1 line 11). > > So my question is, what am I missing? Maybe the above doesn't > mean what I think it does? Or, is there some configuration > that I should be using but don't know about? > > Thanks -- Jim > > > -- > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in > the body of a message to majordomo@xxxxxxxxxxxxxxx > More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
