On Fri, 13 Apr 2018, Dan Van Der Ster wrote: > > On 12 Apr 2018, at 17:44, Sage Weil <sage@xxxxxxxxxxxx> wrote: > > > > On Thu, 12 Apr 2018, Dan Van Der Ster wrote: > >> Hi all, > >> > >> We noticed that changing existing pools from being class-agnostic to > >> choose a class (e.g. hdd [1]) then a lot, perhaps all, data will move. > >> > >> I think this is because the crush IDs in the shadow tree are too > >> different, or maybe just a different order, from the original bucket > >> IDs. > >> > >> But I wonder if we could use `ceph osd crush swap-bucket` (or a more > >> elaborate version of swap bucket) to move the shadow buckets into the > >> correct place in the tree, thereby preventing data movement. > >> > >> Did anyone already ponder on this topic? > > > > We could do a one-time transition that swaps the "real" ids with the > > shadow ids for one of the classes, and at the same time change the crush > > rule for the pool. > > > > This will only really work/help if the majority of the devices are one > > class. That's probably usually the case? > > That's indeed our use-case. We have an all-hdd cluster for rgw -- we want to add *add* some ssd osds to each host in the cluster, then use the device class rules to map the bucket_data/index pools accordingly. > > Do you expect no data movement if we use crushtool to decompile, swap > the IDs, then recompile and setcrushmap? Correct. You can test it out with the osdmaptool --test-map-pgs on the before and after maps. I think you'll want to edit the CRUSH rule to use the device class in place as well, and you'll probably need to create a rule using the class ahead of time so that all the shadow ids are there to swap with. Let me know if there are problems! sage _______________________________________________ Ceph-large mailing list Ceph-large@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-large-ceph.com
