Hi Paul, Many thanks for the super quick replys and analysis on this. Is it a case of removing the weights from the new hosts and there osds then moving them? After reweighing them correctly? I already have a bug open, I will get this email chain added to this. Warren ________________________________ From: Paul Emmerich [paul.emmerich@xxxxxxxx] Sent: 16 March 2018 16:48 To: Jeffs, Warren (STFC,RAL,ISIS) Cc: ceph-users@xxxxxxxx Subject: Re: Crush Bucket move crashes mons Hi, looks like it fails to adjust the number of weight set entries when moving the entries. The good news is that this is 100% reproducible with your crush map: you should open a bug at http://tracker.ceph.com/ to get this fixed. Deleting the weight set fixes the problem. Moving the item manually with manual adjustment of the weight set also works in my quick test. Paul 2018-03-16 16:03 GMT+01:00 <warren.jeffs@xxxxxxxxxx<mailto:warren.jeffs@xxxxxxxxxx>>: Hi Paul Many thanks for the reply. The command is: crush move rack04 room=R80-Upper Crush map is here: https://pastebin.com/CX7GKtBy I’ve done some more testing, and the following all work: • Moving machines between the racks under the default root. • Renaming racks/hosts under the default root • Renaming the default root • Creating a new root • Adding rack05 and rack04 + hosts nina408 and nina508 into the new root But when trying to move anything into the default root it fails. I have tried moving the following into default root: • Nina408 – with hosts in and without • Nina508 – with hosts in and without • Rack04 • Rack05 • Rack03 – which I created with nothing in it to try and move. Since first email, I have got the cluster to HEALTH_OK with reweight mapping drives, so everything cluster wise appears to be functioning fine. I have not tried manually editing the crush map and reimporting for the risk that it makes the cluster fall over, as this is currently in production. With the CLI I can at least cancel the command the monitor comes back up fine. Many thanks. Warren From: Paul Emmerich [mailto:paul.emmerich@xxxxxxxx<mailto:paul.emmerich@xxxxxxxx>] Sent: 16 March 2018 13:54 To: Jeffs, Warren (STFC,RAL,ISIS) <warren.jeffs@xxxxxxxxxx<mailto:warren.jeffs@xxxxxxxxxx>> Cc: ceph-users@xxxxxxxx<mailto:ceph-users@xxxxxxxx> Subject: Re: Crush Bucket move crashes mons Hi, the error looks like there might be something wrong with the device classes (which are managed via separate trees with magic names behind the scenes). Can you post your crush map and the command that you are trying to run? Paul 2018-03-15 16:27 GMT+01:00 <warren.jeffs@xxxxxxxxxx<mailto:warren.jeffs@xxxxxxxxxx>>: Hi All, Having some interesting challenges. I am trying to move 2 new nodes + 2 new racks into my default root, I have added them to the cluster outside of the Root=default. They are all in and up – happy it seems. The new nodes have all 12 OSDs in them and they are all ‘UP’ So when going to move them into the correctly room bucket under the default root they fail. This is the error log at the time: https://pastebin.com/mHfkEp3X I can create another host in the crush and move that in and out of rack buckets – all while being outside of the default root. Trying to move an empty Rack bucket into the default root fails too. All of the cluster is on 12.2.4. I do have 2 backfill full osds which is the reason for needing these disks in the cluster asap. Any thoughts? Cheers Warren Jeffs ISIS Infrastructure Services STFC Rutherford Appleton Laboratory e-mail: warren.jeffs@xxxxxxxxxx<mailto:warren.jeffs@xxxxxxxxxx> _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx<mailto:ceph-users@xxxxxxxxxxxxxx> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com -- -- Paul Emmerich croit GmbH Freseniusstr. 31h<https://maps.google.com/?q=Freseniusstr.+31h+%0D%0A81247+M%C3%BCnchen&entry=gmail&source=g> 81247 München www.croit.io<http://www.croit.io> Tel: +49 89 1896585 90<tel:+49%2089%20189658590> -- -- Paul Emmerich croit GmbH Freseniusstr. 31h 81247 München www.croit.io<http://www.croit.io> Tel: +49 89 1896585 90 _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |