You should prioritise recovering quorum of your monitors. Cephs documentation can help here https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-mon/ Check to see if the failed mon is still part of the monmap on the other nodes, if it is you might need to remove it manually (which isn't difficult). I would not recommend running two mons on different major versions. And, ideally, you should run one or three to aid quorum though I would think the priority should be on getting one mon running by itself successfully first. On Mon, 3 Jan 2022 at 09:36, Mini Serve <soanican@xxxxxxxxx> wrote: > Hi, we have 3 node ceph cluster, long time ago installed by another team. > > Currently, we had to reinstall OS(due to disk failure) at one of them(node > 3), so we lost all configs on that node. > As all other hard drives on that node3 are intact, after installing fresh > ceph, "cephadm ceph-volume lvm list" returns all OSD information. > > node 3 was also one of the mon node. > It looks like mon doesnt work - config is messed, quorum is lost. > (mon nodes were node1 and node 3. We tried to make it as node1 and node2 - > didnt work) > "ceph -s" : node1 do not respond - just stall, node2 returns "[errno 13] > RADOS permission denied (error connecting to the cluster)" > > At all nodes we have "cephadm ceph-volume lvm list" working properly. > node1, node2 ceph version is 15.2.11 > node 3 cep version 16.2.6 > > So now I would like to ask my question: > What would be the best way to recover the ceph system? > I am ready to reinstall the cephs, configure mon & quorum etc. But its > gonna be first time. > We need to reconfigure OSDs and get the data back. > > Your kind assistance and forwarding to related solutions are highly > appreciated. > > Best, > _______________________________________________ > ceph-users mailing list -- ceph-users@xxxxxxx > To unsubscribe send an email to ceph-users-leave@xxxxxxx > -- [image: Maven Securities Ltd] <https://htmlsig.com/t/000001FNR3GK> Michael Moyles | Linux Engineer michael.moyles@xxxxxxxxxxxxxxxxxxx Maven Securities Ltd 140 Leadenhall Street, London, EC3V 4QT mavensecurities.com This e-mail message may contain confidential or legally privileged information and is intended only for the use of the intended recipient(s). Any unauthorized disclosure, dissemination, distribution, copying or the taking of any action in reliance on the information herein is prohibited. E-mails are not secure and cannot be guaranteed to be error free as they can be intercepted, amended, or contain viruses. Anyone who communicates with us by e-mail is deemed to have accepted these risks. Maven Securities Ltd is not responsible for errors or omissions in this message and denies any responsibility for any damage arising from the use of e-mail. Any opinion and other statement contained in this message and any attachment are solely those of the author and do not necessarily represent those of the company. -- This e-mail together with any attachments (the "Message") is confidential and may contain privileged information. If you are not the intended recipient or if you have received this e-mail in error, please notify the sender immediately and permanently delete this Message from your system. Do not copy, disclose or distribute the information contained in this Message. _Maven Investment Partners Ltd (No. 07511928), Maven Investment Partners US Ltd (No. 11494299), Maven Europe Ltd (No. 08966), Maven Derivatives Asia Limited (No.10361312) & Maven Securities Holding Ltd (No. 07505438) are registered as companies in England and Wales and their registered address is Level 3, 6 Bevis Marks, London EC3A 7BA, United Kingdom. The companies’ VAT No. is 135539016. Maven Asia (Hong Kong) Ltd (No. 2444041) is registered in Hong Kong and its registered address is 20/F, Tai Tung Building, 8 Fleming Road, Wan Chai, Hong Kong. Maven Derivatives Amsterdam B.V. (71291377) is registered in the Netherlands and its registered address is 12.02, Spaces, Barbara Strozzilaan 201, Amsterdam, 1083 HN, Netherlands. Maven Europe Ltd is authorised and regulated by the Financial Conduct Authority (FRN:770542). Maven Asia (Hong Kong) Ltd is registered and regulated by the Securities and Futures Commission (CE No: BJF060). Maven Derivatives Amsterdam B.V. is licensed and regulated by the Dutch Authority for the Financial Markets._ _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |