-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 If you had multiple monitors, you should recover if possible more than 50% of them (they will need to form a quorum). If you can't, it is messy but, you can manually remove enough monitors to start a quorum. >From /etc/ceph/ you will want the keyring and the ceph.conf at a minimim. The keys for the monitor I think are in the store.db which will let the monitors start, but the keyring has the admin key which lets you manage the cluster once you get it up. rbdmap is not needed for recovery (only automatically mounting RBDs at boot time), we can deal with that later if needed. - ---------------- Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1 On Wed, Jul 29, 2015 at 4:40 PM, Peter Hinman wrote: > Ok - that is encouraging. I've believe I've got data from a previous > monitor. I see files in a store.db dated yesterday, with a MANIFEST-######## > file that is significantly greater than the MANIFEST-000007 file listed for > the current monitors. > > I've actually found data for two previous monitors. Any idea which one I > should select? The one with the highest manifest number? The most recent > time stamp? > > What files should I be looking for in /etc/conf? Just the keyring and > rbdmap files? How important is it to use the same keyring file? > > -- > Peter Hinman > International Bridge / ParcelPool.com > > On 7/29/2015 3:47 PM, Robert LeBlanc wrote: >> >> -----BEGIN PGP SIGNED MESSAGE----- >> Hash: SHA256 >> >> The default is /var/lib/ceph/mon/- (/var/lib/ceph/mon/ceph-mon1 for >> me). You will also need the information from /etc/ceph/ to reconstruct >> the data. I *think* you should be able to just copy this to a new box >> with the same name and IP address and start it up. >> >> I haven't actually done this, so there still may be some bumps. >> - ---------------- >> Robert LeBlanc >> PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1 >> >> >> On Wed, Jul 29, 2015 at 3:44 PM, Peter Hinman wrote: >>> >>> Thanks Robert - >>> >>> Where would that monitor data (database) be found? >>> >>> -- >>> Peter Hinman >>> >>> >>> On 7/29/2015 3:39 PM, Robert LeBlanc wrote: >>>> >>>> -----BEGIN PGP SIGNED MESSAGE----- >>>> Hash: SHA256 >>>> >>>> If you built new monitors, this will not work. You would have to >>>> recover the monitor data (database) from at least one monitor and >>>> rebuild the monitor. The new monitors would not have any information >>>> about pools, OSDs, PGs, etc to allow an OSD to be rejoined. >>>> - ---------------- >>>> Robert LeBlanc >>>> PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1 >>>> >>>> >>>> On Wed, Jul 29, 2015 at 2:46 PM, Peter Hinman wrote: >>>>> >>>>> Hi Greg - >>>>> >>>>> So at the moment, I seem to be trying to resolve a permission error. >>>>> >>>>> === osd.3 === >>>>> Mounting xfs on stor-2:/var/lib/ceph/osd/ceph-3 >>>>> 2015-07-29 13:35:08.809536 7f0a0262e700 0 librados: osd.3 >>>>> authentication >>>>> error (1) Operation not permitted >>>>> Error connecting to cluster: PermissionError >>>>> failed: 'timeout 30 /usr/bin/ceph -c /etc/ceph/ceph.conf >>>>> --name=osd.3 >>>>> --keyring=/var/lib/ceph/osd/ceph-3/keyring osd crush create-or-move -- >>>>> 3 >>>>> 3.64 host=stor-2 root=default' >>>>> ceph-disk: Error: ceph osd start failed: Command >>>>> '['/usr/sbin/service', >>>>> 'ceph', '--cluster', 'ceph', 'start', 'osd.3']' returned non-zero exit >>>>> status 1 >>>>> ceph-disk: Error: One or more partitions failed to activate >>>>> >>>>> >>>>> Is there a way to identify the cause of this PermissionError? I've >>>>> copied >>>>> the client.bootstrap-osd key from the output of ceph auth list, and >>>>> pasted >>>>> it into /var/lib/ceph/bootstrap-osd/ceph.keyring, but that has not >>>>> resolve >>>>> the error. >>>>> >>>>> But it sounds like you are saying that even once I get this resolved, I >>>>> have >>>>> no hope of recovering the data? >>>>> >>>>> -- >>>>> Peter Hinman >>>>> >>>>> On 7/29/2015 1:57 PM, Gregory Farnum wrote: >>>>> >>>>> This sounds like you're trying to reconstruct a cluster after >>>>> destroying >>>>> the >>>>> monitors. That is...not going to work well. The monitors define the >>>>> cluster >>>>> and you can't move OSDs into different clusters. We have ideas for how >>>>> to >>>>> reconstruct monitors and it can be done manually with a lot of hassle, >>>>> but >>>>> the process isn't written down and there aren't really fools I help >>>>> with >>>>> it. >>>>> :/ >>>>> -Greg >>>>> >>>>> On Wed, Jul 29, 2015 at 5:48 PM Peter Hinman wrote: >>>>>> >>>>>> I've got a situation that seems on the surface like it should be >>>>>> recoverable, but I'm struggling to understand how to do it. >>>>>> >>>>>> I had a cluster of 3 monitors, 3 osd disks, and 3 journal ssds. After >>>>>> multiple hardware failures, I pulled the 3 osd disks and 3 journal >>>>>> ssds >>>>>> and am attempting to bring them back up again on new hardware in a new >>>>>> cluster. I see plenty of documentation on how to zap and initialize >>>>>> and >>>>>> add "new" osds, but I don't see anything on rebuilding with existing >>>>>> osd >>>>>> disks. >>>>>> >>>>>> Could somebody provide guidance on how to do this? I'm running 94.2 >>>>>> on >>>>>> all machines. >>>>>> >>>>>> Thanks, >>>>>> >>>>>> -- >>>>>> Peter Hinman >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> ceph-users mailing list >>>>>> ceph-users@xxxxxxxxxxxxxx >>>>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >>>>> >>>>> >>>>> >>>>> _______________________________________________ >>>>> ceph-users mailing list >>>>> ceph-users@xxxxxxxxxxxxxx >>>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >>>>> >>>> -----BEGIN PGP SIGNATURE----- >>>> Version: Mailvelope v0.13.1 >>>> Comment: https://www.mailvelope.com >>>> >>>> wsFcBAEBCAAQBQJVuUfwCRDmVDuy+mK58QAASJIP/0kEBx+h7LZfpQkgUvPG >>>> hKzIlSzbWIkig9O5cYzXKh03jPFz1hj38YVQ+cdYuRA1VhrNdTkwyNnzVDFk >>>> 7R98PUF4eljNNnSdQ0nIIVCS8rtGWfSUU4ECo1/4Gm8ebIMmY/g6umE87oqy >>>> fBmXW9luFZ3HQyoaqfALKWsesNJ9EJT/EgMH3+XisJZYPtpEbVDr0DiV2sbt >>>> st1xtsQwkKOGAOr+7sGe7g9dED7zCERLWsNOpeHkeJaArbKDzGY1abpoiyUt >>>> BQ5lCHGKZCBqXINaVTmwPGMTdKpED5eBxIXQ+QeEXwBONQuei4zkDz8TWRKO >>>> zaNcogcaQilSg3KyjyHzovPzVoS0OGLmEK1FVtveUfMPfMQ9XXyGnhWiZ6u7 >>>> +grlQoe4E5AZTqEMtCzKyrqldWdzL8A+S9ZidtvSi1dCZpJutEkFbI/m8A5j >>>> dA6Q7zijNJDPVMMsXXA08z6Pu7611mShXjW0fLu871++JsE/eS8GCfc9Cgyu >>>> aUgcSaWCuRVa2laXak3BI+44AexsU3ZKyveDeuFdm7y3F+DS5FKZK2V8OfJn >>>> /mbolRFyGCaBEj83FQJGCBrsSOzYDhas8aEDa4W9kKLbKeBaeRUE0mXQYfvu >>>> 12lZxpzn0UasrH/mcgu8ij9ElLN5Fq0wSp1SNKbg/RczcYVt/DjjGbCRDTgO >>>> b23I >>>> =1NQh >>>> -----END PGP SIGNATURE----- >>> >>> >>> >> -----BEGIN PGP SIGNATURE----- >> Version: Mailvelope v0.13.1 >> Comment: https://www.mailvelope.com >> >> wsFcBAEBCAAQBQJVuUnoCRDmVDuy+mK58QAAM3MP/jcJUwKHRUjI47y+S1St >> 5Tmkpmob0CdaplzdTvTYF9nc0U6GxhEwTxs/e6CuJ8raKKCNdV2Bu303kQQz >> t9xAJXbzUe4w5/f3WwlM91R3JavcqRVKyiVptzRf3EuHMCWUmLhWKlOK6WNF >> fcO3yRRyn5gkGHR2w7MAAVvABSyZsrLSvCU8Ckila3/cdZNRkOUCfiwSfYlL >> TXd1fsOVP629ltvl5xFQ2u4Tmx+hHq+oxx9PesWz8eKbXwM0Cbq84WfwojlQ >> GHL9DZlCOSGwsinEHQycODGmkL7QSfEBYEZYbMgtDmUbUOXLfUifvKvuk7yD >> I1QxzjhWLxa0k4k/8YzP8055euFTIzZ3E4g3jG50KzvxIaUhHuiGT0CXEtEi >> vEXg1iGtY3lhNjoMNMjQcSbi/PlbBL+08Y6y/VboXu4Tk6jm62g1gJvBHhUE >> OuE3Yiw1wQZI8LjGpCHQOt/ck3aFotsQI73hZF2DwANFeEKKcZvi+Pj0QsdC >> XVTCjI4qM4k36HjJG3tT/94Y6PbRVyeK6bLCa9klmx/u4ToWUjgD5e7Nc6Pv >> OcIkUfG+U6v5TjMAsGzEu1ZjQWNdpEGPoRr+cqwBYNlZu8QUacIAZ86jIyPe >> 77lrKrRbiYsdaYzZ48jrBXM+RiJY/sHgWnAr/Y12ysGbel3LVqxp8gGfI6zg >> beQf >> =gokk >> -----END PGP SIGNATURE----- > > > -----BEGIN PGP SIGNATURE----- Version: Mailvelope v0.13.1 Comment: https://www.mailvelope.com wsFcBAEBCAAQBQJVuV6VCRDmVDuy+mK58QAAa48QAIf3BuhUZ0p4zyKGOzeC 3gyYhH8RuS8ucb06aqyIDvvd9qecSkEXBlZdCt6qlykWLnI5ZgWt3mMQOuPl rHM8KarfDesiKJIb6tITm3L68z1Vr0XsccWIv+MndWyi6LKX03KCFJkJq0z4 H/fev0M8mchIAYFdvZpMX3ppB4r69XP9ExoGN4fteJwKN0iN8dPTHNFylcdw JjxMuL8z1SmjS83hefKJcTgH/ydfn88xS0k0mnYY7fCHCwcCEf/aLOrSPd/a vPoKHJSN2BOEer9YkPcK8zePeTlISk8JO/S2J1nuvVUwbgzRY4Ivnw+JqFt3 2Q8vw6ACA5zfHi5vmtm+B3t4uOxNxqIxJLK0ToMqg9K44AaWXn4ypbPJsIjy BetE6zQru1kKiHhoAeVFK+jn3ryKOaB/vkKtexlUdsodTYddS90YfPp5PTJC IkpRh3A2G1EpSSYE3EEo1JTs/J2FeMUDe8DlIMMBZ361ZqaEEcfs3TALduN7 Jg06Xsru1mmTGVorkikmXc9yeGyELV1gDyIOGrLtLZjTzWXX8QGefRIsTRGB RSb9+wipWGieu4jH5tf1uETIx9d/13YCu1V/hl3Anei9TGfqt08v/UlcesdV TIYyQYXzG5RoZY3AkNw7u3DFWVKVWj2mez4NhEgqjeXUc2VjmPWqX1O0EHkO j0d4 =grn/ -----END PGP SIGNATURE----- _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |