-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 I had this problem because CentOS and Debian have different versions of leveldb (Debian's was newer) and the old version would not read the new version. I just had to blow away the OSDs and let them backfill. Going from CentOS to Debian didn't require it, but going back required the backfill. - ---------------- Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1 On Wed, Dec 9, 2015 at 5:25 AM, Jacek Jarosiewicz wrote: > Hi, > > I have a working ceph cluster with storage nodes running Ubuntu 14.04 and > ceph hammer 0.94.5. > > Now I want to switch to CentOS 7.1 (forget about the reasons for now, I can > explain, but it would be a long story and irrelevant to my question). > > I've set the osd noout flag and norebalance,norecover for the time of > reinstall. The new system is installed with the same version of ceph. > > I've made a backup of /var/lib/ceph directory (after stopping ceph services > obviously) and kept the osd's intact. > > But after reinstall, when I try to start the daemons (the machine run one > monitor and three osd's) I get these messages in the logs: > > monitor: > 2015-12-09 13:15:28.223872 7f4ccd41b880 0 ceph version 0.94.5 > (9764da52395923e0b32908d83a9f7304401fee43), process ceph-mon, pid 5800 > 2015-12-09 13:15:29.411448 7f4ccd41b880 -1 error opening mon data directory > at '/var/lib/ceph/mon/ceph-cf03': (22) Invalid argument > > > osds: > > 2015-12-09 13:11:50.480625 7fac03c7f880 0 ceph version 0.94.5 > (9764da52395923e0b32908d83a9f7304401fee43), process ceph-osd, pid 3092 > 2015-12-09 13:11:50.508803 7fac03c7f880 0 > filestore(/var/lib/ceph/osd/ceph-5) backend xfs (magic 0x58465342) > 2015-12-09 13:11:50.640410 7fac03c7f880 0 > genericfilestorebackend(/var/lib/ceph/osd/ceph-5) detect_features: FIEMAP > ioctl is supported and appears to work > 2015-12-09 13:11:50.640429 7fac03c7f880 0 > genericfilestorebackend(/var/lib/ceph/osd/ceph-5) detect_features: FIEMAP > ioctl is disabled via 'filestore fiemap' config option > 2015-12-09 13:11:50.640890 7fac03c7f880 0 > genericfilestorebackend(/var/lib/ceph/osd/ceph-5) detect_features: syncfs(2) > syscall fully supported (by glibc and kernel) > 2015-12-09 13:11:50.646915 7fac03c7f880 0 > xfsfilestorebackend(/var/lib/ceph/osd/ceph-5) detect_feature: extsize is > supported and kernel 3.10.0-229.20.1.el7.x86_64 >= 3.5 > 2015-12-09 13:11:51.171377 7fac03c7f880 -1 > filestore(/var/lib/ceph/osd/ceph-5) Error initializing leveldb : Corruption: > 29 missing files; e.g.: /var/lib/ceph/osd/ceph-5/current/omap/046388.sst > > 2015-12-09 13:11:51.171399 7fac03c7f880 -1 osd.5 0 OSD:init: unable to mount > object store > 2015-12-09 13:11:51.171404 7fac03c7f880 -1 ** ERROR: osd init failed: (1) > Operation not permitted > > > can anyone help? I don't see any sst files on any of my other (working) ceph > nodes, the directories are fine with correct permissions.. > > I can readd this machine from scratch without data loss, but the > rebalancing/recovering will last a week (been there, done that), so I was > hoping I could start the osds with only some data out of date. > > Is it possible? What can I do? > > Cheers, > J > > -- > Jacek Jarosiewicz > Administrator Systemów Informatycznych > > ---------------------------------------------------------------------------------------- > SUPERMEDIA Sp. z o.o. z siedzibą w Warszawie > ul. Senatorska 13/15, 00-075 Warszawa > Sąd Rejonowy dla m.st.Warszawy, XII Wydział Gospodarczy Krajowego Rejestru > Sądowego, > nr KRS 0000029537; kapitał zakładowy 42.756.000 zł > NIP: 957-05-49-503 > Adres korespondencyjny: ul. Jubilerska 10, 04-190 Warszawa > > ---------------------------------------------------------------------------------------- > SUPERMEDIA -> http://www.supermedia.pl > dostep do internetu - hosting - kolokacja - lacza - telefonia > _______________________________________________ > ceph-users mailing list > ceph-users@xxxxxxxxxxxxxx > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com -----BEGIN PGP SIGNATURE----- Version: Mailvelope v1.3.2 Comment: https://www.mailvelope.com wsFcBAEBCAAQBQJWaQs6CRDmVDuy+mK58QAAoz8P/3wg1VTehoNaE2+cp1lB JDPG5g4uwh68kJ96wBuiKVjoabEut1QiQkaM+5Ne+gkt7ZJPUqdF8UEdhTZ3 5JV7Xn+3pqjNeAu1R91aF40xp0RAXlhSC6MjfU7GEl0KSvYbLomoR4nyGxrP //woszrmbau80f7A5Of0T2ILwx77FJbokOdGSsjL711LDjuqo/nXa/eLzYN1 jtEC/pwqkbSBYR6avi8ZxqQZoMmPHeXxaTSK4dQyY7l6fWyBk0LB6MZYYaVl 8T96wI8uMFBGPu13OYysuq6qrpJ/Cc0YglcmTqBpIOCjLHsZMUMtEgcvLvIB m9qobcoxoqyTxjpa5EOhTV+7Qy9qPRv4vjyS0dcrCHUHwNP9QQOmbZIfqskd W+ENtNK2vBLN+CVd6rXt47RP2+oP5/rIIqSW6KX+LyiCy3FDrn3+C7BFT1re gyldiVz7KATwJX2/6Wsn2j942x62eZRS+szMqUemTsBJ8gAuaEIhHwJJd5Et 2wKirSOFB0eVtvpuOC5VkmrsRTpwpV9ZHeWbjp7Vf0XrrWyWFwK1a4fydmCd M0EwhAJjQiSUJL3LTdfkSBMtoJcZzwM9K4eZwSkqgu9blYQM9OvzsNI8Ssk2 C7oXMHGgInqOeShQ3qGzKINo8rbirXwnvWkBL7g1We5s2nJ3acrlWRaTDdnA kSpX =+Zg+ -----END PGP SIGNATURE----- _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |