Hello, I seem to vaguely remember a Ceph leveldb package, which might help in this case, or something from the CentOS equivalent to backports maybe. Christian On Wed, 9 Dec 2015 22:18:56 -0700 Robert LeBlanc wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA256 > > I had this problem because CentOS and Debian have different versions > of leveldb (Debian's was newer) and the old version would not read the > new version. I just had to blow away the OSDs and let them backfill. > Going from CentOS to Debian didn't require it, but going back required > the backfill. > - ---------------- > Robert LeBlanc > PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1 > > > On Wed, Dec 9, 2015 at 5:25 AM, Jacek Jarosiewicz wrote: > > Hi, > > > > I have a working ceph cluster with storage nodes running Ubuntu 14.04 > > and ceph hammer 0.94.5. > > > > Now I want to switch to CentOS 7.1 (forget about the reasons for now, > > I can explain, but it would be a long story and irrelevant to my > > question). > > > > I've set the osd noout flag and norebalance,norecover for the time of > > reinstall. The new system is installed with the same version of ceph. > > > > I've made a backup of /var/lib/ceph directory (after stopping ceph > > services obviously) and kept the osd's intact. > > > > But after reinstall, when I try to start the daemons (the machine run > > one monitor and three osd's) I get these messages in the logs: > > > > monitor: > > 2015-12-09 13:15:28.223872 7f4ccd41b880 0 ceph version 0.94.5 > > (9764da52395923e0b32908d83a9f7304401fee43), process ceph-mon, pid 5800 > > 2015-12-09 13:15:29.411448 7f4ccd41b880 -1 error opening mon data > > directory at '/var/lib/ceph/mon/ceph-cf03': (22) Invalid argument > > > > > > osds: > > > > 2015-12-09 13:11:50.480625 7fac03c7f880 0 ceph version 0.94.5 > > (9764da52395923e0b32908d83a9f7304401fee43), process ceph-osd, pid 3092 > > 2015-12-09 13:11:50.508803 7fac03c7f880 0 > > filestore(/var/lib/ceph/osd/ceph-5) backend xfs (magic 0x58465342) > > 2015-12-09 13:11:50.640410 7fac03c7f880 0 > > genericfilestorebackend(/var/lib/ceph/osd/ceph-5) detect_features: > > FIEMAP ioctl is supported and appears to work > > 2015-12-09 13:11:50.640429 7fac03c7f880 0 > > genericfilestorebackend(/var/lib/ceph/osd/ceph-5) detect_features: > > FIEMAP ioctl is disabled via 'filestore fiemap' config option > > 2015-12-09 13:11:50.640890 7fac03c7f880 0 > > genericfilestorebackend(/var/lib/ceph/osd/ceph-5) detect_features: > > syncfs(2) syscall fully supported (by glibc and kernel) > > 2015-12-09 13:11:50.646915 7fac03c7f880 0 > > xfsfilestorebackend(/var/lib/ceph/osd/ceph-5) detect_feature: extsize > > is supported and kernel 3.10.0-229.20.1.el7.x86_64 >= 3.5 > > 2015-12-09 13:11:51.171377 7fac03c7f880 -1 > > filestore(/var/lib/ceph/osd/ceph-5) Error initializing leveldb : > > Corruption: 29 missing files; > > e.g.: /var/lib/ceph/osd/ceph-5/current/omap/046388.sst > > > > 2015-12-09 13:11:51.171399 7fac03c7f880 -1 osd.5 0 OSD:init: unable to > > mount object store > > 2015-12-09 13:11:51.171404 7fac03c7f880 -1 ** ERROR: osd init failed: > > (1) Operation not permitted > > > > > > can anyone help? I don't see any sst files on any of my other > > (working) ceph nodes, the directories are fine with correct > > permissions.. > > > > I can readd this machine from scratch without data loss, but the > > rebalancing/recovering will last a week (been there, done that), so I > > was hoping I could start the osds with only some data out of date. > > > > Is it possible? What can I do? > > > > Cheers, > > J > > > > -- > > Jacek Jarosiewicz > > Administrator Systemów Informatycznych > > > > ---------------------------------------------------------------------------------------- > > SUPERMEDIA Sp. z o.o. z siedzibą w Warszawie > > ul. Senatorska 13/15, 00-075 Warszawa > > Sąd Rejonowy dla m.st.Warszawy, XII Wydział Gospodarczy Krajowego > > Rejestru Sądowego, > > nr KRS 0000029537; kapitał zakładowy 42.756.000 zł > > NIP: 957-05-49-503 > > Adres korespondencyjny: ul. Jubilerska 10, 04-190 Warszawa > > > > ---------------------------------------------------------------------------------------- > > SUPERMEDIA -> http://www.supermedia.pl > > dostep do internetu - hosting - kolokacja - lacza - telefonia > > _______________________________________________ > > ceph-users mailing list > > ceph-users@xxxxxxxxxxxxxx > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > -----BEGIN PGP SIGNATURE----- > Version: Mailvelope v1.3.2 > Comment: https://www.mailvelope.com > > wsFcBAEBCAAQBQJWaQs6CRDmVDuy+mK58QAAoz8P/3wg1VTehoNaE2+cp1lB > JDPG5g4uwh68kJ96wBuiKVjoabEut1QiQkaM+5Ne+gkt7ZJPUqdF8UEdhTZ3 > 5JV7Xn+3pqjNeAu1R91aF40xp0RAXlhSC6MjfU7GEl0KSvYbLomoR4nyGxrP > //woszrmbau80f7A5Of0T2ILwx77FJbokOdGSsjL711LDjuqo/nXa/eLzYN1 > jtEC/pwqkbSBYR6avi8ZxqQZoMmPHeXxaTSK4dQyY7l6fWyBk0LB6MZYYaVl > 8T96wI8uMFBGPu13OYysuq6qrpJ/Cc0YglcmTqBpIOCjLHsZMUMtEgcvLvIB > m9qobcoxoqyTxjpa5EOhTV+7Qy9qPRv4vjyS0dcrCHUHwNP9QQOmbZIfqskd > W+ENtNK2vBLN+CVd6rXt47RP2+oP5/rIIqSW6KX+LyiCy3FDrn3+C7BFT1re > gyldiVz7KATwJX2/6Wsn2j942x62eZRS+szMqUemTsBJ8gAuaEIhHwJJd5Et > 2wKirSOFB0eVtvpuOC5VkmrsRTpwpV9ZHeWbjp7Vf0XrrWyWFwK1a4fydmCd > M0EwhAJjQiSUJL3LTdfkSBMtoJcZzwM9K4eZwSkqgu9blYQM9OvzsNI8Ssk2 > C7oXMHGgInqOeShQ3qGzKINo8rbirXwnvWkBL7g1We5s2nJ3acrlWRaTDdnA > kSpX > =+Zg+ > -----END PGP SIGNATURE----- > _______________________________________________ > ceph-users mailing list > ceph-users@xxxxxxxxxxxxxx > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com -- Christian Balzer Network/Systems Engineer chibi@xxxxxxx Global OnLine Japan/Fusion Communications http://www.gol.com/ _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |