On Wed, Nov 13, 2013 at 8:04 AM, Alfredo Deza <alfredo.deza@xxxxxxxxxxx> wrote: > On Tue, Nov 12, 2013 at 10:19 PM, Berant Lemmenes <berant@xxxxxxxxxxxx> wrote: >> >> On Tue, Nov 12, 2013 at 7:28 PM, Joao Eduardo Luis <joao.luis@xxxxxxxxxxx> >> wrote: >>> >>> >>> This looks an awful lot like you started another instance of an OSD with >>> the same ID while another was running. I'll walk you through the log lines >>> that point me towards this conclusion. Would still be weird if the admin >>> sockets vanished because of that, so maybe that's a different issue. Are >>> you able to reproduce the admin socket issue often? >>> >>> Walking through: >> >> >> Thanks for taking the time to walk through these logs, I appreciate the >> explanation. >> >>>> 2013-11-12 09:47:09.670813 7f8151b5f780 0 ceph version 0.72 >>>> (5832e2603c7db5d40b433d0953408993a9b7c217), process ceph-osd, pid 2769 >>>> 2013-11-12 09:47:09.673789 7f8151b5f780 0 >>>> filestore(/var/lib/ceph/osd/ceph-19) lock_fsid failed to lock >>>> /var/lib/ceph/osd/ceph-19/fsid, is another ceph-osd still running? (11) >>>> Resource temporarily unavailable >>> >>> >>> This last line tells us that ceph-osd believes another instance is >>> running, so you should first find out whether there's actually another >>> instance being run somewhere, somehow. How did you start these daemons? >> >> >> That proved to be the crux of it, both upstart and the Sys V init scripts >> were trying to start the ceph daemons. Looking in /etc/rc2.d there are >> symlinks from S20ceph to ../init.d/ceph >> >> Upstart thought it was controlling things - doing an 'initctl list | grep >> ceph' would show the correct PIDs, and 'service ceph status' thought they >> were not running. >> >> So that would seem to inidcate that sys V was trying to start it first, and >> upstart was the one that had started the instance that generated those logs. >> >> The part that doesn't make sense is that is if the Sys V init script was >> starting before upstart, why wouldn't it be the one that was writing to >> /var/log/ceph/xxxx? >> >> After running 'update-rc.d ceph disable', the admin sockets were present >> after a system reboot. >> >> I wonder, was the Sys V init scripts being enabled a ceph-deploy artifact or >> a issue with the packages? > > ceph-deploy will detect what type of script controller you should be > using depending on the > distribution it is deploying the OSD to. > > Specifically, it will use upstart for Ubuntu and sysvinit for anything else. > > ceph-deploy in turn will pass that information to ceph-disk-activate > and will output whatever it is doing > to the log files as well. > > Something like: "will use type: sysvinit" > > Do you have the ceph-deploy logs around? Oh, and I just noticed you did say Ubuntu, so sysvinit should have *not* been enabled at all, this should've been Upstart all along. Again, logs would be super useful! Hopefully you still have those around. >> >> Thanks for pointing me in the right direction! >> >> Thanks, >> Berant >> >> _______________________________________________ >> ceph-users mailing list >> ceph-users@xxxxxxxxxxxxxx >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |