On Tue, Nov 12, 2013 at 10:19 PM, Berant Lemmenes <berant@xxxxxxxxxxxx> wrote: > > On Tue, Nov 12, 2013 at 7:28 PM, Joao Eduardo Luis <joao.luis@xxxxxxxxxxx> > wrote: >> >> >> This looks an awful lot like you started another instance of an OSD with >> the same ID while another was running. I'll walk you through the log lines >> that point me towards this conclusion. Would still be weird if the admin >> sockets vanished because of that, so maybe that's a different issue. Are >> you able to reproduce the admin socket issue often? >> >> Walking through: > > > Thanks for taking the time to walk through these logs, I appreciate the > explanation. > >>> 2013-11-12 09:47:09.670813 7f8151b5f780 0 ceph version 0.72 >>> (5832e2603c7db5d40b433d0953408993a9b7c217), process ceph-osd, pid 2769 >>> 2013-11-12 09:47:09.673789 7f8151b5f780 0 >>> filestore(/var/lib/ceph/osd/ceph-19) lock_fsid failed to lock >>> /var/lib/ceph/osd/ceph-19/fsid, is another ceph-osd still running? (11) >>> Resource temporarily unavailable >> >> >> This last line tells us that ceph-osd believes another instance is >> running, so you should first find out whether there's actually another >> instance being run somewhere, somehow. How did you start these daemons? > > > That proved to be the crux of it, both upstart and the Sys V init scripts > were trying to start the ceph daemons. Looking in /etc/rc2.d there are > symlinks from S20ceph to ../init.d/ceph > > Upstart thought it was controlling things - doing an 'initctl list | grep > ceph' would show the correct PIDs, and 'service ceph status' thought they > were not running. > > So that would seem to inidcate that sys V was trying to start it first, and > upstart was the one that had started the instance that generated those logs. > > The part that doesn't make sense is that is if the Sys V init script was > starting before upstart, why wouldn't it be the one that was writing to > /var/log/ceph/xxxx? > > After running 'update-rc.d ceph disable', the admin sockets were present > after a system reboot. > > I wonder, was the Sys V init scripts being enabled a ceph-deploy artifact or > a issue with the packages? ceph-deploy will detect what type of script controller you should be using depending on the distribution it is deploying the OSD to. Specifically, it will use upstart for Ubuntu and sysvinit for anything else. ceph-deploy in turn will pass that information to ceph-disk-activate and will output whatever it is doing to the log files as well. Something like: "will use type: sysvinit" Do you have the ceph-deploy logs around? > > Thanks for pointing me in the right direction! > > Thanks, > Berant > > _______________________________________________ > ceph-users mailing list > ceph-users@xxxxxxxxxxxxxx > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |