On Tue, Nov 12, 2013 at 7:28 PM, Joao Eduardo Luis <joao.luis@xxxxxxxxxxx> wrote:
This looks an awful lot like you started another instance of an OSD with the same ID while another was running. I'll walk you through the log lines that point me towards this conclusion. Would still be weird if the admin sockets vanished because of that, so maybe that's a different issue. Are you able to reproduce the admin socket issue often?
Walking through:
Thanks for taking the time to walk through these logs, I appreciate the explanation.
This last line tells us that ceph-osd believes another instance is running, so you should first find out whether there's actually another instance being run somewhere, somehow. How did you start these daemons?2013-11-12 09:47:09.670813 7f8151b5f780 0 ceph version 0.72
(5832e2603c7db5d40b433d0953408993a9b7c217), process ceph-osd, pid 2769
2013-11-12 09:47:09.673789 7f8151b5f780 0
filestore(/var/lib/ceph/osd/ceph-19) lock_fsid failed to lock
/var/lib/ceph/osd/ceph-19/fsid, is another ceph-osd still running? (11)
Resource temporarily unavailable
That proved to be the crux of it, both upstart and the Sys V init scripts were trying to start the ceph daemons. Looking in /etc/rc2.d there are symlinks from S20ceph to ../init.d/ceph
Upstart thought it was controlling things - doing an 'initctl list | grep ceph' would show the correct PIDs, and 'service ceph status' thought they were not running.
So that would seem to inidcate that sys V was trying to start it first, and upstart was the one that had started the instance that generated those logs.
The part that doesn't make sense is that is if the Sys V init script was starting before upstart, why wouldn't it be the one that was writing to /var/log/ceph/xxxx?
After running 'update-rc.d ceph disable', the admin sockets were present after a system reboot.
I wonder, was the Sys V init scripts being enabled a ceph-deploy artifact or a issue with the packages?
Thanks for pointing me in the right direction!
Thanks,
Berant
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
