Hi Upendra, Based on the output you posted, init-ceph is doing something on every node. However, I only see a kill for certain nodes. Does /var/run/ceph/ exist on all nodes, or just some of them? Does the appropriate pid file exist on all nodes? What happens when you ssh in to those nodes manually and run init-ceph stop? Colin On Fri, Feb 18, 2011 at 6:11 AM, Upendra Moturi <upendra.m@xxxxxxxxxxxx> wrote: > Hi Colin > > I am using the ubuntu 11.04 (32 bit) and got the ceph package from apt-get > i am using the default init script.(Found at /etc/init.d/ceph) > > Regarding issue 1) > > On the osd node I tried to start that osd but it did not work > Steps followed > > 1) Started 3 nodes(ceph.conf is same as i sent earlier) > > 2) mkcephfs -c /etc/ceph/ceph.conf -a --mkbtrfs -k /etc/ceph/keyring.bin > > 3)/etc/init.d/ceph start osd0 (Did this on first osd) > > 4) ps -ef | grep ceph or ps -ef | grep cosd ---- does not show any process > > 5)/etc/init.d/ceph -a start ---- shows all process on all nodes > > Regading Issue 2 > It says -x option is not available > > Tried /etc/init.d/ceph -ax stop and /etc/init.d/ceph -x stop and even > tried /etc/init.d/ceph -x -a stop > but nothing worked. > > Then tried with /etc/init.d/ceph -a -v stop this also did not stop > ceph on all nodes but got the output as > > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "auto start" "" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "user" "" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "ssh path" "/etc/ceph" > === mon.0 === > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "pid file" > "/var/run/ceph/mon.0.pid" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "log dir" "/var/log/ceph" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "log sym dir" "" > --- ssh ceph0 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "lock file" > "/var/lock/subsys/ceph" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "pre stop command" "" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "post stop command" "" > Stopping Ceph mon.0 on ceph0...--- ssh ceph0 "cd /etc/ceph ; ulimit > -c unlimited ; while [ 1 ]; do > [ -e /var/run/ceph/mon.0.pid ] || break > pid=`cat /var/run/ceph/mon.0.pid` > while [ -e /proc/$pid ] && grep -q cmon /proc/$pid/cmdline ; do > cmd="kill $pid" > echo -n $cmd... > $cmd > sleep 1 > continue > done > break > done" > kill 1668...done > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "auto start" "" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "user" "" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "ssh path" "/etc/ceph" > === mon.1 === > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "pid file" > "/var/run/ceph/mon.1.pid" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "log dir" "/var/log/ceph" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "log sym dir" "" > --- ssh ceph1 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "lock file" > "/var/lock/subsys/ceph" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "pre stop command" "" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "post stop command" "" > Stopping Ceph mon.1 on ceph1...--- ssh ceph1 "cd /etc/ceph ; ulimit > -c unlimited ; while [ 1 ]; do > [ -e /var/run/ceph/mon.1.pid ] || break > pid=`cat /var/run/ceph/mon.1.pid` > while [ -e /proc/$pid ] && grep -q cmon /proc/$pid/cmdline ; do > cmd="kill $pid" > echo -n $cmd... > $cmd > sleep 1 > continue > done > break > done" > done > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "auto start" "" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "user" "" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "ssh path" "/etc/ceph" > === mon.2 === > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "pid file" > "/var/run/ceph/mon.2.pid" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "log dir" "/var/log/ceph" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "log sym dir" "" > --- ssh ceph2 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "lock file" > "/var/lock/subsys/ceph" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "pre stop command" "" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "post stop command" "" > Stopping Ceph mon.2 on ceph2...--- ssh ceph2 "cd /etc/ceph ; ulimit > -c unlimited ; while [ 1 ]; do > [ -e /var/run/ceph/mon.2.pid ] || break > pid=`cat /var/run/ceph/mon.2.pid` > while [ -e /proc/$pid ] && grep -q cmon /proc/$pid/cmdline ; do > cmd="kill $pid" > echo -n $cmd... > $cmd > sleep 1 > continue > done > break > done" > done > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "auto start" "" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "user" "" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "ssh path" "/etc/ceph" > === mds.0 === > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "pid file" > "/var/run/ceph/mds.0.pid" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "log dir" "/var/log/ceph" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "log sym dir" "" > --- ssh ceph0 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "lock file" > "/var/lock/subsys/ceph" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "pre stop command" "" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "post stop command" "" > Stopping Ceph mds0 on ceph0...--- ssh ceph0 "cd /etc/ceph ; ulimit -c > unlimited ; while [ 1 ]; do > [ -e /var/run/ceph/mds.0.pid ] || break > pid=`cat /var/run/ceph/mds.0.pid` > while [ -e /proc/$pid ] && grep -q cmds /proc/$pid/cmdline ; do > cmd="kill $pid" > echo -n $cmd... > $cmd > sleep 1 > continue > done > break > done" > kill 1844...done > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "auto start" "" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "user" "" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "ssh path" "/etc/ceph" > === mds.1 === > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "pid file" > "/var/run/ceph/mds.1.pid" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "log dir" "/var/log/ceph" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "log sym dir" "" > --- ssh ceph1 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "lock file" > "/var/lock/subsys/ceph" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "pre stop command" "" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "post stop command" "" > Stopping Ceph mds1 on ceph1...--- ssh ceph1 "cd /etc/ceph ; ulimit -c > unlimited ; while [ 1 ]; do > [ -e /var/run/ceph/mds.1.pid ] || break > pid=`cat /var/run/ceph/mds.1.pid` > while [ -e /proc/$pid ] && grep -q cmds /proc/$pid/cmdline ; do > cmd="kill $pid" > echo -n $cmd... > $cmd > sleep 1 > continue > done > break > done" > done > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "auto start" "" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "user" "" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "ssh path" "/etc/ceph" > === osd.0 === > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "pid file" > "/var/run/ceph/osd.0.pid" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "log dir" "/var/log/ceph" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "log sym dir" "" > --- ssh ceph0 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "osd data" "" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "btrfs path" "/data/osd0" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "btrfs devs" "" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "lock file" > "/var/lock/subsys/ceph" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "pre stop command" "" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "post stop command" "" > Stopping Ceph osd0 on ceph0...--- ssh ceph0 "cd /etc/ceph ; ulimit -c > unlimited ; while [ 1 ]; do > [ -e /var/run/ceph/osd.0.pid ] || break > pid=`cat /var/run/ceph/osd.0.pid` > while [ -e /proc/$pid ] && grep -q cosd /proc/$pid/cmdline ; do > cmd="kill $pid" > echo -n $cmd... > $cmd > sleep 1 > continue > done > break > done" > kill 2033...done > Unmounting Btrfs on ceph0:/data/osd0 > --- ssh root@ceph0 "cd /etc/ceph ; ulimit -c unlimited ; umount > /data/osd0 || true" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "auto start" "" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "user" "" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "ssh path" "/etc/ceph" > === osd.1 === > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "pid file" > "/var/run/ceph/osd.1.pid" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "log dir" "/var/log/ceph" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "log sym dir" "" > --- ssh ceph1 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "osd data" "" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "btrfs path" "/data/osd1" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "btrfs devs" "" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "lock file" > "/var/lock/subsys/ceph" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "pre stop command" "" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "post stop command" "" > Stopping Ceph osd1 on ceph1...--- ssh ceph1 "cd /etc/ceph ; ulimit -c > unlimited ; while [ 1 ]; do > [ -e /var/run/ceph/osd.1.pid ] || break > pid=`cat /var/run/ceph/osd.1.pid` > while [ -e /proc/$pid ] && grep -q cosd /proc/$pid/cmdline ; do > cmd="kill $pid" > echo -n $cmd... > $cmd > sleep 1 > continue > done > break > done" > done > Unmounting Btrfs on ceph1:/data/osd1 > --- ssh root@ceph1 "cd /etc/ceph ; ulimit -c unlimited ; umount > /data/osd1 || true" > umount: /data/osd1: device is busy. > (In some cases useful info about processes that use > the device is found by lsof(8) or fuser(1)) > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "auto start" "" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "user" "" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "ssh path" "/etc/ceph" > === osd.2 === > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "pid file" > "/var/run/ceph/osd.2.pid" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "log dir" "/var/log/ceph" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "log sym dir" "" > --- ssh ceph2 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "osd data" "" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "btrfs path" "/data/osd2" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "btrfs devs" "" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "lock file" > "/var/lock/subsys/ceph" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "pre stop command" "" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "post stop command" "" > Stopping Ceph osd2 on ceph2...--- ssh ceph2 "cd /etc/ceph ; ulimit -c > unlimited ; while [ 1 ]; do > [ -e /var/run/ceph/osd.2.pid ] || break > pid=`cat /var/run/ceph/osd.2.pid` > while [ -e /proc/$pid ] && grep -q cosd /proc/$pid/cmdline ; do > cmd="kill $pid" > echo -n $cmd... > $cmd > sleep 1 > continue > done > break > done" > done > Unmounting Btrfs on ceph2:/data/osd2 > --- ssh root@ceph2 "cd /etc/ceph ; ulimit -c unlimited ; umount > /data/osd2 || true" > umount: /data/osd2: device is busy. > (In some cases useful info about processes that use > the device is found by lsof(8) or fuser(1)) > > > > > On Fri, Feb 18, 2011 at 12:35 PM, Colin McCabe <cmccabe@xxxxxxxxxxxxxx> wrote: >> Hi Upendra, >> >> Are you running init-ceph from the source directory? If you do that, >> it will use the ceph.conf in the source directory itself, which is >> probably not what you want. So it might be good to double-check that. >> >> If all else fails, running init-ceph with -x will show you exactly >> what the script is doing. If all goes well, its exit status should be >> 0. Are you getting exit status 0? >> >> Colin >> >> >> On Thu, Feb 17, 2011 at 1:55 AM, Upendra Moturi <upendra.m@xxxxxxxxxxxx> wrote: >>> Hi Colin >>> >>> Here is my ceph.conf : >>> >>> [global] >>> pid file = /var/run/ceph/$name.pid >>> debug ms = 1 >>> [mon] >>> mon data = /data/mon$id >>> [mon.0] >>> host = ceph0 >>> mon addr = 192.168.155.5:6789 >>> [mon.1] >>> host = ceph1 >>> mon addr = 192.168.155.6:6789 >>> [mon.2] >>> host = ceph2 >>> mon addr = 192.168.155.7:6789 >>> [mds] >>> >>> [mds0] >>> host = ceph0 >>> [mds1] >>> host = ceph1 >>> >>> [osd] >>> sudo = true >>> osd data = /data/osd$id >>> osd journal = /data/osd$id/journal >>> osd journal size = 512 >>> osd use stale snap = true >>> [osd0] >>> host = ceph0 >>> btrfs devs = /dev/sdb >>> [osd1] >>> host = ceph1 >>> btrfs devs = /dev/sdb >>> [osd2] >>> host = ceph2 >>> btrfs devs = /dev/sdb >>> >>> >>> >>> >>> On Thu, Feb 17, 2011 at 1:06 PM, Colin McCabe <cmccabe@xxxxxxxxxxxxxx> wrote: >>>> I'm using head of line from the master branch. But that particular >>>> code hasn't changed since January, which is 0.24.2 is from. >>>> >>>> In my ceph.conf, I just had an osd that was on a remote machine, and >>>> everything else local. >>>> >>>> If you could post your ceph.conf here or in IRC, perhaps we might spot >>>> an issue that's causing the problems that you see. >>>> >>>> Colin >>>> >>>> >>>> On Wed, Feb 16, 2011 at 11:01 PM, Upendra Moturi <upendra.m@xxxxxxxxxxxx> wrote: >>>>> Hi Colin >>>>> I am using >>>>> ceph version 0.24.2 commit:f7572de5cb87eb7157217be4975ae66d90831bb7 >>>>> ubuntu 11.04 32 bit with upgraded kernal of 2.6.38-2-generic >>>>> >>>>> Installed ceph form apt source. >>>>> >>>>> With above configurations i still able yo reproduce. >>>>> Can you please share me ur configurations? >>>>> >>>>> On Thu, Feb 17, 2011 at 3:31 AM, Colin McCabe <cmccabe@xxxxxxxxxxxxxx> wrote: >>>>>> On Wed, Feb 16, 2011 at 1:41 PM, Colin McCabe <cmccabe@xxxxxxxxxxxxxx> wrote: >>>>>>> On Wed, Feb 16, 2011 at 6:44 AM, Upendra Moturi <upendra.m@xxxxxxxxxxxx> wrote: >>>>>>>> But if we want to start a particular osd or mon or mds ,its not >>>>>>>> working and there is no error >>>>>>>> eg:/etc/init.d/ceph start osd1 does not start osd1 and don't get any error >>>>>>> >>>>>>> That is expected, unless you are running init-ceph on the same node as >>>>>>> osd1 is on. >>>>>>> >>>>>>> It might be nice to have some kind of interface like "run command X on >>>>>>> osd1", but init-ceph is not that. >>>>>>> >>>>>>>> /etc/init.d/ceph -a stop also does not stop ceph on all nodes.It stops >>>>>>>> on current node only >>>>>>>> where as >>>>>>>> /etc/init.d/ceph -a killall works fine. >>>>>>> >>>>>>> That sounds like a bug. I'll see if I can fix it. >>>>>> >>>>>> I'm afraid I can't reproduce this. >>>>>> >>>>>> I ran /etc/init.d/ceph -a stop >>>>>> >>>>>> and it stopped ceph daemons running on remote nodes too. Looking at >>>>>> the code, it looks correct. >>>>>> >>>>>> Colin >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> Thanks and Regards, >>>>> Upendra.M >>>>> >>>> >>> >>> >>> >>> -- >>> Thanks and Regards, >>> Upendra.M >>> >> > > > > -- > Thanks and Regards, > Upendra.M > -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
