On Fri, 11 Sep 2015, ?? wrote: > Thank Sage Weil: > > 1. I delete some testing pools in the past, but is was a long time ago (may be 2 months ago), in recently upgrade, do not delete pools. > 2. ceph osd dump please see the (attachment file ceph.osd.dump.log) > 3. debug osd = 20' and 'debug filestore = 20 (attachment file ceph.osd.5.log.tar.gz) This one is failing on pool 54, which has been deleted. In this case you can work around it by renaming current/54.* out of the way. > 4. i install the ceph-test, but output error > ceph-kvstore-tool /ceph/data5/current/db list > Invalid argument: /ceph/data5/current/db: does not exist (create_if_missing is false) Sorry, I should have said current/omap, not current/db. I'm still curious to see the key dump. I'm not sure why the leveldb key for these pgs is missing... Thanks! sage > > ls -l /ceph/data5/current/db > total 0 > -rw-r--r-- 1 root root 0 Sep 11 09:41 LOCK > -rw-r--r-- 1 root root 0 Sep 11 09:54 LOG > -rw-r--r-- 1 root root 0 Sep 11 09:54 LOG.old > > Thanks very much! > Wang Rui > > ------------------ Original ------------------ > From: "Sage Weil"<sage@xxxxxxxxxxxx>; > Date: Fri, Sep 11, 2015 06:23 AM > To: "??"<wangrui@xxxxxxxxxxxx>; > Cc: "ceph-devel"<ceph-devel@xxxxxxxxxxxxxxx>; > Subject: Re: Failed on starting osd-daemon after upgrade giant-0.87.1 tohammer-0.94.3 > > Hi! > > On Wed, 9 Sep 2015, ?? wrote: > > Hi all: > > > > I got on error after upgrade my ceph cluster from giant-0.87.2 to hammer-0.94.3, my local environment is: > > CentOS 6.7 x86_64 > > Kernel 3.10.86-1.el6.elrepo.x86_64 > > HDD: XFS, 2TB > > Install Package: ceph.com official RPMs x86_64 > > > > step 1: > > Upgrade MON server from 0.87.1 to 0.94.3, all is fine! > > > > step 2: > > Upgrade OSD server from 0.87.1 to 0.94.3. i just upgrade two servers and noticed that some osds can not started! > > server-1 have 4 osds, all of them can not started; > > server-2 have 3 osds, 2 of them can not started, but 1 of them successfully started and work in good. > > > > Error log 1: > > service ceph start osd.4 > > /var/log/ceph/ceph-osd.24.log > > (attachment file: ceph.24.log) > > > > Error log 2: > > /usr/bin/ceph-osd -c /etc/ceph/ceph.conf -i 4 -f > > (attachment file: cli.24.log) > > This looks a lot like a problem with a stray directory that older versions > did not clean up (#11429)... but not quite. Have you deleted pools in the > past? (Can you attach a 'ceph osd dump'?)? Also, i fyou start the osd > with 'debug osd = 20' and 'debug filestore = 20' we can see which PG is > problematic. If you install the 'ceph-test' package which contains > ceph-kvstore-tool, the output of > > ceph-kvstore-tool /var/lib/ceph/osd/ceph-$id/current/db list > > would also be helpful. > > Thanks! > sage -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
