It was a permission issue. While I followed the process and haven't changed any data. The "current map" files on each OSD were still listed as owner root as they were created while the older ceph processes were still running.
Changing that after the fact was still a necessity and I will make sure that those are also properly changed.
On Mon, Jun 6, 2016 at 12:12 PM Samuel Just <sjust@xxxxxxxxxx> wrote:
Oh, what was the problem (for posterity)?
-Sam
On Mon, Jun 6, 2016 at 12:11 PM, Tu Holmes <tu.holmes@xxxxxxxxx> wrote:
> It totally did and I see what the problem is.
>
> Thanks for your input. I truly appreciate it.
>
>
> On Mon, Jun 6, 2016 at 12:01 PM Samuel Just <sjust@xxxxxxxxxx> wrote:
>>
>> If you reproduce with
>>
>> debug osd = 20
>> debug filestore = 20
>> debug ms = 1
>>
>> that might make it clearer what is going on.
>> -Sam
>>
>> On Mon, Jun 6, 2016 at 11:53 AM, Tu Holmes <tu.holmes@xxxxxxxxx> wrote:
>> > Hey cephers. I have been following the upgrade documents and I have done
>> > everything regarding upgrading the client to the latest version of
>> > Hammer,
>> > then to Jewel.
>> >
>> > I made sure that the owner of log partitions and all other items is the
>> > ceph
>> > user and I've gone through the process as was described in the
>> > documents,
>> > but I am getting this on my nodes as I upgrade.
>> >
>> >
>> > --- begin dump of recent events ---
>> >
>> > -72> 2016-06-06 11:49:37.720315 7f09d0152800 5 asok(0x7f09dbb78280)
>> > register_command perfcounters_dump hook 0x7f09dbb58050
>> >
>> > -71> 2016-06-06 11:49:37.720328 7f09d0152800 5 asok(0x7f09dbb78280)
>> > register_command 1 hook 0x7f09dbb58050
>> >
>> > -70> 2016-06-06 11:49:37.720330 7f09d0152800 5 asok(0x7f09dbb78280)
>> > register_command perf dump hook 0x7f09dbb58050
>> >
>> > -69> 2016-06-06 11:49:37.720332 7f09d0152800 5 asok(0x7f09dbb78280)
>> > register_command perfcounters_schema hook 0x7f09dbb58050
>> >
>> > -68> 2016-06-06 11:49:37.720333 7f09d0152800 5 asok(0x7f09dbb78280)
>> > register_command 2 hook 0x7f09dbb58050
>> >
>> > -67> 2016-06-06 11:49:37.720334 7f09d0152800 5 asok(0x7f09dbb78280)
>> > register_command perf schema hook 0x7f09dbb58050
>> >
>> > -66> 2016-06-06 11:49:37.720335 7f09d0152800 5 asok(0x7f09dbb78280)
>> > register_command perf reset hook 0x7f09dbb58050
>> >
>> > -65> 2016-06-06 11:49:37.720337 7f09d0152800 5 asok(0x7f09dbb78280)
>> > register_command config show hook 0x7f09dbb58050
>> >
>> > -64> 2016-06-06 11:49:37.720338 7f09d0152800 5 asok(0x7f09dbb78280)
>> > register_command config set hook 0x7f09dbb58050
>> >
>> > -63> 2016-06-06 11:49:37.720339 7f09d0152800 5 asok(0x7f09dbb78280)
>> > register_command config get hook 0x7f09dbb58050
>> >
>> > -62> 2016-06-06 11:49:37.720340 7f09d0152800 5 asok(0x7f09dbb78280)
>> > register_command config diff hook 0x7f09dbb58050
>> >
>> > -61> 2016-06-06 11:49:37.720342 7f09d0152800 5 asok(0x7f09dbb78280)
>> > register_command log flush hook 0x7f09dbb58050
>> >
>> > -60> 2016-06-06 11:49:37.720343 7f09d0152800 5 asok(0x7f09dbb78280)
>> > register_command log dump hook 0x7f09dbb58050
>> >
>> > -59> 2016-06-06 11:49:37.720344 7f09d0152800 5 asok(0x7f09dbb78280)
>> > register_command log reopen hook 0x7f09dbb58050
>> >
>> > -58> 2016-06-06 11:49:37.723459 7f09d0152800 0 set uid:gid to
>> > 1000:1000
>> > (ceph:ceph)
>> >
>> > -57> 2016-06-06 11:49:37.723476 7f09d0152800 0 ceph version 10.2.1
>> > (3a66dd4f30852819c1bdaa8ec23c795d4ad77269), process ceph-osd, pid 9943
>> >
>> > -56> 2016-06-06 11:49:37.727080 7f09d0152800 1 -- 10.253.50.213:0/0
>> > learned my addr 10.253.50.213:0/0
>> >
>> > -55> 2016-06-06 11:49:37.727092 7f09d0152800 1
>> > accepter.accepter.bind
>> > my_inst.addr is 10.253.50.213:6806/9943 need_addr=0
>> >
>> > -54> 2016-06-06 11:49:37.727104 7f09d0152800 1 -- 172.16.1.3:0/0
>> > learned
>> > my addr 172.16.1.3:0/0
>> >
>> > -53> 2016-06-06 11:49:37.727109 7f09d0152800 1
>> > accepter.accepter.bind
>> > my_inst.addr is 172.16.1.3:6806/9943 need_addr=0
>> >
>> > -52> 2016-06-06 11:49:37.727119 7f09d0152800 1 -- 172.16.1.3:0/0
>> > learned
>> > my addr 172.16.1.3:0/0
>> >
>> > -51> 2016-06-06 11:49:37.727129 7f09d0152800 1
>> > accepter.accepter.bind
>> > my_inst.addr is 172.16.1.3:6807/9943 need_addr=0
>> >
>> > -50> 2016-06-06 11:49:37.727139 7f09d0152800 1 -- 10.253.50.213:0/0
>> > learned my addr 10.253.50.213:0/0
>> >
>> > -49> 2016-06-06 11:49:37.727143 7f09d0152800 1
>> > accepter.accepter.bind
>> > my_inst.addr is 10.253.50.213:6807/9943 need_addr=0
>> >
>> > -48> 2016-06-06 11:49:37.727148 7f09d0152800 0 pidfile_write: ignore
>> > empty --pid-file
>> >
>> > -47> 2016-06-06 11:49:37.728364 7f09d0152800 5 asok(0x7f09dbb78280)
>> > init
>> > /var/run/ceph/ceph-osd.8.asok
>> >
>> > -46> 2016-06-06 11:49:37.728417 7f09d0152800 5 asok(0x7f09dbb78280)
>> > bind_and_listen /var/run/ceph/ceph-osd.8.asok
>> >
>> > -45> 2016-06-06 11:49:37.728472 7f09d0152800 5 asok(0x7f09dbb78280)
>> > register_command 0 hook 0x7f09dbb54110
>> >
>> > -44> 2016-06-06 11:49:37.728488 7f09d0152800 5 asok(0x7f09dbb78280)
>> > register_command version hook 0x7f09dbb54110
>> >
>> > -43> 2016-06-06 11:49:37.728493 7f09d0152800 5 asok(0x7f09dbb78280)
>> > register_command git_version hook 0x7f09dbb54110
>> >
>> > -42> 2016-06-06 11:49:37.728498 7f09d0152800 5 asok(0x7f09dbb78280)
>> > register_command help hook 0x7f09dbb58230
>> >
>> > -41> 2016-06-06 11:49:37.728502 7f09d0152800 5 asok(0x7f09dbb78280)
>> > register_command get_command_descriptions hook 0x7f09dbb58220
>> >
>> > -40> 2016-06-06 11:49:37.728544 7f09d0152800 10 monclient(hunting):
>> > build_initial_monmap
>> >
>> > -39> 2016-06-06 11:49:37.734765 7f09c9df5700 5 asok(0x7f09dbb78280)
>> > entry start
>> >
>> > -38> 2016-06-06 11:49:37.736541 7f09d0152800 5 adding auth protocol:
>> > cephx
>> >
>> > -37> 2016-06-06 11:49:37.736552 7f09d0152800 5 adding auth protocol:
>> > cephx
>> >
>> > -36> 2016-06-06 11:49:37.736672 7f09d0152800 5 asok(0x7f09dbb78280)
>> > register_command objecter_requests hook 0x7f09dbb58270
>> >
>> > -35> 2016-06-06 11:49:37.736710 7f09d0152800 1 --
>> > 10.253.50.213:6806/9943 messenger.start
>> >
>> > -34> 2016-06-06 11:49:37.736732 7f09d0152800 1 -- :/0
>> > messenger.start
>> >
>> > -33> 2016-06-06 11:49:37.736748 7f09d0152800 1 --
>> > 10.253.50.213:6807/9943 messenger.start
>> >
>> > -32> 2016-06-06 11:49:37.736763 7f09d0152800 1 --
>> > 172.16.1.3:6807/9943
>> > messenger.start
>> >
>> > -31> 2016-06-06 11:49:37.736774 7f09d0152800 1 --
>> > 172.16.1.3:6806/9943
>> > messenger.start
>> >
>> > -30> 2016-06-06 11:49:37.736786 7f09d0152800 1 -- :/0
>> > messenger.start
>> >
>> > -29> 2016-06-06 11:49:37.736821 7f09d0152800 2 osd.8 0 mounting
>> > /var/lib/ceph/osd/ceph-8 /var/lib/ceph/osd/ceph-8/journal
>> >
>> > -28> 2016-06-06 11:49:37.736864 7f09d0152800 0
>> > filestore(/var/lib/ceph/osd/ceph-8) backend xfs (magic 0x58465342)
>> >
>> > -27> 2016-06-06 11:49:37.737113 7f09d0152800 0
>> > genericfilestorebackend(/var/lib/ceph/osd/ceph-8) detect_features:
>> > FIEMAP
>> > ioctl is disabled via 'filestore fiemap' config option
>> >
>> > -26> 2016-06-06 11:49:37.737127 7f09d0152800 0
>> > genericfilestorebackend(/var/lib/ceph/osd/ceph-8) detect_features:
>> > SEEK_DATA/SEEK_HOLE is disabled via 'filestore seek data hole' config
>> > option
>> >
>> > -25> 2016-06-06 11:49:37.737144 7f09d0152800 0
>> > genericfilestorebackend(/var/lib/ceph/osd/ceph-8) detect_features:
>> > splice is
>> > supported
>> >
>> > -24> 2016-06-06 11:49:37.748618 7f09d0152800 0
>> > genericfilestorebackend(/var/lib/ceph/osd/ceph-8) detect_features:
>> > syncfs(2)
>> > syscall fully supported (by glibc and kernel)
>> >
>> > -23> 2016-06-06 11:49:37.748704 7f09d0152800 0
>> > xfsfilestorebackend(/var/lib/ceph/osd/ceph-8) detect_feature: extsize is
>> > disabled by conf
>> >
>> > -22> 2016-06-06 11:49:37.749225 7f09d0152800 1 leveldb: Recovering
>> > log
>> > #62532
>> >
>> > -21> 2016-06-06 11:49:37.837041 7f09d0152800 1 leveldb: Delete
>> > type=3
>> > #62531
>> >
>> >
>> > -20> 2016-06-06 11:49:37.837098 7f09d0152800 1 leveldb: Delete
>> > type=0
>> > #62532
>> >
>> >
>> > -19> 2016-06-06 11:49:37.837539 7f09d0152800 0
>> > filestore(/var/lib/ceph/osd/ceph-8) mount: enabling WRITEAHEAD journal
>> > mode:
>> > checkpoint is not enabled
>> >
>> > -18> 2016-06-06 11:49:37.839069 7f09d0152800 2 journal open
>> > /var/lib/ceph/osd/ceph-8/journal fsid
>> > 12375982-ca61-4170-93a0-03f1930ced83
>> > fs_op_seq 58991306
>> >
>> > -17> 2016-06-06 11:49:37.839106 7f09d0152800 1 journal _open
>> > /var/lib/ceph/osd/ceph-8/journal fd 18: 10737418240 bytes, block size
>> > 4096
>> > bytes, directio = 1, aio = 1
>> >
>> > -16> 2016-06-06 11:49:37.873359 7f09d0152800 2 journal No further
>> > valid
>> > entries found, journal is most likely valid
>> >
>> > -15> 2016-06-06 11:49:37.873378 7f09d0152800 2 journal No further
>> > valid
>> > entries found, journal is most likely valid
>> >
>> > -14> 2016-06-06 11:49:37.873382 7f09d0152800 3 journal
>> > journal_replay:
>> > end of journal, done.
>> >
>> > -13> 2016-06-06 11:49:37.873430 7f09d0152800 1 journal _open
>> > /var/lib/ceph/osd/ceph-8/journal fd 18: 10737418240 bytes, block size
>> > 4096
>> > bytes, directio = 1, aio = 1
>> >
>> > -12> 2016-06-06 11:49:37.896293 7f09d0152800 1
>> > filestore(/var/lib/ceph/osd/ceph-8) upgrade
>> >
>> > -11> 2016-06-06 11:49:37.896346 7f09d0152800 2 osd.8 0 boot
>> >
>> > -10> 2016-06-06 11:49:37.896597 7f09d0152800 1 <cls>
>> > cls/replica_log/cls_replica_log.cc:141: Loaded replica log class!
>> >
>> > -9> 2016-06-06 11:49:37.898335 7f09d0152800 1 <cls>
>> > cls/refcount/cls_refcount.cc:232: Loaded refcount class!
>> >
>> > -8> 2016-06-06 11:49:37.898423 7f09d0152800 1 <cls>
>> > cls/timeindex/cls_timeindex.cc:259: Loaded timeindex class!
>> >
>> > -7> 2016-06-06 11:49:37.898507 7f09d0152800 1 <cls>
>> > cls/statelog/cls_statelog.cc:306: Loaded log class!
>> >
>> > -6> 2016-06-06 11:49:37.898587 7f09d0152800 1 <cls>
>> > cls/log/cls_log.cc:317: Loaded log class!
>> >
>> > -5> 2016-06-06 11:49:37.908832 7f09d0152800 1 <cls>
>> > cls/user/cls_user.cc:375: Loaded user class!
>> >
>> > -4> 2016-06-06 11:49:37.910755 7f09d0152800 1 <cls>
>> > cls/rgw/cls_rgw.cc:3207: Loaded rgw class!
>> >
>> > -3> 2016-06-06 11:49:37.920661 7f09d0152800 0 <cls>
>> > cls/cephfs/cls_cephfs.cc:202: loading cephfs_size_scan
>> >
>> > -2> 2016-06-06 11:49:37.920774 7f09d0152800 0 <cls>
>> > cls/hello/cls_hello.cc:305: loading cls_hello
>> >
>> > -1> 2016-06-06 11:49:37.920926 7f09d0152800 1 <cls>
>> > cls/version/cls_version.cc:228: Loaded version class!
>> >
>> > 0> 2016-06-06 11:49:37.940701 7f09d0152800 -1 osd/OSD.h: In
>> > function
>> > 'OSDMapRef OSDService::get_map(epoch_t)' thread 7f09d0152800 time
>> > 2016-06-06
>> > 11:49:37.921083
>> >
>> > osd/OSD.h: 885: FAILED assert(ret)
>> >
>> >
>> > ceph version 10.2.1 (3a66dd4f30852819c1bdaa8ec23c795d4ad77269)
>> >
>> > 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char
>> > const*)+0x8b) [0x7f09d0b50e2b]
>> >
>> > 2: (OSDService::get_map(unsigned int)+0x3d) [0x7f09d055c85d]
>> >
>> > 3: (OSD::init()+0x1ed2) [0x7f09d0512052]
>> >
>> > 4: (main()+0x29d1) [0x7f09d0479cb1]
>> >
>> > 5: (__libc_start_main()+0xf5) [0x7f09cd075ec5]
>> >
>> > 6: (()+0x353987) [0x7f09d04c2987]
>> >
>> > NOTE: a copy of the executable, or `objdump -rdS <executable>` is
>> > needed to
>> > interpret this.
>> >
>> >
>> > --- logging levels ---
>> >
>> > 0/ 5 none
>> >
>> > 0/ 1 lockdep
>> >
>> > 0/ 1 context
>> >
>> > 1/ 1 crush
>> >
>> > 1/ 5 mds
>> >
>> > 1/ 5 mds_balancer
>> >
>> > 1/ 5 mds_locker
>> >
>> > 1/ 5 mds_log
>> >
>> > 1/ 5 mds_log_expire
>> >
>> > 1/ 5 mds_migrator
>> >
>> > 0/ 1 buffer
>> >
>> > 0/ 1 timer
>> >
>> > 0/ 1 filer
>> >
>> > 0/ 1 striper
>> >
>> > 0/ 1 objecter
>> >
>> > 0/ 5 rados
>> >
>> > 0/ 5 rbd
>> >
>> > 0/ 5 rbd_mirror
>> >
>> > 0/ 5 rbd_replay
>> >
>> > 0/ 5 journaler
>> >
>> > 0/ 5 objectcacher
>> >
>> > 0/ 5 client
>> >
>> > 0/ 5 osd
>> >
>> > 0/ 5 optracker
>> >
>> > 0/ 5 objclass
>> >
>> > 1/ 3 filestore
>> >
>> > 1/ 3 journal
>> >
>> > 0/ 5 ms
>> >
>> > 1/ 5 mon
>> >
>> > 0/10 monc
>> >
>> > 1/ 5 paxos
>> >
>> > 0/ 5 tp
>> >
>> > 1/ 5 auth
>> >
>> > 1/ 5 crypto
>> >
>> > 1/ 1 finisher
>> >
>> > 1/ 5 heartbeatmap
>> >
>> > 1/ 5 perfcounter
>> >
>> > 1/ 5 rgw
>> >
>> > 1/10 civetweb
>> >
>> > 1/ 5 javaclient
>> >
>> > 1/ 5 asok
>> >
>> > 1/ 1 throttle
>> >
>> > 0/ 0 refs
>> >
>> > 1/ 5 xio
>> >
>> > 1/ 5 compressor
>> >
>> > 1/ 5 newstore
>> >
>> > 1/ 5 bluestore
>> >
>> > 1/ 5 bluefs
>> >
>> > 1/ 3 bdev
>> >
>> > 1/ 5 kstore
>> >
>> > 4/ 5 rocksdb
>> >
>> > 4/ 5 leveldb
>> >
>> > 1/ 5 kinetic
>> >
>> > 1/ 5 fuse
>> >
>> > -2/-2 (syslog threshold)
>> >
>> > -1/-1 (stderr threshold)
>> >
>> > max_recent 10000
>> >
>> > max_new 1000
>> >
>> > log_file /var/log/ceph/ceph-osd.8.log
>> >
>> > --- end dump of recent events ---
>> >
>> >
>> > Now, if I do a zap on the disk and add everything back manually, that
>> > works,
>> > however I would really not like to do that for 12 x 10 nodes.
>> >
>> > Does anyone have any ideas?
>> >
>> > Thanks.
>> >
>> > -Tu Holmes
>> >
>> >
>> > _______________________________________________
>> > ceph-users mailing list
>> > ceph-users@xxxxxxxxxxxxxx
>> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>> >
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
