So after much testing, it appears there may be some residual data left on a disk by a ceph installation? I can't think of another explanation. I've done repeated installs using /dev/sdd, the first install worked, the rest failed. Subsequent installs would appear to work, the OSD daemon would start, but the OSD would never get marked as up. There were no errors in the logs indicating a disk issue. On a whim, I changed my configuration to use /dev/sdb, and it came up immediately. I used mkfs.xfs to format the disk. I'll do some additional testing to confirm this issue. I plan to completely zero the drive out and see if that works. > -----Original Message----- > From: Don Talton (dotalton) > Sent: Monday, July 29, 2013 1:53 PM > To: Gregory Farnum > Cc: ceph-users@xxxxxxxxxxxxxx > Subject: RE: OSD never gets marked up? > > Sorry, forgot to point out this: > > 2013-07-29 20:46:26.366916 7f4f28c6e700 0 -- 2.4.1.7:6801/13319 >> > 2.4.1.8:6802/18344 pipe(0x284378 0 sd=30 :53729 s=1 pgs=0 > cs=0 l=0).connect claims to be 2.4.1.8:6802/17272 not 2.4.1.8:6802/18344 - > wrong node! > > Not sure what that means? > > > -----Original Message----- > > From: ceph-users-bounces@xxxxxxxxxxxxxx [mailto:ceph-users- > > bounces@xxxxxxxxxxxxxx] On Behalf Of Don Talton (dotalton) > > Sent: Monday, July 29, 2013 1:49 PM > > To: Gregory Farnum > > Cc: ceph-users@xxxxxxxxxxxxxx > > Subject: Re: OSD never gets marked up? > > > > The entirety of the osd log file is below. I tried this on both > > bobtail and cuttlefish. Bobtail I noticed errors about all of the xfs > > features not being supported, which has gone away in cuttlefish. So I > > am assuming that issue is resolved. I don't see any other errors > > > > 2013-07-29 20:46:25.813383 7f4f3e909780 0 ceph version 0.61.7 > > (8f010aff684e820ecc837c25ac77c7a05d71 91ff), process > ceph- > > osd, pid 13319 > > 2013-07-29 20:46:25.814051 7f4f3e909780 1 -- 2.4.1.7:0/0 learned my > > addr > > 2.4.1.7:0/0 > > 2013-07-29 20:46:25.814068 7f4f3e909780 1 accepter.accepter.bind > > my_inst.addr is 2.4.1.7:6800/13319 need_addr=0 > > 2013-07-29 20:46:25.814085 7f4f3e909780 1 -- 2.4.1.7:0/0 learned my > > addr > > 2.4.1.7:0/0 > > 2013-07-29 20:46:25.814090 7f4f3e909780 1 accepter.accepter.bind > > my_inst.addr is 2.4.1.7:6801/13319 need_addr=0 > > 2013-07-29 20:46:25.814103 7f4f3e909780 1 -- 2.4.1.7:0/0 learned my > > addr > > 2.4.1.7:0/0 > > 2013-07-29 20:46:25.814108 7f4f3e909780 1 accepter.accepter.bind > > my_inst.addr is 2.4.1.7:6802/13319 need_addr=0 > > 2013-07-29 20:46:25.893876 7f4f3e909780 0 > > filestore(/var/lib/ceph/osd/osd.0) mount FIEMAP ioctl is supported and > > appears to work > > 2013-07-29 20:46:25.893892 7f4f3e909780 0 > > filestore(/var/lib/ceph/osd/osd.0) mount FIEMAP ioctl is disabled via > > 'filestore fiemap' config option > > 2013-07-29 20:46:25.894237 7f4f3e909780 0 > > filestore(/var/lib/ceph/osd/osd.0) mount did NOT detect b trfs > > 2013-07-29 20:46:25.943640 7f4f3e909780 0 > > filestore(/var/lib/ceph/osd/osd.0) mount syncfs(2) syscal l > > fully supported (by glibc and kernel) > > 2013-07-29 20:46:25.943784 7f4f3e909780 0 > > filestore(/var/lib/ceph/osd/osd.0) mount found snaps <> > > 2013-07-29 20:46:26.027401 7f4f3e909780 0 > > filestore(/var/lib/ceph/osd/osd.0) mount: enabling WRITEA HEAD journal > > mode: btrfs not detected > > 2013-07-29 20:46:26.034732 7f4f3e909780 -1 journal FileJournal::_open: > > disabling aio for non-block j ournal. Use journal_force_aio > to > > force use of aio anyway > > 2013-07-29 20:46:26.034828 7f4f3e909780 1 journal _open > > /var/lib/ceph/osd/osd.0/journal fd 20: 4294 967296 bytes, > > block size 4096 bytes, directio = 1, aio = 0 > > 2013-07-29 20:46:26.035140 7f4f3e909780 1 journal _open > > /var/lib/ceph/osd/osd.0/journal fd 20: 4294 967296 bytes, > > block size 4096 bytes, directio = 1, aio = 0 > > 2013-07-29 20:46:26.036189 7f4f3e909780 1 journal close > > /var/lib/ceph/osd/osd.0/journal > > 2013-07-29 20:46:26.036905 7f4f3e909780 1 -- 2.4.1.7:6800/13319 > > messenger.start > > 2013-07-29 20:46:26.036958 7f4f3e909780 1 -- :/0 messenger.start > > 2013-07-29 20:46:26.036975 7f4f3e909780 1 -- 2.4.1.7:6802/13319 > > messenger.start > > 2013-07-29 20:46:26.036993 7f4f3e909780 1 -- 2.4.1.7:6801/13319 > > messenger.start > > 2013-07-29 20:46:26.102317 7f4f3e909780 0 > > filestore(/var/lib/ceph/osd/osd.0) mount FIEMAP ioctl is supported and > > appears to work > > 2013-07-29 20:46:26.102329 7f4f3e909780 0 > > filestore(/var/lib/ceph/osd/osd.0) mount FIEMAP ioctl is disabled via > > 'filestore fiemap' config option > > 2013-07-29 20:46:26.102643 7f4f3e909780 0 > > filestore(/var/lib/ceph/osd/osd.0) mount did NOT detect b trfs > > 2013-07-29 20:46:26.201852 7f4f3e909780 0 > > filestore(/var/lib/ceph/osd/osd.0) mount syncfs(2) syscal l > > fully supported (by glibc and kernel) > > 2013-07-29 20:46:26.201928 7f4f3e909780 0 > > filestore(/var/lib/ceph/osd/osd.0) mount found snaps <> > > 2013-07-29 20:46:26.268809 7f4f3e909780 0 > > filestore(/var/lib/ceph/osd/osd.0) mount: enabling WRITEA HEAD journal > > mode: btrfs not detected > > 2013-07-29 20:46:26.272557 7f4f3e909780 -1 journal FileJournal::_open: > > disabling aio for non-block j ournal. Use journal_force_aio > to > > force use of aio anyway > > 2013-07-29 20:46:26.272577 7f4f3e909780 1 journal _open > > /var/lib/ceph/osd/osd.0/journal fd 28: 4294 967296 bytes, > > block size 4096 bytes, directio = 1, aio = 0 > > 2013-07-29 20:46:26.272819 7f4f3e909780 1 journal _open > > /var/lib/ceph/osd/osd.0/journal fd 28: 4294 967296 bytes, > > block size 4096 bytes, directio = 1, aio = 0 > > 2013-07-29 20:46:26.359004 7f4f3e909780 1 accepter.accepter.start > > 2013-07-29 20:46:26.359104 7f4f3e909780 1 accepter.accepter.start > > 2013-07-29 20:46:26.359226 7f4f3e909780 1 accepter.accepter.start > > 2013-07-29 20:46:26.360175 7f4f3e909780 1 -- 2.4.1.7:6800/13319 --> > > 2.4.1.4:6789/0 -- auth(proto 0 26 bytes epoch 0) v1 -- ?+0 > > 0x28086c0 con 0x285a160 > > 2013-07-29 20:46:26.362067 7f4f31580700 1 -- 2.4.1.7:6800/13319 <== mon.0 > > 2.4.1.4:6789/0 1 ==== mon _map v1 ==== 191+0+0 > > (4283215978 0 0) 0x2d68000 con 0x285a160 > > 2013-07-29 20:46:26.362175 7f4f31580700 1 monclient(hunting): found > > mon.0 > > 2013-07-29 20:46:26.362183 7f4f31580700 1 -- 2.4.1.7:6800/13319 <== mon.0 > > 2.4.1.4:6789/0 2 ==== aut h_reply(proto 2 0 Success) v1 > ==== > > 33+0+0 (658115354 0 0) 0x2d68400 con 0x285a160 > > 2013-07-29 20:46:26.362381 7f4f31580700 1 -- 2.4.1.7:6800/13319 --> > > 2.4.1.4:6789/0 -- auth(proto 2 32 bytes epoch 0) v1 -- ?+0 > > 0x2d66b40 con 0x285a160 > > 2013-07-29 20:46:26.363316 7f4f31580700 1 -- 2.4.1.7:6800/13319 <== mon.0 > > 2.4.1.4:6789/0 3 ==== aut h_reply(proto 2 0 Success) v1 > ==== > > 206+0+0 (453976051 0 0) 0x2d68200 con 0x285a160 > > 2013-07-29 20:46:26.363470 7f4f31580700 1 -- 2.4.1.7:6800/13319 --> > > 2.4.1.4:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- ?+0 > > 0x2d66d80 con 0x285a160 > > 2013-07-29 20:46:26.364569 7f4f31580700 1 -- 2.4.1.7:6800/13319 <== mon.0 > > 2.4.1.4:6789/0 4 ==== aut h_reply(proto 2 0 Success) v1 > ==== > > 393+0+0 (184374621 0 0) 0x2d68600 con 0x285a160 > > 2013-07-29 20:46:26.364662 7f4f31580700 1 -- 2.4.1.7:6800/13319 --> > > 2.4.1.4:6789/0 -- mon_subscribe ({monmap=0+}) v2 -- ?+0 > > 0x2829700 con 0x285a160 > > 2013-07-29 20:46:26.364690 7f4f31580700 1 -- 2.4.1.7:6800/13319 --> > > 2.4.1.4:6789/0 -- mon_subscribe > > ({monmap=0+,osd_pg_creates=0}) v2 -- ?+0 0x2829a80 con 0x285a160 > > 2013-07-29 20:46:26.364759 7f4f31580700 1 -- 2.4.1.7:6800/13319 --> > > 2.4.1.4:6789/0 -- auth(proto 2 2 bytes epoch 0) v1 -- ?+0 > > 0x2d66b40 con 0x285a160 > > 2013-07-29 20:46:26.364866 7f4f3e909780 5 monclient: authenticate > > success, global_id 4267 > > 2013-07-29 20:46:26.365466 7f4f31580700 1 -- 2.4.1.7:6800/13319 <== mon.0 > > 2.4.1.4:6789/0 5 ==== mon _map v1 ==== 191+0+0 > > (4283215978 0 0) 0x2d68400 con 0x285a160 > > 2013-07-29 20:46:26.365518 7f4f31580700 1 -- 2.4.1.7:6800/13319 <== mon.0 > > 2.4.1.4:6789/0 6 ==== mon _subscribe_ack(300s) v1 ==== > > 20+0+0 (1055034598 0 0) 0x2829a80 con 0x285a160 > > 2013-07-29 20:46:26.365555 7f4f31580700 1 -- 2.4.1.7:6800/13319 <== mon.0 > > 2.4.1.4:6789/0 7 ==== mon _map v1 ==== 191+0+0 > > (4283215978 0 0) 0x2d68000 con 0x285a160 > > 2013-07-29 20:46:26.365581 7f4f31580700 1 -- 2.4.1.7:6800/13319 <== mon.0 > > 2.4.1.4:6789/0 8 ==== mon _subscribe_ack(300s) v1 ==== > > 20+0+0 (1055034598 0 0) 0x2829700 con 0x285a160 > > 2013-07-29 20:46:26.365732 7f4f31580700 1 -- 2.4.1.7:6800/13319 <== mon.0 > > 2.4.1.4:6789/0 9 ==== aut h_reply(proto 2 0 Success) v1 > ==== > > 194+0+0 (3275840853 0 0) 0x2d68800 con 0x285a160 > > 2013-07-29 20:46:26.366916 7f4f28c6e700 0 -- 2.4.1.7:6801/13319 >> > > 2.4.1.8:6802/18344 pipe(0x284378 0 sd=30 :53729 s=1 pgs=0 > > cs=0 l=0).connect claims to be 2.4.1.8:6802/17272 not > > 2.4.1.8:6802/18344 - wrong node! > > 2013-07-29 20:46:26.367028 7f4f28c6e700 0 -- 2.4.1.7:6801/13319 >> > > 2.4.1.8:6802/18344 pipe(0x284378 0 sd=30 :53729 s=1 pgs=0 > > cs=0 l=0).fault with nothing to send, going to standby > > 2013-07-29 20:46:26.369488 7f4f3e909780 1 -- 2.4.1.7:6800/13319 --> > > 2.4.1.4:6789/0 -- mon_get_versi on(what=osdmap > handle=1) > > v1 -- ?+0 0x2829540 con 0x285a160 > > 2013-07-29 20:46:26.370168 7f4f31580700 1 -- 2.4.1.7:6800/13319 <== mon.0 > > 2.4.1.4:6789/0 10 ==== mo n_check_map_ack(handle=1 > > version=5) v2 ==== 24+0+0 (3346734780 0 0) 0x2829e00 con 0x285a160 > > 2013-07-29 20:46:26.370303 7f4f2d578700 1 -- 2.4.1.7:6800/13319 --> > > 2.4.1.4:6789/0 -- osd_boot(osd. 0 booted 0 v64) v3 -- ?+0 > > 0x2833400 con 0x285a160 > > root@ceph-osd0:~# cat /var/log/ceph/ceph-osd.0.log > > 2013-07-29 20:46:25.813383 7f4f3e909780 0 ceph version 0.61.7 > > (8f010aff684e820ecc837c25ac77c7a05d71 91ff), process > ceph- > > osd, pid 13319 > > 2013-07-29 20:46:25.814051 7f4f3e909780 1 -- 2.4.1.7:0/0 learned my > > addr > > 2.4.1.7:0/0 > > 2013-07-29 20:46:25.814068 7f4f3e909780 1 accepter.accepter.bind > > my_inst.addr is 2.4.1.7:6800/13319 need_addr=0 > > 2013-07-29 20:46:25.814085 7f4f3e909780 1 -- 2.4.1.7:0/0 learned my > > addr > > 2.4.1.7:0/0 > > 2013-07-29 20:46:25.814090 7f4f3e909780 1 accepter.accepter.bind > > my_inst.addr is 2.4.1.7:6801/13319 need_addr=0 > > 2013-07-29 20:46:25.814103 7f4f3e909780 1 -- 2.4.1.7:0/0 learned my > > addr > > 2.4.1.7:0/0 > > 2013-07-29 20:46:25.814108 7f4f3e909780 1 accepter.accepter.bind > > my_inst.addr is 2.4.1.7:6802/13319 need_addr=0 > > 2013-07-29 20:46:25.893876 7f4f3e909780 0 > > filestore(/var/lib/ceph/osd/osd.0) mount FIEMAP ioctl is supported and > > appears to work > > 2013-07-29 20:46:25.893892 7f4f3e909780 0 > > filestore(/var/lib/ceph/osd/osd.0) mount FIEMAP ioctl is disabled via > > 'filestore fiemap' config option > > 2013-07-29 20:46:25.894237 7f4f3e909780 0 > > filestore(/var/lib/ceph/osd/osd.0) mount did NOT detect b trfs > > 2013-07-29 20:46:25.943640 7f4f3e909780 0 > > filestore(/var/lib/ceph/osd/osd.0) mount syncfs(2) syscal l > > fully supported (by glibc and kernel) > > 2013-07-29 20:46:25.943784 7f4f3e909780 0 > > filestore(/var/lib/ceph/osd/osd.0) mount found snaps <> > > 2013-07-29 20:46:26.027401 7f4f3e909780 0 > > filestore(/var/lib/ceph/osd/osd.0) mount: enabling WRITEA HEAD journal > > mode: btrfs not detected > > 2013-07-29 20:46:26.034732 7f4f3e909780 -1 journal FileJournal::_open: > > disabling aio for non-block j ournal. Use journal_force_aio > to > > force use of aio anyway > > 2013-07-29 20:46:26.034828 7f4f3e909780 1 journal _open > > /var/lib/ceph/osd/osd.0/journal fd 20: 4294 967296 bytes, > > block size 4096 bytes, directio = 1, aio = 0 > > 2013-07-29 20:46:26.035140 7f4f3e909780 1 journal _open > > /var/lib/ceph/osd/osd.0/journal fd 20: 4294 967296 bytes, > > block size 4096 bytes, directio = 1, aio = 0 > > 2013-07-29 20:46:26.036189 7f4f3e909780 1 journal close > > /var/lib/ceph/osd/osd.0/journal > > 2013-07-29 20:46:26.036905 7f4f3e909780 1 -- 2.4.1.7:6800/13319 > > messenger.start > > 2013-07-29 20:46:26.036958 7f4f3e909780 1 -- :/0 messenger.start > > 2013-07-29 20:46:26.036975 7f4f3e909780 1 -- 2.4.1.7:6802/13319 > > messenger.start > > 2013-07-29 20:46:26.036993 7f4f3e909780 1 -- 2.4.1.7:6801/13319 > > messenger.start > > 2013-07-29 20:46:26.102317 7f4f3e909780 0 > > filestore(/var/lib/ceph/osd/osd.0) mount FIEMAP ioctl is supported and > > appears to work > > 2013-07-29 20:46:26.102329 7f4f3e909780 0 > > filestore(/var/lib/ceph/osd/osd.0) mount FIEMAP ioctl is disabled via > > 'filestore fiemap' config option > > 2013-07-29 20:46:26.102643 7f4f3e909780 0 > > filestore(/var/lib/ceph/osd/osd.0) mount did NOT detect b trfs > > 2013-07-29 20:46:26.201852 7f4f3e909780 0 > > filestore(/var/lib/ceph/osd/osd.0) mount syncfs(2) syscal l > > fully supported (by glibc and kernel) > > 2013-07-29 20:46:26.201928 7f4f3e909780 0 > > filestore(/var/lib/ceph/osd/osd.0) mount found snaps <> > > 2013-07-29 20:46:26.268809 7f4f3e909780 0 > > filestore(/var/lib/ceph/osd/osd.0) mount: enabling WRITEA HEAD journal > > mode: btrfs not detected > > 2013-07-29 20:46:26.272557 7f4f3e909780 -1 journal FileJournal::_open: > > disabling aio for non-block j ournal. Use journal_force_aio > to > > force use of aio anyway > > 2013-07-29 20:46:26.272577 7f4f3e909780 1 journal _open > > /var/lib/ceph/osd/osd.0/journal fd 28: 4294 967296 bytes, > > block size 4096 bytes, directio = 1, aio = 0 > > 2013-07-29 20:46:26.272819 7f4f3e909780 1 journal _open > > /var/lib/ceph/osd/osd.0/journal fd 28: 4294 967296 bytes, > > block size 4096 bytes, directio = 1, aio = 0 > > 2013-07-29 20:46:26.359004 7f4f3e909780 1 accepter.accepter.start > > 2013-07-29 20:46:26.359104 7f4f3e909780 1 accepter.accepter.start > > 2013-07-29 20:46:26.359226 7f4f3e909780 1 accepter.accepter.start > > 2013-07-29 20:46:26.360175 7f4f3e909780 1 -- 2.4.1.7:6800/13319 --> > > 2.4.1.4:6789/0 -- auth(proto 0 26 bytes epoch 0) v1 -- ?+0 > > 0x28086c0 con 0x285a160 > > 2013-07-29 20:46:26.362067 7f4f31580700 1 -- 2.4.1.7:6800/13319 <== mon.0 > > 2.4.1.4:6789/0 1 ==== mon _map v1 ==== 191+0+0 > > (4283215978 0 0) 0x2d68000 con 0x285a160 > > 2013-07-29 20:46:26.362175 7f4f31580700 1 monclient(hunting): found > > mon.0 > > 2013-07-29 20:46:26.362183 7f4f31580700 1 -- 2.4.1.7:6800/13319 <== mon.0 > > 2.4.1.4:6789/0 2 ==== aut h_reply(proto 2 0 Success) v1 > ==== > > 33+0+0 (658115354 0 0) 0x2d68400 con 0x285a160 > > 2013-07-29 20:46:26.362381 7f4f31580700 1 -- 2.4.1.7:6800/13319 --> > > 2.4.1.4:6789/0 -- auth(proto 2 32 bytes epoch 0) v1 -- ?+0 > > 0x2d66b40 con 0x285a160 > > 2013-07-29 20:46:26.363316 7f4f31580700 1 -- 2.4.1.7:6800/13319 <== mon.0 > > 2.4.1.4:6789/0 3 ==== aut h_reply(proto 2 0 Success) v1 > ==== > > 206+0+0 (453976051 0 0) 0x2d68200 con 0x285a160 > > 2013-07-29 20:46:26.363470 7f4f31580700 1 -- 2.4.1.7:6800/13319 --> > > 2.4.1.4:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- ?+0 > > 0x2d66d80 con 0x285a160 > > 2013-07-29 20:46:26.364569 7f4f31580700 1 -- 2.4.1.7:6800/13319 <== mon.0 > > 2.4.1.4:6789/0 4 ==== aut h_reply(proto 2 0 Success) v1 > ==== > > 393+0+0 (184374621 0 0) 0x2d68600 con 0x285a160 > > 2013-07-29 20:46:26.364662 7f4f31580700 1 -- 2.4.1.7:6800/13319 --> > > 2.4.1.4:6789/0 -- mon_subscribe ({monmap=0+}) v2 -- ?+0 > > 0x2829700 con 0x285a160 > > 2013-07-29 20:46:26.364690 7f4f31580700 1 -- 2.4.1.7:6800/13319 --> > > 2.4.1.4:6789/0 -- mon_subscribe > > ({monmap=0+,osd_pg_creates=0}) v2 -- ?+0 0x2829a80 con 0x285a160 > > 2013-07-29 20:46:26.364759 7f4f31580700 1 -- 2.4.1.7:6800/13319 --> > > 2.4.1.4:6789/0 -- auth(proto 2 2 bytes epoch 0) v1 -- ?+0 > > 0x2d66b40 con 0x285a160 > > 2013-07-29 20:46:26.364866 7f4f3e909780 5 monclient: authenticate > > success, global_id 4267 > > 2013-07-29 20:46:26.365466 7f4f31580700 1 -- 2.4.1.7:6800/13319 <== mon.0 > > 2.4.1.4:6789/0 5 ==== mon _map v1 ==== 191+0+0 > > (4283215978 0 0) 0x2d68400 con 0x285a160 > > 2013-07-29 20:46:26.365518 7f4f31580700 1 -- 2.4.1.7:6800/13319 <== mon.0 > > 2.4.1.4:6789/0 6 ==== mon _subscribe_ack(300s) v1 ==== > > 20+0+0 (1055034598 0 0) 0x2829a80 con 0x285a160 > > 2013-07-29 20:46:26.365555 7f4f31580700 1 -- 2.4.1.7:6800/13319 <== mon.0 > > 2.4.1.4:6789/0 7 ==== mon _map v1 ==== 191+0+0 > > (4283215978 0 0) 0x2d68000 con 0x285a160 > > 2013-07-29 20:46:26.365581 7f4f31580700 1 -- 2.4.1.7:6800/13319 <== mon.0 > > 2.4.1.4:6789/0 8 ==== mon _subscribe_ack(300s) v1 ==== > > 20+0+0 (1055034598 0 0) 0x2829700 con 0x285a160 > > 2013-07-29 20:46:26.365732 7f4f31580700 1 -- 2.4.1.7:6800/13319 <== mon.0 > > 2.4.1.4:6789/0 9 ==== aut h_reply(proto 2 0 Success) v1 > ==== > > 194+0+0 (3275840853 0 0) 0x2d68800 con 0x285a160 > > 2013-07-29 20:46:26.366916 7f4f28c6e700 0 -- 2.4.1.7:6801/13319 >> > > 2.4.1.8:6802/18344 pipe(0x284378 0 sd=30 :53729 s=1 pgs=0 > > cs=0 l=0).connect claims to be 2.4.1.8:6802/17272 not > > 2.4.1.8:6802/18344 - wrong node! > > 2013-07-29 20:46:26.367028 7f4f28c6e700 0 -- 2.4.1.7:6801/13319 >> > > 2.4.1.8:6802/18344 pipe(0x284378 0 sd=30 :53729 s=1 pgs=0 > > cs=0 l=0).fault with nothing to send, going to standby > > 2013-07-29 20:46:26.369488 7f4f3e909780 1 -- 2.4.1.7:6800/13319 --> > > 2.4.1.4:6789/0 -- mon_get_versi on(what=osdmap > handle=1) > > v1 -- ?+0 0x2829540 con 0x285a160 > > 2013-07-29 20:46:26.370168 7f4f31580700 1 -- 2.4.1.7:6800/13319 <== mon.0 > > 2.4.1.4:6789/0 10 ==== mo n_check_map_ack(handle=1 > > version=5) v2 ==== 24+0+0 (3346734780 0 0) 0x2829e00 con 0x285a160 > > 2013-07-29 20:46:26.370303 7f4f2d578700 1 -- 2.4.1.7:6800/13319 --> > > 2.4.1.4:6789/0 -- osd_boot(osd. 0 booted 0 v64) v3 -- ?+0 > > 0x2833400 con 0x285a160 > > > > > > > > > > > > > > > > > -----Original Message----- > > > From: Gregory Farnum [mailto:greg@xxxxxxxxxxx] > > > Sent: Monday, July 29, 2013 12:17 PM > > > To: Don Talton (dotalton) > > > Cc: ceph-users@xxxxxxxxxxxxxx > > > Subject: Re: OSD never gets marked up? > > > > > > On Mon, Jul 29, 2013 at 11:36 AM, Don Talton (dotalton) > > > <dotalton@xxxxxxxxx> wrote: > > > > Hello, > > > > > > > > I have a small test cluster that I deploy using puppet-ceph. Both > > > > the MON > > > and the OSDs deploy properly, and appear to have all of the correct > > > configurations. However, the OSDs are never marked as up. Any input > > > is appreciated. The daemons are running on each OSD server, the OSDs > > > are listed in the crushmap, and I can see them successfully > > > authenticate with the MON eg. > > > > > > > > 2013-07-29 18:34:19.231905 7fbfeaaa5700 1 -- 2.4.1.7:0/14269 <== > > > > mon.0 2.4.1.4:6789/0 8 ==== > > > > mon_command_ack([osd,crush,create-or- > > > move,0,0.91,root=default,host=cep > > > > h-osd0]=0 create-or-move updated item id 0 name 'osd.0' weight > > > > 0.91 at location {host=ceph-osd0,root=default} to crush map v8) v1 > > > > ==== > > > > 223+0+0 (1170550300 0 0) 0x7fbfe00016d0 con > > > > 223+0+0x16b1770create-or-move > > > > updated item id 0 name 'osd.0' weight 0.91 at location > > > > {host=ceph-osd0,root=default} to crush map > > > > > > This is actually not the OSDs doing this, but the ceph admin tool > > > using the bootstrap-osd key. Do you have anything that's from the > > > OSDs? There might be something helpful in the OSD logs; if not you > > > can add "debug ms = 1" and restart them and there certainly will be. > > > -Greg > > > Software Engineer #42 @ http://inktank.com | http://ceph.com > > > > > > > > > > > > > > > My ceph.conf > > > > > > > > [global] > > > > auth cluster required = cephx > > > > auth service required = cephx > > > > auth client required = cephx > > > > keyring = /etc/ceph/keyring > > > > > > > > fsid = e80afa94-a64c-486c-9e34-d55e85f26406 > > > > debug ms = 1/5 > > > > > > > > [mon] > > > > mon data = /var/lib/ceph/mon/mon.$id > > > > debug mon = 20 > > > > debug paxos = 1/5 > > > > debug auth = 2 > > > > > > > > [osd] > > > > osd journal size = 4096 > > > > cluster network = 2.4.1.0/24 > > > > public network = 2.4.1.0/24 > > > > filestore flusher = false > > > > osd data = /var/lib/ceph/osd/osd.$id > > > > osd journal = /var/lib/ceph/osd/osd.$id/journal > > > > osd mkfs type = xfs > > > > keyring = /var/lib/ceph/osd/osd.$id/keyring > > > > debug osd = 1/5 > > > > debug filestore = 1/5 > > > > debug journal = 1 > > > > debug monc = 5/20 > > > > > > > > [mds] > > > > mds data = /var/lib/ceph/mds/mds.$id > > > > keyring = /var/lib/ceph/mds/mds.$id/keyring > > > > > > > > [mon.0] > > > > host = ceph-mon0 > > > > mon addr = 2.4.1.4:6789 > > > > _______________________________________________ > > > > ceph-users mailing list > > > > ceph-users@xxxxxxxxxxxxxx > > > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > _______________________________________________ > > ceph-users mailing list > > ceph-users@xxxxxxxxxxxxxx > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |