I've seen this before too. CentOS starts up without networking on by default. In my case, the problem was that the monitors cannot form a quorum and OSDs cannot find each other or monitors. Hence, you get that broken pipe error. You either need to have the networking start on startup before the OSDs, or start ceph after you boot up and ensure the network is running properly. The nodes have to be able to reach each other for Ceph to work. As for Ubuntu, I believe the networking is on by default. On Wed, Nov 6, 2013 at 1:35 PM, Trivedi, Narendra <Narendra.Trivedi@xxxxxxxxxx> wrote: > Hi All, > > > > I did a fresh install of Ceph (this might be like 10th or 11th install) on 4 > new VMs (one admin, one MON and two OSDs) built from CentOS 6.4 (x64) .iso , > did a yum update on all of them. They are all running on vmware ESXi 5.1.0. > I did everything sage et al suggested (i.e. creation of /ceph/osd* and > making sure /etc/ceph is present on all nodes. /etc/ceph gets created all > the ceph-deploy install and contains rbdmap FYI). Unusually, I ended up with > the same problem while activating OSDs (the last 4 lines keep going on and > on forever): > > > > 2013-11-06 14:37:39,626 [ceph_deploy.cli][INFO ] Invoked (1.3): > /usr/bin/ceph-deploy osd activate ceph-node2-osd0-centos-6-4:/ceph/osd0 > ceph-node3-osd1-centos-6-4:/ceph/osd1 > > 2013-11-06 14:37:39,627 [ceph_deploy.osd][DEBUG ] Activating cluster ceph > disks ceph-node2-osd0-centos-6-4:/ceph/osd0: > ceph-node3-osd1-centos-6-4:/ceph/osd1: > > 2013-11-06 14:37:39,901 [ceph-node2-osd0-centos-6-4][DEBUG ] connected to > host: ceph-node2-osd0-centos-6-4 > > 2013-11-06 14:37:39,902 [ceph-node2-osd0-centos-6-4][DEBUG ] detect platform > information from remote host > > 2013-11-06 14:37:39,917 [ceph-node2-osd0-centos-6-4][DEBUG ] detect machine > type > > 2013-11-06 14:37:39,925 [ceph_deploy.osd][INFO ] Distro info: CentOS 6.4 > Final > > 2013-11-06 14:37:39,925 [ceph_deploy.osd][DEBUG ] activating host > ceph-node2-osd0-centos-6-4 disk /ceph/osd0 > > 2013-11-06 14:37:39,925 [ceph_deploy.osd][DEBUG ] will use init type: > sysvinit > > 2013-11-06 14:37:39,925 [ceph-node2-osd0-centos-6-4][INFO ] Running > command: sudo ceph-disk-activate --mark-init sysvinit --mount /ceph/osd0 > > 2013-11-06 14:37:40,145 [ceph-node2-osd0-centos-6-4][ERROR ] 2013-11-06 > 14:37:41.075310 7fac2414c700 0 -- :/1029546 >> 10.12.0.70:6789/0 > pipe(0x7fac20024480 sd=4 :0 s=1 pgs=0 cs=0 l=1 c=0x7fac200246e0).fault > > 2013-11-06 14:37:43,167 [ceph-node2-osd0-centos-6-4][ERROR ] 2013-11-06 > 14:37:44.071697 7fac1ebfd700 0 -- :/1029546 >> 10.12.0.70:6789/0 > pipe(0x7fac14000c00 sd=4 :0 s=1 pgs=0 cs=0 l=1 c=0x7fac14000e60).fault > > 2013-11-06 14:37:46,140 [ceph-node2-osd0-centos-6-4][ERROR ] 2013-11-06 > 14:37:47.071938 7fac2414c700 0 -- :/1029546 >> 10.12.0.70:6789/0 > pipe(0x7fac14003010 sd=4 :0 s=1 pgs=0 cs=0 l=1 c=0x7fac14003270).fault > > 2013-11-06 14:37:50,165 [ceph-node2-osd0-centos-6-4][ERROR ] 2013-11-06 > 14:37:51.071245 7fac1ebfd700 0 -- :/1029546 >> 10.12.0.70:6789/0 > pipe(0x7fac14003a70 sd=4 :0 s=1 pgs=0 cs=0 l=1 c=0x7fac14003cd0).fault > > > > It might be bad luck but I want to try a manual installation without > ceph-deploy because it seems I am jinxed with ceph-deploy. Could anyone > please forward me the steps. I am happy to share the ceph.log with anyone > who would like to research on this error but I don’t a have clue. > > > > > > Thanks a lot! > > Narendra Trivedi | savviscloud > > > > > This message contains information which may be confidential and/or > privileged. Unless you are the intended recipient (or authorized to receive > for the intended recipient), you may not read, use, copy or disclose to > anyone the message or any information contained in the message. If you have > received the message in error, please advise the sender by reply e-mail and > delete the message and any attachment(s) thereto without retaining any > copies. > > _______________________________________________ > ceph-users mailing list > ceph-users@xxxxxxxxxxxxxx > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > -- John Wilkins Senior Technical Writer Intank john.wilkins@xxxxxxxxxxx (415) 425-9599 http://inktank.com _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |