Hi Nabil 1) I hope you have taken ceph services bounce back after copying keyring files 2) from your OSD node ( ceph-node2 ) are you able to check your cluster status #ceph status , it should return output similarly to ceph-node1 ( monitor node ) if not then there is connectivity problem between two. 3) check for iptables between machines ( if this is your testing cluster disable iptables ) Note: in my setup and installation guide, the Admin node(ceph-deploy) is a separate server other than ceph-node1(Monitor) and ceph-node2, ceph-node3(OSD), and the admin node doesn't required ceph installation only ceph-deploy, is that right ? also admin node (ceph-deploy) gather keys from ceph-node1 (monitor node) only, right ? yes this seems to be right 4) also check copying keys from ceph-deploy node to ceph-node2 Regards Karan ----- Original Message ----- From: "Nabil Naim" <nabil_naim@xxxxxxxxxxxxxxxxxx> To: "Karan Singh" <ksingh@xxxxxx> Cc: ceph-users@xxxxxxxxxxxxxx Sent: Wednesday, 30 October, 2013 1:24:33 PM Subject: RE: activate disk error Hi Karan, Thank you for reply and help, to keep names simple let us use the installation guide naming http://ceph.com/docs/master/_images/ditaa-ab0a88be6a09668151342b36da8ceabaf0528f79.png so I Copy <cluster_name>.client.admin.keyring from ceph-node1 (Monitor node) to /etc/ceph at ceph-node2 (1st OSD node) sudo scp ceph-node1:/etc/ceph/ceph.client.admin.keyring ceph-node2:/etc/ceph/ And Copy /var/lib/ceph/bootstrap-osd/ceph.keyring from ceph-node1 (Monitor node) to /var/lib/ceph/bootstrap-osd ceph-node2 (1st OSD node) sudo scp ceph-server02:/var/lib/ceph/bootstrap-osd/ceph.keyring ceph-server02:/var/lib/ceph/bootstrap-osd/ then using the Admin node (ceph-deploy) I run [ceph@ceph-deploy my-cluster]$ ceph-deploy disk list ceph-node2 [ceph_deploy.cli][INFO ] Invoked (1.2.7): /usr/bin/ceph-deploy disk list ceph-node2 [ceph_deploy.sudo_pushy][DEBUG ] will use a remote connection with sudo [ceph_deploy.osd][INFO ] Distro info: CentOS 6.4 Final [ceph_deploy.osd][DEBUG ] Listing disks on ceph-node2... [ceph-server02][INFO ] Running command: ceph-disk list [ceph-server02][INFO ] /dev/sda : [ceph-server02][INFO ] /dev/sda1 other, ext4, mounted on /boot [ceph-server02][INFO ] /dev/sda2 other, LVM2_member [ceph-server02][INFO ] /dev/sdb : [ceph-server02][INFO ] /dev/sdb1 ceph data, prepared, cluster ceph, journal /dev/sdb2 [ceph-server02][INFO ] /dev/sdb2 ceph journal, for /dev/sdb1 [ceph-server02][INFO ] /dev/sr0 other, unknown Then also using ceph Admin node (ceph-deply) I run [ceph@ceph-deploy my-cluster]$ ceph-deploy osd activate ceph-node2:/dev/sdb1 [ceph_deploy.cli][INFO ] Invoked (1.2.7): /usr/bin/ceph-deploy osd activate ceph-node2:/dev/sdb1 [ceph_deploy.osd][DEBUG ] Activating cluster ceph disks ceph-node2:/dev/sdb1: [ceph_deploy.sudo_pushy][DEBUG ] will use a remote connection with sudo [ceph_deploy.osd][INFO ] Distro info: CentOS 6.4 Final [ceph_deploy.osd][DEBUG ] activating host ceph-node2 disk /dev/sdb1 [ceph_deploy.osd][DEBUG ] will use init type: sysvinit [ceph-server02][INFO ] Running command: ceph-disk-activate --mark-init sysvinit --mount /dev/sdb1 Hang for 5 min While hanging the logs at ceph-node2 [ceph@ceph-node2 ceph]$ ls -ltr total 12 -rw-r--r-- 1 root root 0 Oct 30 03:44 ceph-osd..log -rw-r--r-- 1 root root 0 Oct 30 03:44 ceph-osd.0.log -rw-r--r-- 1 root root 0 Oct 30 03:44 ceph-client.admin.log And logs at ceph-node1 [ceph@ceph-node1 ceph]$ ls -ltr total 4520 -rw-r--r-- 1 root root 0 Oct 29 19:33 ceph-osd.ceph-server02.log -rw-r--r-- 1 root root 0 Oct 30 03:13 ceph-osd..log -rw-r--r-- 1 root root 0 Oct 30 03:13 ceph-osd.0.log -rw------- 1 root root 0 Oct 30 03:13 ceph.log -rw-r--r-- 1 root root 0 Oct 30 03:13 ceph-client.admin.log -rw-r--r-- 1 root root 4415099 Oct 30 14:11 ceph-mon.ceph-server01.log [ceph@ceph-node1 ceph]$ sudo tail ceph-mon.ceph-server01.log 2013-10-30 14:10:59.151566 7f6a0c1a4700 1 mon.ceph-server01@0(leader).paxos(paxos active c 1..80) is_readable now=2013-10-30 14:10:59.151567 lease_expire=0.000000 has v0 lc 80 2013-10-30 14:11:04.151701 7f6a0c1a4700 1 mon.ceph-server01@0(leader).paxos(paxos active c 1..80) is_readable now=2013-10-30 14:11:04.151708 lease_expire=0.000000 has v0 lc 80 2013-10-30 14:11:04.151744 7f6a0c1a4700 1 mon.ceph-server01@0(leader).paxos(paxos active c 1..80) is_readable now=2013-10-30 14:11:04.151745 lease_expire=0.000000 has v0 lc 80 2013-10-30 14:11:04.151755 7f6a0c1a4700 1 mon.ceph-server01@0(leader).paxos(paxos active c 1..80) is_readable now=2013-10-30 14:11:04.151756 lease_expire=0.000000 has v0 lc 80 2013-10-30 14:11:09.151908 7f6a0c1a4700 1 mon.ceph-server01@0(leader).paxos(paxos active c 1..80) is_readable now=2013-10-30 14:11:09.151915 lease_expire=0.000000 has v0 lc 80 2013-10-30 14:11:09.151948 7f6a0c1a4700 1 mon.ceph-server01@0(leader).paxos(paxos active c 1..80) is_readable now=2013-10-30 14:11:09.151950 lease_expire=0.000000 has v0 lc 80 2013-10-30 14:11:09.151959 7f6a0c1a4700 1 mon.ceph-server01@0(leader).paxos(paxos active c 1..80) is_readable now=2013-10-30 14:11:09.151960 lease_expire=0.000000 has v0 lc 80 2013-10-30 14:11:14.152094 7f6a0c1a4700 1 mon.ceph-server01@0(leader).paxos(paxos active c 1..80) is_readable now=2013-10-30 14:11:14.152102 lease_expire=0.000000 has v0 lc 80 2013-10-30 14:11:14.152138 7f6a0c1a4700 1 mon.ceph-server01@0(leader).paxos(paxos active c 1..80) is_readable now=2013-10-30 14:11:14.152140 lease_expire=0.000000 has v0 lc 80 2013-10-30 14:11:14.152150 7f6a0c1a4700 1 mon.ceph-server01@0(leader).paxos(paxos active c 1..80) is_readable now=2013-10-30 14:11:14.152150 lease_expire=0.000000 has v0 lc 80 Then ceph admin (ceph-deploy) node comes with error [ceph-server02][ERROR ] 2013-10-30 10:52:03.307808 7f48a8169700 0 -- :/1002416 >> 192.168.115.91:6789/0 pipe(0x7f489800cae0 sd=10 :0 s=1 pgs=0 cs=0 l=1 c=0x7f4898000a60).fault [ceph-server02][ERROR ] 2013-10-30 10:52:06.308108 7f48a826a700 0 -- :/1002416 >> 192.168.115.91:6789/0 pipe(0x7f489800d110 sd=10 :0 s=1 pgs=0 cs=0 l=1 c=0x7f48980008c0).fault [ceph-server02][ERROR ] 2013-10-30 10:52:09.308682 7f48a8169700 0 -- :/1002416 >> 192.168.115.91:6789/0 pipe(0x7f489800cae0 sd=10 :0 s=1 pgs=0 cs=0 l=1 c=0x7f48980130e0).fault [ceph-server02][ERROR ] 2013-10-30 10:52:12.309033 7f48a826a700 0 -- :/1002416 >> 192.168.115.91:6789/0 pipe(0x7f489800f100 sd=10 :0 s=1 pgs=0 cs=0 l=1 c=0x7f4898002290).fault [ceph-server02][ERROR ] 2013-10-30 10:52:15.309477 7f48a8169700 0 -- :/1002416 >> 192.168.115.91:6789/0 pipe(0x7f489800d110 sd=10 :0 s=1 pgs=0 cs=0 l=1 c=0x7f48980130e0).fault [ceph-server02][ERROR ] 2013-10-30 10:52:18.309909 7f48a826a700 0 -- :/1002416 >> 192.168.115.91:6789/0 pipe(0x7f489800f100 sd=10 :0 s=1 pgs=0 cs=0 l=1 c=0x7f4898002290).fault [ceph-server02][ERROR ] 2013-10-30 10:52:21.310938 7f48a8169700 0 -- :/1002416 >> 192.168.115.91:6789/0 pipe(0x7f489800d110 sd=10 :0 s=1 pgs=0 cs=0 l=1 c=0x7f4898002ef0).fault [ceph-server02][ERROR ] 2013-10-30 10:52:24.310740 7f48a826a700 0 -- :/1002416 >> 192.168.115.91:6789/0 pipe(0x7f489800f100 sd=10 :0 s=1 pgs=0 cs=0 l=1 c=0x7f48980037c0).fault [ceph-server02][ERROR ] 2013-10-30 10:52:27.311184 7f48a8169700 0 -- :/1002416 >> 192.168.115.91:6789/0 pipe(0x7f489800d110 sd=10 :0 s=1 pgs=0 cs=0 l=1 c=0x7f4898002ef0).fault [ceph-server02][ERROR ] 2013-10-30 10:52:30.265401 7f48a98fb700 0 monclient(hunting): authenticate timed out after 300 [ceph-server02][ERROR ] 2013-10-30 10:52:30.265482 7f48a98fb700 0 librados: client.bootstrap-osd authentication error (110) Connection timed out [ceph-server02][ERROR ] Error connecting to cluster: Error [ceph-server02][ERROR ] ERROR:ceph-disk:Failed to activate After error the logs @ node1(Monitor) & node2(OSD) doesn't change, only logs in ceph-deploy show less ceph.log | grep '2013-10-30' | more 2013-10-30 14:04:26,090 [ceph_deploy.cli][INFO ] Invoked (1.2.7): /usr/bin/ceph-deploy disk list ceph-server02 2013-10-30 14:04:26,111 [ceph_deploy.sudo_pushy][DEBUG ] will use a remote connection with sudo 2013-10-30 14:04:26,818 [ceph_deploy.osd][INFO ] Distro info: CentOS 6.4 Final 2013-10-30 14:04:26,819 [ceph_deploy.osd][DEBUG ] Listing disks on ceph-server02... 2013-10-30 14:04:26,819 [ceph-server02][INFO ] Running command: ceph-disk list 2013-10-30 14:04:27,807 [ceph-server02][INFO ] /dev/sda : 2013-10-30 14:04:27,808 [ceph-server02][INFO ] /dev/sda1 other, ext4, mounted on /boot 2013-10-30 14:04:27,808 [ceph-server02][INFO ] /dev/sda2 other, LVM2_member 2013-10-30 14:04:27,808 [ceph-server02][INFO ] /dev/sdb : 2013-10-30 14:04:27,809 [ceph-server02][INFO ] /dev/sdb1 ceph data, prepared, cluster ceph, journal /dev/sdb2 2013-10-30 14:04:27,809 [ceph-server02][INFO ] /dev/sdb2 ceph journal, for /dev/sdb1 2013-10-30 14:04:27,809 [ceph-server02][INFO ] /dev/sr0 other, unknown 2013-10-30 14:06:15,558 [ceph_deploy.cli][INFO ] Invoked (1.2.7): /usr/bin/ceph-deploy osd activate ceph-server02:/dev/sdb 2013-10-30 14:06:15,559 [ceph_deploy.osd][DEBUG ] Activating cluster ceph disks ceph-server02:/dev/sdb: 2013-10-30 14:06:15,560 [ceph_deploy.sudo_pushy][DEBUG ] will use a remote connection with sudo 2013-10-30 14:06:15,953 [ceph_deploy.osd][INFO ] Distro info: CentOS 6.4 Final 2013-10-30 14:06:15,954 [ceph_deploy.osd][DEBUG ] activating host ceph-server02 disk /dev/sdb 2013-10-30 14:06:15,954 [ceph_deploy.osd][DEBUG ] will use init type: sysvinit 2013-10-30 14:06:15,955 [ceph-server02][INFO ] Running command: ceph-disk-activate --mark-init sysvinit --mount /dev/sdb 2013-10-30 14:06:16,615 [ceph-server02][ERROR ] ERROR:ceph-disk:Failed to activate 2013-10-30 14:06:23,805 [ceph_deploy.cli][INFO ] Invoked (1.2.7): /usr/bin/ceph-deploy osd activate ceph-server02:/dev/sdb1 2013-10-30 14:06:23,806 [ceph_deploy.osd][DEBUG ] Activating cluster ceph disks ceph-server02:/dev/sdb1: 2013-10-30 14:06:23,806 [ceph_deploy.sudo_pushy][DEBUG ] will use a remote connection with sudo 2013-10-30 14:06:24,199 [ceph_deploy.osd][INFO ] Distro info: CentOS 6.4 Final 2013-10-30 14:06:24,200 [ceph_deploy.osd][DEBUG ] activating host ceph-server02 disk /dev/sdb1 2013-10-30 14:06:24,200 [ceph_deploy.osd][DEBUG ] will use init type: sysvinit 2013-10-30 14:06:24,200 [ceph-server02][INFO ] Running command: ceph-disk-activate --mark-init sysvinit --mount /dev/sdb1 2013-10-30 14:11:25,115 [ceph-server02][ERROR ] 2013-10-30 10:47:30.266005 7f48a826a700 0 -- :/1002416 >> 192.168.115.91:6789/0 pipe(0x7f48a4024480 sd=9 :0 s=1 pgs=0 cs=0 l=1 c=0x7f48a40246e0).fault 2013-10-30 14:11:25,115 [ceph-server02][ERROR ] 2013-10-30 10:47:33.266367 7f48a8169700 0 -- :/1002416 >> 192.168.115.91:6789/0 pipe(0x7f4898000c00 sd=10 :0 s=1 pgs=0 cs=0 l=1 c=0x7f4898000e60).fault 2013-10-30 14:11:25,116 [ceph-server02][ERROR ] 2013-10-30 10:47:36.267392 7f48a826a700 0 -- :/1002416 >> 192.168.115.91:6789/0 pipe(0x7f4898003010 sd=10 :0 s=1 pgs=0 cs=0 l=1 c=0x7f4898003270).fault 2013-10-30 14:11:25,116 [ceph-server02][ERROR ] 2013-10-30 10:47:39.267733 7f48a8169700 0 -- :/1002416 >> 192.168.115.91:6789/0 pipe(0x7f4898003850 sd=10 :0 s=1 pgs=0 cs=0 l=1 c=0x7f4898003ab0).fault 2013-10-30 14:11:25,116 [ceph-server02][ERROR ] 2013-10-30 10:47:42.268123 7f48a826a700 0 -- :/1002416 >> 192.168.115.91:6789/0 pipe(0x7f48980025d0 sd=10 :0 s=1 pgs=0 cs=0 l=1 c=0x7f4898002830).fault 2013-10-30 14:11:25,139 [ceph-server02][ERROR ] 2013-10-30 10:51:42.305798 7f48a826a700 0 -- :/1002416 >> 192.168.115.91:6789/0 pipe(0x7f4898004430 sd=10 :0 s=1 pgs=0 cs=0 l=1 c=0x7f489800c940).fault 2013-10-30 14:11:25,139 [ceph-server02][ERROR ] 2013-10-30 10:51:45.305590 7f48a8169700 0 -- :/1002416 >> 192.168.115.91:6789/0 pipe(0x7f489800cae0 sd=10 :0 s=1 pgs=0 cs=0 l=1 c=0x7f489800df70).fault 2013-10-30 14:11:25,140 [ceph-server02][ERROR ] 2013-10-30 10:51:48.305889 7f48a826a700 0 -- :/1002416 >> 192.168.115.91:6789/0 pipe(0x7f489800c450 sd=10 :0 s=1 pgs=0 cs=0 l=1 c=0x7f489800c6b0).fault 2013-10-30 14:11:25,140 [ceph-server02][ERROR ] 2013-10-30 10:51:51.306287 7f48a8169700 0 -- :/1002416 >> 192.168.115.91:6789/0 pipe(0x7f489800cae0 sd=10 :0 s=1 pgs=0 cs=0 l=1 c=0x7f489800df70).fault 2013-10-30 14:11:25,141 [ceph-server02][ERROR ] 2013-10-30 10:51:54.307138 7f48a826a700 0 -- :/1002416 >> 192.168.115.91:6789/0 pipe(0x7f489800d110 sd=10 :0 s=1 pgs=0 cs=0 l=1 c=0x7f489800f100).fault 2013-10-30 14:11:25,141 [ceph-server02][ERROR ] 2013-10-30 10:51:57.307693 7f48a8169700 0 -- :/1002416 >> 192.168.115.91:6789/0 pipe(0x7f489800cae0 sd=10 :0 s=1 pgs=0 cs=0 l=1 c=0x7f489800df70).fault 2013-10-30 14:11:25,142 [ceph-server02][ERROR ] 2013-10-30 10:52:00.307526 7f48a826a700 0 -- :/1002416 >> 192.168.115.91:6789/0 pipe(0x7f489800d110 sd=10 :0 s=1 pgs=0 cs=0 l=1 c=0x7f48980008c0).fault 2013-10-30 14:11:25,142 [ceph-server02][ERROR ] 2013-10-30 10:52:03.307808 7f48a8169700 0 -- :/1002416 >> 192.168.115.91:6789/0 pipe(0x7f489800cae0 sd=10 :0 s=1 pgs=0 cs=0 l=1 c=0x7f4898000a60).fault 2013-10-30 14:11:25,142 [ceph-server02][ERROR ] 2013-10-30 10:52:06.308108 7f48a826a700 0 -- :/1002416 >> 192.168.115.91:6789/0 pipe(0x7f489800d110 sd=10 :0 s=1 pgs=0 cs=0 l=1 c=0x7f48980008c0).fault 2013-10-30 14:11:25,142 [ceph-server02][ERROR ] 2013-10-30 10:52:09.308682 7f48a8169700 0 -- :/1002416 >> 192.168.115.91:6789/0 pipe(0x7f489800cae0 sd=10 :0 s=1 pgs=0 cs=0 l=1 c=0x7f48980130e0).fault 2013-10-30 14:11:25,143 [ceph-server02][ERROR ] 2013-10-30 10:52:12.309033 7f48a826a700 0 -- :/1002416 >> 192.168.115.91:6789/0 pipe(0x7f489800f100 sd=10 :0 s=1 pgs=0 cs=0 l=1 c=0x7f4898002290).fault 2013-10-30 14:11:25,143 [ceph-server02][ERROR ] 2013-10-30 10:52:15.309477 7f48a8169700 0 -- :/1002416 >> 192.168.115.91:6789/0 pipe(0x7f489800d110 sd=10 :0 s=1 pgs=0 cs=0 l=1 c=0x7f48980130e0).fault 2013-10-30 14:11:25,143 [ceph-server02][ERROR ] 2013-10-30 10:52:18.309909 7f48a826a700 0 -- :/1002416 >> 192.168.115.91:6789/0 pipe(0x7f489800f100 sd=10 :0 s=1 pgs=0 cs=0 l=1 c=0x7f4898002290).fault 2013-10-30 14:11:25,143 [ceph-server02][ERROR ] 2013-10-30 10:52:21.310938 7f48a8169700 0 -- :/1002416 >> 192.168.115.91:6789/0 pipe(0x7f489800d110 sd=10 :0 s=1 pgs=0 cs=0 l=1 c=0x7f4898002ef0).fault 2013-10-30 14:11:25,144 [ceph-server02][ERROR ] 2013-10-30 10:52:24.310740 7f48a826a700 0 -- :/1002416 >> 192.168.115.91:6789/0 pipe(0x7f489800f100 sd=10 :0 s=1 pgs=0 cs=0 l=1 c=0x7f48980037c0).fault 2013-10-30 14:11:25,144 [ceph-server02][ERROR ] 2013-10-30 10:52:27.311184 7f48a8169700 0 -- :/1002416 >> 192.168.115.91:6789/0 pipe(0x7f489800d110 sd=10 :0 s=1 pgs=0 cs=0 l=1 c=0x7f4898002ef0).fault 2013-10-30 14:11:25,144 [ceph-server02][ERROR ] 2013-10-30 10:52:30.265401 7f48a98fb700 0 monclient(hunting): authenticate timed out after 300 2013-10-30 14:11:25,144 [ceph-server02][ERROR ] 2013-10-30 10:52:30.265482 7f48a98fb700 0 librados: client.bootstrap-osd authentication error (110) Connection timed out 2013-10-30 14:11:25,145 [ceph-server02][ERROR ] Error connecting to cluster: Error 2013-10-30 14:11:25,145 [ceph-server02][ERROR ] ERROR:ceph-disk:Failed to activate Note: in my setup and installation guide, the Admin node(ceph-deploy) is a separate server other than ceph-node1(Monitor) and ceph-node2, ceph-node3(OSD), and the admin node doesn't required ceph installation only ceph-deploy, is that right ? also admin node (ceph-deploy) gather keys from ceph-node1 (monitor node) only, right ? Regards, Nabil Naim -----Original Message----- From: Karan Singh [mailto:ksingh@xxxxxx] Sent: Wednesday, October 30, 2013 11:44 AM To: Nabil Naim Cc: ceph-users@xxxxxxxxxxxxxx Subject: Re: activate disk error Hey Nabil Reinstallation would not be a solution , during my ceph installation i done 8 time reinstallation of ceph in just 3 days , and then realised its not a solution. Anyway , lets dig your problem if you like :-) Your Logs says that some problem with connecting with cluster s=1 pgs=0 cs=0 l=1 c=0x7f0da800f3d0).fault [ceph-server02][ERROR ] 2013-10-29 21:54:47.679151 7f0db997a700 0 monclient(hunting): authenticate timed out after 300 [ceph-server02][ERROR ] 2013-10-29 21:54:47.679252 7f0db997a700 0 librados: client.bootstrap-osd authentication error (110) Connection timed out [ceph-server02][ERROR ] Error connecting to cluster: Error [ceph-server02][ERROR ] ERROR:ceph-disk:Failed to activate This problem might be related to key rings , pleas try this 1) On ceph-deploy , cd to your ceph installatino directory usually /etc/ceph or my-cluster if you have changed it. 2) scp <cluster_name>.c lient.admin.keyring ceph-server02:/etc/ceph ( or your ceph-server02 installation directory ) 3) scp /var/lib/ceph/bootstrap-osd/ceph.keyring ceph-server02:/var/lib/ceph/bootstrap-osd ( create bootstrap-osd direcrory on ceph-server02 if not there ) 4) Again try to activate your OSD , it should work PS : Check again for your server name , file name , and directory as the names would be specific to your environment. Here the plan is to move keyrings from MONITOR node to OSD node. Regards Karan Singh System Specialist Storage | CSC IT centre for science Espoo Finland karan.singh@xxxxxx ----- Original Message ----- From: "Nabil Naim" <nabil_naim@xxxxxxxxxxxxxxxxxx> To: ceph-users@xxxxxxxxxxxxxx Sent: Tuesday, 29 October, 2013 9:15:06 PM Subject: Re: activate disk error Also the prepare step done successfully [ceph@ceph-deploy my-cluster]$ ceph-deploy disk list ceph-server02 [ceph_deploy.cli][INFO ] Invoked (1.2.7): /usr/bin/ceph-deploy disk list ceph-server02 [ceph_deploy.sudo_pushy][DEBUG ] will use a remote connection with sudo [ceph_deploy.osd][INFO ] Distro info: CentOS 6.4 Final [ceph_deploy.osd][DEBUG ] Listing disks on ceph-server02... [ceph-server02][INFO ] Running command: ceph-disk list [ceph-server02][INFO ] /dev/sda : [ceph-server02][INFO ] /dev/sda1 other, ext4, mounted on /boot [ceph-server02][INFO ] /dev/sda2 other, LVM2_member [ceph-server02][INFO ] /dev/sdb : [ceph-server02][INFO ] /dev/sdb1 ceph data, prepared, cluster ceph, journal /dev/sdb2 [ceph-server02][INFO ] /dev/sdb2 ceph journal, for /dev/sdb1 [ceph-server02][INFO ] /dev/sr0 other, unknown Regards, Nabil Naim -----Original Message----- From: Nabil Naim Sent: Tuesday, October 29, 2013 9:07 PM To: 'ceph-users@xxxxxxxxxxxxxx' Subject: RE: activate disk error Nothing on the ceph-server02 log ceph-deploy osd activate ceph-server02:/dev/sdb1 s=1 pgs=0 cs=0 l=1 c=0x7f0da8013a80).fault [ceph-server02][ERROR ] 2013-10-29 21:54:38.712639 7f0db81e8700 0 -- :/1002801 >> 192.168.115.91:6789/0 pipe(0x7f0da800b350 sd=10 :0 s=1 pgs=0 cs=0 l=1 c=0x7f0da800f3d0).fault [ceph-server02][ERROR ] 2013-10-29 21:54:42.712477 7f0db82e9700 0 -- :/1002801 >> 192.168.115.91:6789/0 pipe(0x7f0da80008c0 sd=10 :0 s=1 pgs=0 cs=0 l=1 c=0x7f0da8013a50).fault [ceph-server02][ERROR ] 2013-10-29 21:54:45.713387 7f0db81e8700 0 -- :/1002801 >> 192.168.115.91:6789/0 pipe(0x7f0da800b350 sd=10 :0 s=1 pgs=0 cs=0 l=1 c=0x7f0da800f3d0).fault [ceph-server02][ERROR ] 2013-10-29 21:54:47.679151 7f0db997a700 0 monclient(hunting): authenticate timed out after 300 [ceph-server02][ERROR ] 2013-10-29 21:54:47.679252 7f0db997a700 0 librados: client.bootstrap-osd authentication error (110) Connection timed out [ceph-server02][ERROR ] Error connecting to cluster: Error [ceph-server02][ERROR ] ERROR:ceph-disk:Failed to activate only way to bypass the error Is to fully installed ceph-server02 as a new cluster and monitor and osd node :-(((( any advice ? :-( Regards, Nabil Naim -----Original Message----- From: Nabil Naim Sent: Monday, October 28, 2013 6:27 PM To: 'Sage Weil' Cc: ceph-users@xxxxxxxxxxxxxx Subject: RE: activate disk error Hi Sage, Thank you for reply I try to use implement CEPH following http://ceph.com/docs/master/start/quick-ceph-deploy/ All my servers are VMware instances, all steps working fine unless prepare/create OSD , I try ceph-deploy osd prepare ceph-node2:/tmp/osd0 ceph-node3:/tmp/osd1 and aslo I try to use extra HD ceph-deploy osd create ceph-node2:/dev/sdb1 ceph-node3:/dev/sdb1 each time in ceph-deploy osd activate I got the same error [root@ceph-deploy my-cluster]# ceph-deploy -v osd activate ceph-server02:/dev/sdb1 it gives ceph_deploy.cli][INFO ] Invoked (1.2.7): /usr/bin/ceph-deploy -v osd activate ceph-server02:/dev/sdb [ceph_deploy.osd][DEBUG ] Activating cluster ceph disks ceph-server02:/dev/sdb: [ceph_deploy.sudo_pushy][DEBUG ] will use a remote connection without sudo [ceph_deploy.osd][INFO ] Distro info: CentOS 6.2 Final [ceph_deploy.osd][DEBUG ] activating host ceph-server02 disk /dev/sdb [ceph_deploy.osd][DEBUG ] will use init type: sysvinit [ceph-server02][INFO ] Running command: ceph-disk-activate --mark-init sysvinit --mount /dev/sdb [root@ceph-deploy my-cluster]# ceph-deploy -v osd activate ceph-server02:/dev/sdb1 [ceph_deploy.cli][INFO ] Invoked (1.2.7): /usr/bin/ceph-deploy -v osd activate ceph-server02:/dev/sdb1 [ceph_deploy.osd][DEBUG ] Activating cluster ceph disks ceph-server02:/dev/sdb1: [ceph_deploy.sudo_pushy][DEBUG ] will use a remote connection without sudo [ceph_deploy.osd][INFO ] Distro info: CentOS 6.2 Final [ceph_deploy.osd][DEBUG ] activating host ceph-server02 disk /dev/sdb1 [ceph_deploy.osd][DEBUG ] will use init type: sysvinit [ceph-server02][INFO ] Running command: ceph-disk-activate --mark-init sysvinit --mount /dev/sdb1 And suspend for a while then [ceph-server02][ERROR ] 2013-10-24 18:36:56.049060 7f35c4986700 0 -- :/1006405 >> x.x.x.x:6789/0 pipe(0x7f35c0020430 sd=9 :0 s=1 pgs=0 cs=0 l=1 c=0x7f35c0020690).fault [ceph-server02][ERROR ] 2013-10-24 18:36:59.047638 7f35c4885700 0 -- :/1006405 >> x.x.x.x:6789/0 pipe(0x7f35b4000c00 sd=9 :0 s=1 pgs=0 cs=0 l=1 c=0x7f35b4000e60).fault [ceph-server02][ERROR ] 2013-10-24 18:37:02.049738 7f35c4986700 0 -- :/1006405 >> x.x.x.x:6789/0 pipe(0x7f35b4003010 sd=9 :0 s=1 pgs=0 cs=0 l=1 c=0x7f35b4003270).fault [ceph-server02][ERROR ] 2013-10-24 18:37:05.049212 7f35c4885700 0 -- :/1006405 >> x.x.x.x:6789/0 pipe(0x7f35b4003850 sd=11 :0 s=1 pgs=0 cs=0 l=1 c=0x7f35b4003ab0).fault [ceph-server02][ERROR ] 2013-10-24 18:37:08.049732 7f35c4986700 0 -- :/1006405 >> x.x.x.x:6789/0 pipe(0x7f35b40025d0 sd=11 :0 s=1 pgs=0 cs=0 l=1 c=0x7f35b4002830).fault [ceph-server02][ERROR ] 2013-10-24 18:37:11.050150 7f35c4885700 0 -- :/1006405 >> x.x.x.x:6789/0 pipe(0x7f35b4002cf0 sd=11 :0 s=1 pgs=0 cs=0 l=1 c=0x7f35b4002f50).fault [ceph-server02][ERROR ] 2013-10-24 18:37:14.050596 7f35c4986700 0 -- :/1006405 >> x.x.x.x:6789/0 pipe(0x7f35b4004110 sd=11 :0 s=1 pgs=0 cs=0 l=1 c=0x7f35b4004370).fault [ceph-server02][ERROR ] 2013-10-24 18:37:17.050835 7f35c4885700 0 -- :/1006405 >> x.x.x.x:6789/0 pipe(0x7f35b4004900 sd=11 :0 s=1 pgs=0 cs=0 l=1 c=0x7f35b4004b60).fault [ceph-server02][ERROR ] 2013-10-24 18:37:20.051166 7f35c4986700 0 -- :/1006405 >> x.x.x.x:6789/0 pipe(0x7f35b4005240 sd=11 :0 s=1 pgs=0 cs=0 l=1 c=0x7f35b40054a0).fault [ceph-server02][ERROR ] 2013-10-24 18:37:23.051520 7f35c4885700 0 -- :/1006405 >> x.x.x.x:6789/0 pipe(0x7f35b4005960 sd=11 :0 s=1 pgs=0 cs=0 l=1 c=0x7f35b4005bc0).fault [ceph-server02][ERROR ] 2013-10-24 18:37:26.051803 7f35c4986700 0 -- :/1006405 >> x.x.x.x:6789/0 pipe(0x7f35b40093b0 sd=11 :0 s=1 pgs=0 cs=0 l=1 c=0x7f35b4009610).fault [ceph-server02][ERROR ] 2013-10-24 18:37:29.052464 7f35c4885700 0 -- :/1006405 >> x.x.x.x:6789/0 pipe(0x7f35b4009a60 sd=11 :0 s=1 pgs=0 cs=0 l=1 c=0x7f35b4009cc0).fault [ceph-server02][ERROR ] 2013-10-24 18:37:32.052918 7f35c4986700 0 -- :/1006405 >> x.x.x.x:6789/0 pipe(0x7f35b400a320 sd=11 :0 s=1 pgs=0 cs=0 l=1 c=0x7f35b400a580).fault [ceph-server02][ERROR ] 2013-10-24 18:37:35.053331 7f35c4885700 0 -- :/1006405 >> x.x.x.x:6789/0 pipe(0x7f35b400ab60 sd=11 :0 s=1 pgs=0 cs=0 l=1 c=0x7f35b400adc0).fault [ceph-server02][ERROR ] 2013-10-24 18:37:38.053733 7f35c4986700 0 -- :/1006405 >> x.x.x.x:6789/0 pipe(0x7f35b4007350 sd=11 :0 s=1 pgs=0 cs=0 l=1 c=0x7f35b40075b0).fault [ceph-server02][ERROR ] 2013-10-24 18:37:41.054145 7f35c4885700 0 -- :/1006405 >> x.x.x.x:6789/0 pipe(0x7f35b400d230 sd=11 :0 s=1 pgs=0 cs=0 l=1 c=0x7f35b400d490).fault [ceph-server02][ERROR ] 2013-10-24 18:37:44.054592 7f35c4986700 0 -- :/1006405 >> x.x.x.x:6789/0 pipe(0x7f35b400dbc0 sd=11 :0 s=1 pgs=0 cs=0 l=1 c=0x7f35b400de20).fault [ceph-server02][ERROR ] 2013-10-24 18:37:47.055107 7f35c4885700 0 -- :/1006405 >> x.x.x.x:6789/0 pipe(0x7f35b4006440 sd=11 :0 s=1 pgs=0 cs=0 l=1 c=0x7f35b40066a0).fault [ceph-server02][ERROR ] 2013-10-24 18:37:50.055587 7f35c4986700 0 -- :/1006405 >> x.x.x.x:6789/0 pipe(0x7f35b4006c30 sd=11 :0 s=1 pgs=0 cs=0 l=1 c=0x7f35b4006e90).fault [ceph-server02][ERROR ] 2013-10-24 18:37:53.055885 7f35c4885700 0 -- :/1006405 >> x.x.x.x:6789/0 pipe(0x7f35b4007c70 sd=11 :0 s=1 pgs=0 cs=0 l=1 c=0x7f35b4007ed0).fault [ceph-server02][ERROR ] 2013-10-24 18:37:56.056305 7f35c4986700 0 -- :/1006405 >> x.x.x.x:6789/0 pipe(0x7f35b40084e0 sd=11 :0 s=1 pgs=0 cs=0 l=1 c=0x7f35b4008740).fault [ceph-server02][ERROR ] 2013-10-24 18:37:59.056735 7f35c4885700 0 -- :/1006405 >> x.x.x.x:6789/0 pipe(0x7f35b4008cd0 sd=11 :0 s=1 pgs=0 cs=0 l=1 c=0x7f35b400aff0).fault [ceph-server02][ERROR ] 2013-10-24 18:38:02.057308 7f35c4986700 0 -- :/1006405 >> x.x.x.x:6789/0 pipe(0x7f35b400b4b0 sd=11 :0 s=1 pgs=0 cs=0 l=1 c=0x7f35b400b710).fault [ceph-server02][ERROR ] 2013-10-24 18:38:05.057724 7f35c4885700 0 -- :/1006405 >> x.x.x.x:6789/0 pipe(0x7f35b400bae0 sd=11 :0 s=1 pgs=0 cs=0 l=1 c=0x7f35b400bd40).fault [ceph-server02][ERROR ] 2013-10-24 18:38:08.058137 7f35c4986700 0 -- :/1006405 >> x.x.x.x:6789/0 pipe(0x7f35b400c340 sd=11 :0 s=1 pgs=0 cs=0 l=1 c=0x7f35b400c5a0).fault [ceph-server02][ERROR ] 2013-10-24 18:38:11.058621 7f35c4885700 0 -- :/1006405 >> x.x.x.x:6789/0 pipe(0x7f35b400cb30 sd=11 :0 s=1 pgs=0 cs=0 l=1 c=0x7f35b400cd90).fault [ceph-server02][ERROR ] 2013-10-24 18:38:14.059029 7f35c4986700 0 -- :/1006405 >> x.x.x.x:6789/0 pipe(0x7f35b400f2f0 sd=11 :0 s=1 pgs=0 cs=0 l=1 c=0x7f35b400f550).fault [ceph-server02][ERROR ] 2013-10-24 18:38:17.059473 7f35c4885700 0 -- :/1006405 >> x.x.x.x:6789/0 pipe(0x7f35b400e270 sd=11 :0 s=1 pgs=0 cs=0 l=1 c=0x7f35b400e4d0).fault [ceph-server02][ERROR ] 2013-10-24 18:38:20.059899 7f35c4986700 0 -- :/1006405 >> x.x.x.x:6789/0 pipe(0x7f35b400fa70 sd=11 :0 s=1 pgs=0 cs=0 l=1 c=0x7f35b400fcd0).fault [ceph-server02][ERROR ] 2013-10-24 18:38:23.060269 7f35c4885700 0 -- :/1006405 >> x.x.x.x:6789/0 pipe(0x7f35b400eb90 sd=11 :0 s=1 pgs=0 cs=0 l=1 c=0x7f35b400edf0).fault [ceph-server02][ERROR ] 2013-10-24 18:38:26.060671 7f35c4986700 0 -- :/1006405 >> x.x.x.x:6789/0 pipe(0x7f35b40110e0 sd=11 :0 s=1 pgs=0 cs=0 l=1 c=0x7f35b4011340).fault [ceph-server02][ERROR ] 2013-10-24 18:38:29.061057 7f35c4885700 0 -- :/1006405 >> x.x.x.x:6789/0 pipe(0x7f35b40008c0 sd=11 :0 s=1 pgs=0 cs=0 l=1 c=0x7f35b400ed60).fault [ceph-server02][ERROR ] 2013-10-24 18:38:32.061511 7f35c4986700 0 -- :/1006405 >> x.x.x.x:6789/0 pipe(0x7f35b4001be0 sd=11 :0 s=1 pgs=0 cs=0 l=1 c=0x7f35b40112b0).fault [ceph-server02][ERROR ] 2013-10-24 18:38:35.061779 7f35c4885700 0 -- :/1006405 >> x.x.x.x:6789/0 pipe(0x7f35b40008c0 sd=11 :0 s=1 pgs=0 cs=0 l=1 c=0x7f35b4003640).fault [ceph-server02][ERROR ] 2013-10-24 18:38:38.062252 7f35c4986700 0 -- :/1006405 >> x.x.x.x:6789/0 pipe(0x7f35b4001be0 sd=11 :0 s=1 pgs=0 cs=0 l=1 c=0x7f35b4002000).fault [ceph-server02][ERROR ] 2013-10-24 18:38:41.062654 7f35c4885700 0 -- :/1006405 >> x.x.x.x:6789/0 pipe(0x7f35b4003640 sd=11 :0 s=1 pgs=0 cs=0 l=1 c=0x7f35b40038a0).fault [ceph-server02][ERROR ] 2013-10-24 18:38:44.063195 7f35c4986700 0 -- :/1006405 >> x.x.x.x:6789/0 pipe(0x7f35b40008c0 sd=11 :0 s=1 pgs=0 cs=0 l=1 c=0x7f35b40025d0).fault [ceph-server02][ERROR ] 2013-10-24 18:38:47.063618 7f35c4885700 0 -- :/1006405 >> x.x.x.x:6789/0 pipe(0x7f35b4003640 sd=11 :0 s=1 pgs=0 cs=0 l=1 c=0x7f35b40038a0).fault [ceph-server02][ERROR ] 2013-10-24 18:38:50.064078 7f35c4986700 0 -- :/1006405 >> x.x.x.x:6789/0 pipe(0x7f35b4001be0 sd=11 :0 s=1 pgs=0 cs=0 l=1 c=0x7f35b40008c0).fault [ceph-server02][ERROR ] 2013-10-24 18:38:53.064514 7f35c4885700 0 -- :/1006405 >> x.x.x.x:6789/0 pipe(0x7f35b4004e10 sd=11 :0 s=1 pgs=0 cs=0 l=1 c=0x7f35b4003640).fault [ceph-server02][ERROR ] 2013-10-24 18:38:56.064950 7f35c4986700 0 -- :/1006405 >> x.x.x.x:6789/0 pipe(0x7f35b4001be0 sd=11 :0 s=1 pgs=0 cs=0 l=1 c=0x7f35b40008c0).fault [ceph-server02][ERROR ] 2013-10-24 18:38:59.065660 7f35c4885700 0 -- :/1006405 >> x.x.x.x:6789/0 pipe(0x7f35b4004e10 sd=9 :0 s=1 pgs=0 cs=0 l=1 c=0x7f35b40090e0).fault [ceph-server02][ERROR ] 2013-10-24 18:39:02.065752 7f35c4986700 0 -- :/1006405 >> x.x.x.x:6789/0 pipe(0x7f35b4001be0 sd=11 :0 s=1 pgs=0 cs=0 l=1 c=0x7f35b40040a0).fault [ceph-server02][ERROR ] 2013-10-24 18:39:05.066009 7f35c4885700 0 -- :/1006405 >> x.x.x.x:6789/0 pipe(0x7f35b4004e10 sd=11 :0 s=1 pgs=0 cs=0 l=1 c=0x7f35b4011850).fault [ceph-server02][ERROR ] 2013-10-24 18:39:08.066380 7f35c4986700 0 -- :/1006405 >> x.x.x.x:6789/0 pipe(0x7f35b4001be0 sd=11 :0 s=1 pgs=0 cs=0 l=1 c=0x7f35b400a110).fault [ceph-server02][ERROR ] 2013-10-24 18:39:11.066675 7f35c4885700 0 -- :/1006405 >> x.x.x.x:6789/0 pipe(0x7f35b4004e10 sd=11 :0 s=1 pgs=0 cs=0 l=1 c=0x7f35b400d090).fault [ceph-server02][ERROR ] 2013-10-24 18:39:14.066973 7f35c4986700 0 -- :/1006405 >> x.x.x.x:6789/0 pipe(0x7f35b4001be0 sd=11 :0 s=1 pgs=0 cs=0 l=1 c=0x7f35b400d9b0).fault [ceph-server02][ERROR ] 2013-10-24 18:39:17.067354 7f35c4885700 0 -- :/1006405 >> x.x.x.x:6789/0 pipe(0x7f35b4004e10 sd=11 :0 s=1 pgs=0 cs=0 l=1 c=0x7f35b4007280).fault [ceph-server02][ERROR ] 2013-10-24 18:39:20.067692 7f35c4986700 0 -- :/1006405 >> x.x.x.x:6789/0 pipe(0x7f35b4001be0 sd=11 :0 s=1 pgs=0 cs=0 l=1 c=0x7f35b400d160).fault [ceph-server02][ERROR ] 2013-10-24 18:39:23.068050 7f35c4885700 0 -- :/1006405 >> x.x.x.x:6789/0 pipe(0x7f35b4004e10 sd=11 :0 s=1 pgs=0 cs=0 l=1 c=0x7f35b4007930).fault [ceph-server02][ERROR ] 2013-10-24 18:39:26.068397 7f35c4986700 0 -- :/1006405 >> x.x.x.x:6789/0 pipe(0x7f35b4001be0 sd=11 :0 s=1 pgs=0 cs=0 l=1 c=0x7f35b4008340).fault [ceph-server02][ERROR ] 2013-10-24 18:39:29.068753 7f35c4885700 0 -- :/1006405 >> x.x.x.x:6789/0 pipe(0x7f35b4004e10 sd=11 :0 s=1 pgs=0 cs=0 l=1 c=0x7f35b4008e70).fault [ceph-server02][ERROR ] 2013-10-24 18:39:32.069143 7f35c4986700 0 -- :/1006405 >> x.x.x.x:6789/0 pipe(0x7f35b4001be0 sd=11 :0 s=1 pgs=0 cs=0 l=1 c=0x7f35b400db50).fault [ceph-server02][ERROR ] 2013-10-24 18:39:35.069420 7f35c4885700 0 -- :/1006405 >> x.x.x.x:6789/0 pipe(0x7f35b4008c60 sd=11 :0 s=1 pgs=0 cs=0 l=1 c=0x7f35b40089f0).fault [ceph-server02][ERROR ] 2013-10-24 18:39:38.069956 7f35c4986700 0 -- :/1006405 >> x.x.x.x:6789/0 pipe(0x7f35b4001be0 sd=11 :0 s=1 pgs=0 cs=0 l=1 c=0x7f35b400b370).fault [ceph-server02][ERROR ] 2013-10-24 18:39:41.070217 7f35c4885700 0 -- :/1006405 >> x.x.x.x:6789/0 pipe(0x7f35b4004e10 sd=11 :0 s=1 pgs=0 cs=0 l=1 c=0x7f35b40089f0).fault [ceph-server02][ERROR ] 2013-10-24 18:39:44.070467 7f35c4986700 0 -- :/1006405 >> x.x.x.x:6789/0 pipe(0x7f35b4001be0 sd=11 :0 s=1 pgs=0 cs=0 l=1 c=0x7f35b400b370).fault [ceph-server02][ERROR ] 2013-10-24 18:39:47.070721 7f35c4885700 0 -- :/1006405 >> x.x.x.x:6789/0 pipe(0x7f35b4004e10 sd=11 :0 s=1 pgs=0 cs=0 l=1 c=0x7f35b400cb90).fault [ceph-server02][ERROR ] 2013-10-24 18:39:50.071074 7f35c4986700 0 -- :/1006405 >> x.x.x.x:6789/0 pipe(0x7f35b4011d50 sd=11 :0 s=1 pgs=0 cs=0 l=1 c=0x7f35b400b370).fault [ceph-server02][ERROR ] 2013-10-24 18:39:53.071415 7f35c4885700 0 -- :/1006405 >> x.x.x.x:6789/0 pipe(0x7f35b4004e10 sd=11 :0 s=1 pgs=0 cs=0 l=1 c=0x7f35b400e920).fault [ceph-server02][ERROR ] 2013-10-24 18:39:56.071697 7f35c4986700 0 -- :/1006405 >> x.x.x.x:6789/0 pipe(0x7f35b4011d50 sd=11 :0 s=1 pgs=0 cs=0 l=1 c=0x7f35b400fa70).fault [ceph-server02][ERROR ] 2013-10-24 18:39:59.072149 7f35c4885700 0 -- :/1006405 >> x.x.x.x:6789/0 pipe(0x7f35b4004e10 sd=11 :0 s=1 pgs=0 cs=0 l=1 c=0x7f35b400cb90).fault [ceph-server02][ERROR ] 2013-10-24 18:40:02.072619 7f35c4986700 0 -- :/1006405 >> x.x.x.x:6789/0 pipe(0x7f35b4011d50 sd=11 :0 s=1 pgs=0 cs=0 l=1 c=0x7f35b400e7d0).fault [ceph-server02][ERROR ] 2013-10-24 18:40:05.072983 7f35c4885700 0 -- :/1006405 >> x.x.x.x:6789/0 pipe(0x7f35b4004e10 sd=11 :0 s=1 pgs=0 cs=0 l=1 c=0x7f35b4000c00).fault [ceph-server02][ERROR ] 2013-10-24 18:40:08.073397 7f35c4986700 0 -- :/1006405 >> x.x.x.x:6789/0 pipe(0x7f35b4011d50 sd=11 :0 s=1 pgs=0 cs=0 l=1 c=0x7f35b4001e50).fault [ceph-server02][ERROR ] 2013-10-24 18:40:11.073780 7f35c4885700 0 -- :/1006405 >> x.x.x.x:6789/0 pipe(0x7f35b4004e10 sd=11 :0 s=1 pgs=0 cs=0 l=1 c=0x7f35b4003d60).fault [ceph-server02][ERROR ] 2013-10-24 18:40:14.074099 7f35c4986700 0 -- :/1006405 >> x.x.x.x:6789/0 pipe(0x7f35b4011d50 sd=11 :0 s=1 pgs=0 cs=0 l=1 c=0x7f35b4003a10).fault [ceph-server02][ERROR ] 2013-10-24 18:40:17.074457 7f35c4885700 0 -- :/1006405 >> x.x.x.x:6789/0 pipe(0x7f35b4004e10 sd=11 :0 s=1 pgs=0 cs=0 l=1 c=0x7f35b4003d60).fault [ceph-server02][ERROR ] 2013-10-24 18:40:20.074818 7f35c4986700 0 -- :/1006405 >> x.x.x.x:6789/0 pipe(0x7f35b4011d50 sd=11 :0 s=1 pgs=0 cs=0 l=1 c=0x7f35b4004890).fault [ceph-server02][ERROR ] 2013-10-24 18:40:23.075174 7f35c4885700 0 -- :/1006405 >> x.x.x.x:6789/0 pipe(0x7f35b4004e10 sd=11 :0 s=1 pgs=0 cs=0 l=1 c=0x7f35b4003d60).fault [ceph-server02][ERROR ] 2013-10-24 18:40:26.075475 7f35c4986700 0 -- :/1006405 >> x.x.x.x:6789/0 pipe(0x7f35b4011d50 sd=11 :0 s=1 pgs=0 cs=0 l=1 c=0x7f35b4004890).fault [ceph-server02][ERROR ] 2013-10-24 18:40:29.075947 7f35c4885700 0 -- :/1006405 >> x.x.x.x:6789/0 pipe(0x7f35b4004e10 sd=11 :0 s=1 pgs=0 cs=0 l=1 c=0x7f35b40058f0).fault [ceph-server02][ERROR ] 2013-10-24 18:40:32.076433 7f35c4986700 0 -- :/1006405 >> x.x.x.x:6789/0 pipe(0x7f35b4011d50 sd=11 :0 s=1 pgs=0 cs=0 l=1 c=0x7f35b4002c90).fault [ceph-server02][ERROR ] 2013-10-24 18:40:35.076933 7f35c4885700 0 -- :/1006405 >> x.x.x.x:6789/0 pipe(0x7f35b4004e10 sd=11 :0 s=1 pgs=0 cs=0 l=1 c=0x7f35b40090e0).fault [ceph-server02][ERROR ] 2013-10-24 18:40:38.077319 7f35c4986700 0 -- :/1006405 >> x.x.x.x:6789/0 pipe(0x7f35b4010210 sd=11 :0 s=1 pgs=0 cs=0 l=1 c=0x7f35b4010470).fault [ceph-server02][ERROR ] 2013-10-24 18:40:41.077636 7f35c4885700 0 -- :/1006405 >> x.x.x.x:6789/0 pipe(0x7f35b4004e10 sd=11 :0 s=1 pgs=0 cs=0 l=1 c=0x7f35b40040a0).fault [ceph-server02][ERROR ] 2013-10-24 18:40:44.078049 7f35c4986700 0 -- :/1006405 >> x.x.x.x:6789/0 pipe(0x7f35b4010210 sd=11 :0 s=1 pgs=0 cs=0 l=1 c=0x7f35b4010470).fault [ceph-server02][ERROR ] 2013-10-24 18:40:47.078406 7f35c4885700 0 -- :/1006405 >> x.x.x.x:6789/0 pipe(0x7f35b4004e10 sd=11 :0 s=1 pgs=0 cs=0 l=1 c=0x7f35b40040a0).fault [ceph-server02][ERROR ] 2013-10-24 18:40:50.078961 7f35c4986700 0 -- :/1006405 >> x.x.x.x:6789/0 pipe(0x7f35b4010210 sd=11 :0 s=1 pgs=0 cs=0 l=1 c=0x7f35b4010470).fault [ceph-server02][ERROR ] 2013-10-24 18:40:53.079379 7f35c4885700 0 -- :/1006405 >> x.x.x.x:6789/0 pipe(0x7f35b4004e10 sd=11 :0 s=1 pgs=0 cs=0 l=1 c=0x7f35b4006510).fault [ceph-server02][ERROR ] 2013-10-24 18:40:56.079689 7f35c4986700 0 -- :/1006405 >> x.x.x.x:6789/0 pipe(0x7f35b4010000 sd=11 :0 s=1 pgs=0 cs=0 l=1 c=0x7f35b4010260).fault [ceph-server02][ERROR ] 2013-10-24 18:40:59.080329 7f35c4885700 0 -- :/1006405 >> x.x.x.x:6789/0 pipe(0x7f35b4004e10 sd=11 :0 s=1 pgs=0 cs=0 l=1 c=0x7f35b4006510).fault [ceph-server02][ERROR ] 2013-10-24 18:41:02.080887 7f35c4986700 0 -- :/1006405 >> x.x.x.x:6789/0 pipe(0x7f35b4006bc0 sd=11 :0 s=1 pgs=0 cs=0 l=1 c=0x7f35b4007c30).fault [ceph-server02][ERROR ] 2013-10-24 18:41:05.081282 7f35c4885700 0 -- :/1006405 >> x.x.x.x:6789/0 pipe(0x7f35b4004e10 sd=11 :0 s=1 pgs=0 cs=0 l=1 c=0x7f35b400b940).fault [ceph-server02][ERROR ] 2013-10-24 18:41:08.081745 7f35c4986700 0 -- :/1006405 >> x.x.x.x:6789/0 pipe(0x7f35b4006bc0 sd=11 :0 s=1 pgs=0 cs=0 l=1 c=0x7f35b400b590).fault [ceph-server02][ERROR ] 2013-10-24 18:41:11.082121 7f35c4885700 0 -- :/1006405 >> x.x.x.x:6789/0 pipe(0x7f35b4004e10 sd=11 :0 s=1 pgs=0 cs=0 l=1 c=0x7f35b400b940).fault [ceph-server02][ERROR ] 2013-10-24 18:41:14.082484 7f35c4986700 0 -- :/1006405 >> x.x.x.x:6789/0 pipe(0x7f35b4006bc0 sd=11 :0 s=1 pgs=0 cs=0 l=1 c=0x7f35b40099e0).fault [ceph-server02][ERROR ] 2013-10-24 18:41:17.082768 7f35c4885700 0 -- :/1006405 >> x.x.x.x:6789/0 pipe(0x7f35b4004e10 sd=11 :0 s=1 pgs=0 cs=0 l=1 c=0x7f35b400bbb0).fault [ceph-server02][ERROR ] 2013-10-24 18:41:20.083182 7f35c4986700 0 -- :/1006405 >> x.x.x.x:6789/0 pipe(0x7f35b40083f0 sd=11 :0 s=1 pgs=0 cs=0 l=1 c=0x7f35b400f8d0).fault [ceph-server02][ERROR ] 2013-10-24 18:41:23.083607 7f35c4885700 0 -- :/1006405 >> x.x.x.x:6789/0 pipe(0x7f35b4004e10 sd=11 :0 s=1 pgs=0 cs=0 l=1 c=0x7f35b400fc10).fault [ceph-server02][ERROR ] 2013-10-24 18:41:26.083897 7f35c4986700 0 -- :/1006405 >> x.x.x.x:6789/0 pipe(0x7f35b40083f0 sd=11 :0 s=1 pgs=0 cs=0 l=1 c=0x7f35b400f8d0).fault [ceph-server02][ERROR ] 2013-10-24 18:41:29.084225 7f35c4885700 0 -- :/1006405 >> x.x.x.x:6789/0 pipe(0x7f35b4004e10 sd=11 :0 s=1 pgs=0 cs=0 l=1 c=0x7f35b40015f0).fault [ceph-server02][ERROR ] 2013-10-24 18:41:32.084656 7f35c4986700 0 -- :/1006405 >> x.x.x.x:6789/0 pipe(0x7f35b40083f0 sd=11 :0 s=1 pgs=0 cs=0 l=1 c=0x7f35b400eb50).fault [ceph-server02][ERROR ] 2013-10-24 18:41:35.085102 7f35c4885700 0 -- :/1006405 >> x.x.x.x:6789/0 pipe(0x7f35b4004e10 sd=11 :0 s=1 pgs=0 cs=0 l=1 c=0x7f35b40015f0).fault [ceph-server02][ERROR ] 2013-10-24 18:41:38.085421 7f35c4986700 0 -- :/1006405 >> x.x.x.x:6789/0 pipe(0x7f35b40083f0 sd=11 :0 s=1 pgs=0 cs=0 l=1 c=0x7f35b400eb50).fault [ceph-server02][ERROR ] 2013-10-24 18:41:41.085742 7f35c4885700 0 -- :/1006405 >> x.x.x.x:6789/0 pipe(0x7f35b4004e10 sd=11 :0 s=1 pgs=0 cs=0 l=1 c=0x7f35b400efa0).fault [ceph-server02][ERROR ] 2013-10-24 18:41:44.086157 7f35c4986700 0 -- :/1006405 >> x.x.x.x:6789/0 pipe(0x7f35b4006bc0 sd=11 :0 s=1 pgs=0 cs=0 l=1 c=0x7f35b400eb50).fault [ceph-server02][ERROR ] 2013-10-24 18:41:47.086679 7f35c4885700 0 -- :/1006405 >> x.x.x.x:6789/0 pipe(0x7f35b4004e10 sd=11 :0 s=1 pgs=0 cs=0 l=1 c=0x7f35b400efa0).fault [ceph-server02][ERROR ] 2013-10-24 18:41:50.087077 7f35c4986700 0 -- :/1006405 >> x.x.x.x:6789/0 pipe(0x7f35b400ff80 sd=11 :0 s=1 pgs=0 cs=0 l=1 c=0x7f35b400eb50).fault [ceph-server02][ERROR ] 2013-10-24 18:41:53.087514 7f35c4885700 0 -- :/1006405 >> x.x.x.x:6789/0 pipe(0x7f35b4004e10 sd=11 :0 s=1 pgs=0 cs=0 l=1 c=0x7f35b400efa0).fault [ceph-server02][ERROR ] 2013-10-24 18:41:56.046946 7f35c8058700 0 monclient(hunting): authenticate timed out after 300 [ceph-server02][ERROR ] 2013-10-24 18:41:56.047026 7f35c8058700 0 librados: client.bootstrap-osd authentication error (110) Connection timed out [ceph-server02][ERROR ] Error connecting to cluster: Error [ceph-server02][ERROR ] ERROR:ceph-disk:Failed to activate Note: I try as tutorial to use ceph user and root user as well with no success -----Original Message----- From: Sage Weil [mailto:sage@xxxxxxxxxxx] Sent: Monday, October 28, 2013 6:10 PM To: Nabil Naim Cc: ceph-users@xxxxxxxxxxxxxx Subject: Re: activate disk error On Mon, 28 Oct 2013, Nabil Naim wrote: > > Any one have clue why this error happen > > 2013-10-28 14:12:23.817719 7fe95437a700 0 -- :/1008986 >> > 192.168.115.91:6789/0 pipe(0x7fe944010d00 sd=5 :0 s=1 pgs=0 cs=0 l=1 > c=0x7fe9440046b0).fault > > When I try to activate disk It looks like 192.168.115.91 is one of your monitors and it was either down or there was a transient tcp connection problem. sage _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |