(no subject)

James Gallagher <james.np.gallagher@xxxxxxxxx> · Thu, 12 Nov 2015 14:21:11 +0000

Hi, I'm having issues activating my OSDs. I have provided the output of the fault. I can see that the error message has said that the connection is timing out however, I am struggling to understand why as I have followed each stage within the quick start guide. For example, I can ping node1 (which is the monitor) from all nodes and I can ssh w/o password to it too. Other things like require tty has been disabled. I have also opened the ports on the firewall for 6789 and 6800-7300. I have also ensured that the ceph-deploy mon create-initial command has been issued too.

Is there anything else that could possibly be preventing the monitor node from communicating?

Furthermore, I tried a reboot of all nodes. Once the nodes came online again, I tried again. However, on this occasion I am with an error due to a fsid mismatch. It should be noted that I retried the mon-create initial command with the --overwrite option to get past another error.

Issue 1
[ceph_deploy.osd][DEBUG ] Activating cluster ceph disks node2:/var/local/osd0: node3:/var/local/osd1:
[node2][DEBUG ] connection detected need for sudo
[node2][DEBUG ] connected to host: node2 
[node2][DEBUG ] detect platform information from remote host
[node2][DEBUG ] detect machine type
[node2][DEBUG ] find the location of an executable
[ceph_deploy.osd][INFO  ] Distro info: CentOS Linux 7.1.1503 Core
[ceph_deploy.osd][DEBUG ] activating host node2 disk /var/local/osd0
[ceph_deploy.osd][DEBUG ] will use init type: sysvinit
[node2][INFO  ] Running command: sudo ceph-disk -v activate --mark-init sysvinit --mount /var/local/osd0
[node2][WARNING] DEBUG:ceph-disk:Cluster uuid is 5f94aba1-e6e1-41e0-bfa1-4030cf24fda8
[node2][WARNING] INFO:ceph-disk:Running command: /usr/bin/ceph-osd --cluster=ceph --show-config-value=fsid
[node2][WARNING] DEBUG:ceph-disk:Cluster name is ceph
[node2][WARNING] DEBUG:ceph-disk:OSD uuid is 183274c6-4941-4f47-97f9-b7bbcd0bdbac
[node2][WARNING] DEBUG:ceph-disk:Allocating OSD id...
[node2][WARNING] INFO:ceph-disk:Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd create --concise 183274c6-4941-4f47-97f9-b7bbcd0bdbac
[node2][WARNING] 2015-11-10 19:43:36.433091 7fd6717fa700  0 -- :/1019898 >> 192.168.43.11:6789/0 pipe(0x7fd67405e280 sd=4 :0 s=1 pgs=0 cs=0 l=1 c=0x7fd67405af80).fault
[node2][WARNING] 2015-11-10 19:43:39.434067 7fd6716f9700  0 -- :/1019898 >> 192.168.43.11:6789/0 pipe(0x7fd668000c00 sd=4 :0 s=1 pgs=0 cs=0 l=1 c=0x7fd668004ef0).fault
[node2][WARNING] 2015-11-10 19:43:42.435321 7fd6717fa700  0 -- :/1019898 >> 192.168.43.11:6789/0 pipe(0x7fd6680081b0 sd=4 :0 s=1 pgs=0 cs=0 l=1 c=0x7fd66800c450).fault
[node2][WARNING] 2015-11-10 19:43:45.436626 7fd6716f9700  0 -- :/1019898 >> 192.168.43.11:6789/0 pipe(0x7fd668000c00 sd=4 :0 s=1 pgs=0 cs=0 l=1 c=0x7fd668006610).fault
[node2][WARNING] 2015-11-10 19:43:48.439097 7fd6717fa700  0 -- :/1019898 >> 192.168.43.11:6789/0 pipe(0x7fd6680081b0 sd=4 :0 s=1 pgs=0 cs=0 l=1 c=0x7fd6680058b0).fault
[node2][WARNING] 2015-11-10 19:43:51.441654 7fd6716f9700  0 -- :/1019898 >> 192.168.43.11:6789/0 pipe(0x7fd668000c00 sd=4 :0 s=1 pgs=0 cs=0 l=1 c=0x7fd668006f80).fault
[node2][WARNING] 2015-11-10 19:43:54.444116 7fd6717fa700  0 -- :/1019898 >> 192.168.43.11:6789/0 pipe(0x7fd6680081b0 sd=4 :0 s=1 pgs=0 cs=0 l=1 c=0x7fd668007640).fault
[node2][WARNING] 2015-11-10 19:43:57.446729 7fd6716f9700  0 -- :/1019898 >> 192.168.43.11:6789/0 pipe(0x7fd668000c00 sd=4 :0 s=1 pgs=0 cs=0 l=1 c=0x7fd668007120).fault

...

...

[node2][WARNING] 2015-11-06 02:18:52.138230 7f819c327700  0 -- :/1020092 >> 192.168.107.11:6789/0 pipe(0x7f81900081b0 sd=4 :0 s=1 pgs=0 cs=0 l=1 c=0x7f8190016fa0).fault
[node2][WARNING] 2015-11-06 02:23:34.971917 7f819c327700  0 -- :/1020092 >> 192.168.107.11:6789/0 pipe(0x7f8190007c10 sd=4 :0 s=1 pgs=0 cs=0 l=1 c=0x7f81900120e0).fault
[node2][WARNING] 2015-11-06 02:23:38.340389 7f819c226700  0 -- :/1020092 >> 192.168.107.11:6789/0 pipe(0x7f81900008c0 sd=4 :0 s=1 pgs=0 cs=0 l=1 c=0x7f8190016fa0).fault
[node2][WARNING] 2015-11-06 02:23:40.117327 7f819ea9c700  0 monclient(hunting): authenticate timed out after 300
[node2][WARNING] 2015-11-06 02:23:40.117766 7f819ea9c700  0 librados: client.bootstrap-osd authentication error (110) Connection timed out
[node2][WARNING] Error connecting to cluster: TimedOut
[node2][WARNING] ceph-disk: Error: ceph osd create failed: Command '/usr/bin/ceph' returned non-zero exit status 1: 
[node2][ERROR ] RuntimeError: command returned non-zero exit status: 1
[ceph_deploy][ERROR ] RuntimeError: Failed to execute command: ceph-disk -v activate --mark-init sysvinit --mount /var/local/osd0

Issue 2
[ceph_deploy.conf][DEBUG ] found configuration file at: /home/user/.cephdeploy.conf
[ceph_deploy.cli][INFO  ] Invoked (1.5.28): /bin/ceph-deploy --overwrite osd activate node2:/var/local/osd0 node3:/var/local/osd1
[ceph_deploy.cli][INFO  ] ceph-deploy options:
[ceph_deploy.cli][INFO  ]  username                      : None
[ceph_deploy.cli][INFO  ]  verbose                       : False
[ceph_deploy.cli][INFO  ]  overwrite_conf                : True
[ceph_deploy.cli][INFO  ]  subcommand                    : activate
[ceph_deploy.cli][INFO  ]  quiet                         : False
[ceph_deploy.cli][INFO  ]  cd_conf                       : <ceph_deploy.conf.cephdeploy.Conf instance at 0x12c4680>
[ceph_deploy.cli][INFO  ]  cluster                       : ceph
[ceph_deploy.cli][INFO  ]  func                          : <function osd at 0x12b92a8>
[ceph_deploy.cli][INFO  ]  ceph_conf                     : None
[ceph_deploy.cli][INFO  ]  default_release               : False
[ceph_deploy.cli][INFO  ]  disk                          : [('node2', '/var/local/osd0', None), ('node3', '/var/local/osd1', None)]
[ceph_deploy.osd][DEBUG ] Activating cluster ceph disks node2:/var/local/osd0: node3:/var/local/osd1:
[node2][DEBUG ] connection detected need for sudo
[node2][DEBUG ] connected to host: node2 
[node2][DEBUG ] detect platform information from remote host
[node2][DEBUG ] detect machine type
[node2][DEBUG ] find the location of an executable
[ceph_deploy.osd][INFO  ] Distro info: CentOS Linux 7.1.1503 Core
[ceph_deploy.osd][DEBUG ] activating host node2 disk /var/local/osd0
[ceph_deploy.osd][DEBUG ] will use init type: sysvinit
[node2][INFO  ] Running command: sudo ceph-disk -v activate --mark-init sysvinit --mount /var/local/osd0
[node2][WARNING] DEBUG:ceph-disk:Cluster uuid is 5f94aba1-e6e1-41e0-bfa1-4030cf24fda8
[node2][WARNING] INFO:ceph-disk:Running command: /usr/bin/ceph-osd --cluster=ceph --show-config-value=fsid
[node2][WARNING] ceph-disk: Error: No cluster conf found in /etc/ceph with fsid 5f94aba1-e6e1-41e0-bfa1-4030cf24fda8
[node2][ERROR ] RuntimeError: command returned non-zero exit status: 1
[ceph_deploy][ERROR ] RuntimeError: Failed to execute command: ceph-disk -v activate --mark-init sysvinit --mount /var/local/osd0
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com