Hello,
I am currently testing ceph to make a replicated block device for a project that would involve 2 data servers accessing this block device, so that if one fails or crashes, the data can still be used and the cluster can be rebuilt.
This project requires that both machines run an OSD and a monitor, and that a 3rd monitor is run somewhere else, so that there is not a single point of failure.
I know it is not the best thing to run an OSD and a monitor on the same machine, but I cannot really find a better solution.
My problem is that, after having read several times and followed the documentation, I cannot succeed to add a second monitor.
I have bootstrapped a first monitor, added 2 OSDs (one on the machine with the monitor, one on the other), and I try to add a second monitor but it doesn't work.
I think I misunderstood something.
Here's what I did :
On the first machine named grenier:
# setup the configuration file /etc/ceph/ceph.conf (see content further)
# bootstrap monitor:
$ ceph-authtool --create-keyring /var/tmp/ceph.mon.keyring --gen-key -n mon. --cap mon 'allow *'
$ sudo ceph-authtool --create-keyring /etc/ceph/ceph.client.admin.keyring --gen-key -n client.admin --set-uid=0 --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow'
$ sudo chown myuser /etc/ceph/ceph.client.admin.keyring
$ ceph-authtool /var/tmp/ceph.mon.keyring --import-keyring /etc/ceph/ceph.client.admin.keyring
$ monmaptool --create --add grenier 172.16.1.11 --fsid $monuuid $tmp/monmap
$ sudo mkdir -p /var/lib/ceph/mon/ceph-grenier
$ sudo chown $ID -R /var/lib/ceph/mon/ceph-grenier
$ ceph-mon --mkfs -i grenier --monmap /var/tmp/monmap --keyring /var/tmp/ceph.mon.keyring
# start monitor:
$ sudo start ceph-mon id=grenier
# add OSD:
$ sudo ceph osd create $osduuid
$ sudo mkdir -p /var/lib/ceph/osd/ceph-0
$ sudo ceph-osd -i 0 --mkfs --mkkey --osd-uuid $osduuid
$ sudo ceph auth add osd.0 osd 'allow *' mon 'allow profile osd' -i /var/lib/ceph/osd/ceph-0/keyring
$ ceph osd crush add-bucket grenier host
$ ceph osd crush move grenier root=default
$ ceph osd crush add osd.0 1.0 host=grenier
# start this OSD
$ sudo ceph-osd -i 0
# copy /etc/ceph/ceph.conf, /etc/ceph/ceph.client.admin.keyring, /var/tmp/ceph/ceph.mon.keyring and /var/tmp/ceph/monmap from grenier to second node named gail:
# add and start OSD on the second node
$ sudo ceph osd create $newosduuid
$ sudo mkdir -p /var/lib/ceph/osd/ceph-1
$ sudo ceph-osd -i 1 --mkfs --mkkey --osd-uuid $newosduuid
$ sudo ceph auth add osd.1 osd 'allow *' mon 'allow profile osd' -i /var/lib/ceph/osd/ceph-1/keyring
$ ceph osd crush add-bucket gail host
$ ceph osd crush move gail root=default
$ ceph osd crush add osd.1 1.0 host=gail
# start this OSD
$ sudo ceph-osd -i 1
There, everything works correctly, I can create and map a block device, and then write on it and the data is replicated on both nodes.
When I perform a ceph -s I get :
cluster a98faf65-b105-4ec7-913c-f8a33a4db4d1
health HEALTH_OK
monmap e1: 1 mons at {grenier=172.16.1.11:6789/0}, election epoch 2, quorum 0 grenier
osdmap e13: 2 osds: 2 up, 2 in
pgmap v47: 192 pgs, 3 pools, 0 bytes data, 0 objects
18400 MB used,
105 GB / 129 GB avail 192 active+clean
And here is what I do when trying to add a second monitor on gail:
$ sudo mkdir -p /var/lib/ceph/mon/ceph-gail
$ ceph mon getmap -o /var/tmp/monmap
$ sudo ceph-mon -i gail --mkfs --monmap /var/tmp/monmap --keyring /var/tmp/ceph.mon.keyring
which prints:
ceph-mon: set fsid to a98faf65-b105-4ec7-913c-f8a33a4db4d1
ceph-mon: created monfs at /var/lib/ceph/mon/ceph-gail for mon.gail
which seems correct (same uuid as in ceph.conf)
$ sudo ceph-mon add gail 172.16.1.12
This command prints:
2014-09-01 17:07:26.033688 7f5538ada700 0 monclient: hunting for new mon
and hangs
Then I would like to do this:
$ sudo ceph-mon -i gail --public-addr 172.16.1.12
but it is useless as the previous command failed.
Would anybody guess what I am doing wrong ?
I use ceph 0.80 on an Ubuntu trusty.
My ceph.conf is as follows :
[global]
fsid = a98faf65-b105-4ec7-913c-f8a33a4db4d1
mon initial members = grenier
mon host = 172.16.1.11
public network = 172.16.0.0/16
auth cluster required = none
auth service required = none
auth client required = none
osd journal size = 1024
filestore xattr use omap = true
osd pool default size = 2
osd pool default min size = 1
osd pool default pg num = 333
osd pool default pgp num = 333
osd crush chooseleaf type = 1
