Oh, I forgot to say: I made a mistake in my first message, the command you suggested to remove was in fact : $ sudo ceph mon add $HOSTNAME $IP and not $ sudo ceph-mon add $HOSTNAME $IP Anyway, removing it makes the whole work. But the doc indicates to execute it. Should the doc be changed? http://ceph.com/docs/master/rados/operations/add-or-rm-mons/#adding-a-monitor-manual Pascal ---- Original message ---- Thanks, it seems to work. I also had to complete the "mon host" line in config file to add my second (and then my third) monitor : mon host = 172.16.1.11,172.16.1.12,172.16.1.13 otherwise it still didn't work when I stoppd 1 of the 3 monitors, I mean when I stopped grenier, the only one which was referenced in the config file. However, the last command you gave me seemed to produce no useful result : $ sudo ceph --admin-daemon /var/run/ceph/ceph-mon.gail.asok add_bootstrap_peer_hint 172.16.1.11 mon already active; ignoring bootstrap hint Then another detail that I find not really fine, but that is probably normal: when there is no quorum, ceph -s doesn't return before a long time (maybe 5 minutes) and prints an error message. This makes diagnostic a little harder as it is not possible to get precise infos when there is no quorum, or maybe there is a hint to do it. Thanks again Pascal Craig Lewis a ?crit, le Wed 03 Sep 2014 ? 05:26:32PM : > "monclient: hunting for new mon" happens whenever the monmap changes. It > will hang if there's no quorum. > > I haven't done this manually in a long time, so I'll refer to the Chef > recipes. The recipe doesn't do the 'ceph-mon add', it just starts the > daemon up. > > > Try: > sudo ceph-mon -i gail --mkfs --monmap /var/tmp/monmap --keyring > /var/tmp/ceph.mon.keyring > sudo ceph-mon -i gail --public-addr 172.16.1.12 > sudo ceph --admin-daemon /var/run/ceph/ceph-mon.gail.asok > add_bootstrap_peer_hint 172.16.1.11 > > > > > > On Mon, Sep 1, 2014 at 8:21 AM, Pascal GREGIS <pgs at synerway.com> wrote: > > > Hello, > > > > I am currently testing ceph to make a replicated block device for a > > project that would involve 2 data servers accessing this block device, so > > that if one fails or crashes, the data can still be used and the cluster > > can be rebuilt. > > > > This project requires that both machines run an OSD and a monitor, and > > that a 3rd monitor is run somewhere else, so that there is not a single > > point of failure. > > I know it is not the best thing to run an OSD and a monitor on the same > > machine, but I cannot really find a better solution. > > > > My problem is that, after having read several times and followed the > > documentation, I cannot succeed to add a second monitor. > > > > I have bootstrapped a first monitor, added 2 OSDs (one on the machine with > > the monitor, one on the other), and I try to add a second monitor but it > > doesn't work. > > I think I misunderstood something. > > > > Here's what I did : > > > > On the first machine named grenier: > > # setup the configuration file /etc/ceph/ceph.conf (see content further) > > # bootstrap monitor: > > $ ceph-authtool --create-keyring /var/tmp/ceph.mon.keyring --gen-key -n > > mon. --cap mon 'allow *' > > $ sudo ceph-authtool --create-keyring /etc/ceph/ceph.client.admin.keyring > > --gen-key -n client.admin --set-uid=0 --cap mon 'allow *' --cap osd 'allow > > *' --cap mds 'allow' > > $ sudo chown myuser /etc/ceph/ceph.client.admin.keyring > > $ ceph-authtool /var/tmp/ceph.mon.keyring --import-keyring > > /etc/ceph/ceph.client.admin.keyring > > $ monmaptool --create --add grenier 172.16.1.11 --fsid $monuuid $tmp/monmap > > $ sudo mkdir -p /var/lib/ceph/mon/ceph-grenier > > $ sudo chown $ID -R /var/lib/ceph/mon/ceph-grenier > > $ ceph-mon --mkfs -i grenier --monmap /var/tmp/monmap --keyring > > /var/tmp/ceph.mon.keyring > > # start monitor: > > $ sudo start ceph-mon id=grenier > > # add OSD: > > $ sudo ceph osd create $osduuid > > $ sudo mkdir -p /var/lib/ceph/osd/ceph-0 > > $ sudo ceph-osd -i 0 --mkfs --mkkey --osd-uuid $osduuid > > $ sudo ceph auth add osd.0 osd 'allow *' mon 'allow profile osd' -i > > /var/lib/ceph/osd/ceph-0/keyring > > $ ceph osd crush add-bucket grenier host > > $ ceph osd crush move grenier root=default > > $ ceph osd crush add osd.0 1.0 host=grenier > > # start this OSD > > $ sudo ceph-osd -i 0 > > > > # copy /etc/ceph/ceph.conf, /etc/ceph/ceph.client.admin.keyring, > > /var/tmp/ceph/ceph.mon.keyring and /var/tmp/ceph/monmap from grenier to > > second node named gail: > > # add and start OSD on the second node > > $ sudo ceph osd create $newosduuid > > $ sudo mkdir -p /var/lib/ceph/osd/ceph-1 > > $ sudo ceph-osd -i 1 --mkfs --mkkey --osd-uuid $newosduuid > > $ sudo ceph auth add osd.1 osd 'allow *' mon 'allow profile osd' -i > > /var/lib/ceph/osd/ceph-1/keyring > > $ ceph osd crush add-bucket gail host > > $ ceph osd crush move gail root=default > > $ ceph osd crush add osd.1 1.0 host=gail > > # start this OSD > > $ sudo ceph-osd -i 1 > > > > There, everything works correctly, I can create and map a block device, > > and then write on it and the data is replicated on both nodes. > > When I perform a ceph -s I get : > > cluster a98faf65-b105-4ec7-913c-f8a33a4db4d1 > > health HEALTH_OK > > monmap e1: 1 mons at {grenier=172.16.1.11:6789/0}, election epoch 2, > > quorum 0 grenier > > osdmap e13: 2 osds: 2 up, 2 in > > pgmap v47: 192 pgs, 3 pools, 0 bytes data, 0 objects > > 18400 MB used, > > 105 GB / 129 GB avail 192 active+clean > > > > And here is what I do when trying to add a second monitor on gail: > > $ sudo mkdir -p /var/lib/ceph/mon/ceph-gail > > $ ceph mon getmap -o /var/tmp/monmap > > $ sudo ceph-mon -i gail --mkfs --monmap /var/tmp/monmap --keyring > > /var/tmp/ceph.mon.keyring > > which prints: > > ceph-mon: set fsid to a98faf65-b105-4ec7-913c-f8a33a4db4d1 > > ceph-mon: created monfs at /var/lib/ceph/mon/ceph-gail for mon.gail > > which seems correct (same uuid as in ceph.conf) > > $ sudo ceph-mon add gail 172.16.1.12 > > This command prints: > > 2014-09-01 17:07:26.033688 7f5538ada700 0 monclient: hunting for new mon > > and hangs > > > > Then I would like to do this: > > $ sudo ceph-mon -i gail --public-addr 172.16.1.12 > > but it is useless as the previous command failed. > > > > > > Would anybody guess what I am doing wrong ? > > > > I use ceph 0.80 on an Ubuntu trusty. > > My ceph.conf is as follows : > > [global] > > fsid = a98faf65-b105-4ec7-913c-f8a33a4db4d1 > > mon initial members = grenier > > mon host = 172.16.1.11 > > public network = 172.16.0.0/16 > > auth cluster required = none > > auth service required = none > > auth client required = none > > osd journal size = 1024 > > filestore xattr use omap = true > > osd pool default size = 2 > > osd pool default min size = 1 > > osd pool default pg num = 333 > > osd pool default pgp num = 333 > > osd crush chooseleaf type = 1 > > _______________________________________________ > > ceph-users mailing list > > ceph-users at lists.ceph.com > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > >
 |