Thanks, it seems to work.
I also had to complete the "mon host" line in config file to add my second (and then my third) monitor :
mon host = 172.16.1.11,172.16.1.12,172.16.1.13
otherwise it still didn't work when I stoppd 1 of the 3 monitors, I mean when I stopped grenier, the only one which was referenced in the config file.
However, the last command you gave me seemed to produce no useful result :
$ sudo ceph --admin-daemon /var/run/ceph/ceph-mon.gail.asok add_bootstrap_peer_hint 172.16.1.11
mon already active; ignoring bootstrap hint
Then another detail that I find not really fine, but that is probably normal: when there is no quorum, ceph -s doesn't return before a long time (maybe 5 minutes) and prints an error message.
This makes diagnostic a little harder as it is not possible to get precise infos when there is no quorum, or maybe there is a hint to do it.
Thanks again
Pascal
Craig Lewis a ?crit, le Wed 03 Sep 2014 ? 05:26:32PM :
> "monclient: hunting for new mon" happens whenever the monmap changes. It
> will hang if there's no quorum.
>
> I haven't done this manually in a long time, so I'll refer to the Chef
> recipes. The recipe doesn't do the 'ceph-mon add', it just starts the
> daemon up.
>
>
> Try:
> sudo ceph-mon -i gail --mkfs --monmap /var/tmp/monmap --keyring
> /var/tmp/ceph.mon.keyring
> sudo ceph-mon -i gail --public-addr 172.16.1.12
> sudo ceph --admin-daemon /var/run/ceph/ceph-mon.gail.asok
> add_bootstrap_peer_hint 172.16.1.11
>
>
>
>
>
> On Mon, Sep 1, 2014 at 8:21 AM, Pascal GREGIS <pgs at synerway.com> wrote:
>
> > Hello,
> >
> > I am currently testing ceph to make a replicated block device for a
> > project that would involve 2 data servers accessing this block device, so
> > that if one fails or crashes, the data can still be used and the cluster
> > can be rebuilt.
> >
> > This project requires that both machines run an OSD and a monitor, and
> > that a 3rd monitor is run somewhere else, so that there is not a single
> > point of failure.
> > I know it is not the best thing to run an OSD and a monitor on the same
> > machine, but I cannot really find a better solution.
> >
> > My problem is that, after having read several times and followed the
> > documentation, I cannot succeed to add a second monitor.
> >
> > I have bootstrapped a first monitor, added 2 OSDs (one on the machine with
> > the monitor, one on the other), and I try to add a second monitor but it
> > doesn't work.
> > I think I misunderstood something.
> >
> > Here's what I did :
> >
> > On the first machine named grenier:
> > # setup the configuration file /etc/ceph/ceph.conf (see content further)
> > # bootstrap monitor:
> > $ ceph-authtool --create-keyring /var/tmp/ceph.mon.keyring --gen-key -n
> > mon. --cap mon 'allow *'
> > $ sudo ceph-authtool --create-keyring /etc/ceph/ceph.client.admin.keyring
> > --gen-key -n client.admin --set-uid=0 --cap mon 'allow *' --cap osd 'allow
> > *' --cap mds 'allow'
> > $ sudo chown myuser /etc/ceph/ceph.client.admin.keyring
> > $ ceph-authtool /var/tmp/ceph.mon.keyring --import-keyring
> > /etc/ceph/ceph.client.admin.keyring
> > $ monmaptool --create --add grenier 172.16.1.11 --fsid $monuuid $tmp/monmap
> > $ sudo mkdir -p /var/lib/ceph/mon/ceph-grenier
> > $ sudo chown $ID -R /var/lib/ceph/mon/ceph-grenier
> > $ ceph-mon --mkfs -i grenier --monmap /var/tmp/monmap --keyring
> > /var/tmp/ceph.mon.keyring
> > # start monitor:
> > $ sudo start ceph-mon id=grenier
> > # add OSD:
> > $ sudo ceph osd create $osduuid
> > $ sudo mkdir -p /var/lib/ceph/osd/ceph-0
> > $ sudo ceph-osd -i 0 --mkfs --mkkey --osd-uuid $osduuid
> > $ sudo ceph auth add osd.0 osd 'allow *' mon 'allow profile osd' -i
> > /var/lib/ceph/osd/ceph-0/keyring
> > $ ceph osd crush add-bucket grenier host
> > $ ceph osd crush move grenier root=default
> > $ ceph osd crush add osd.0 1.0 host=grenier
> > # start this OSD
> > $ sudo ceph-osd -i 0
> >
> > # copy /etc/ceph/ceph.conf, /etc/ceph/ceph.client.admin.keyring,
> > /var/tmp/ceph/ceph.mon.keyring and /var/tmp/ceph/monmap from grenier to
> > second node named gail:
> > # add and start OSD on the second node
> > $ sudo ceph osd create $newosduuid
> > $ sudo mkdir -p /var/lib/ceph/osd/ceph-1
> > $ sudo ceph-osd -i 1 --mkfs --mkkey --osd-uuid $newosduuid
> > $ sudo ceph auth add osd.1 osd 'allow *' mon 'allow profile osd' -i
> > /var/lib/ceph/osd/ceph-1/keyring
> > $ ceph osd crush add-bucket gail host
> > $ ceph osd crush move gail root=default
> > $ ceph osd crush add osd.1 1.0 host=gail
> > # start this OSD
> > $ sudo ceph-osd -i 1
> >
> > There, everything works correctly, I can create and map a block device,
> > and then write on it and the data is replicated on both nodes.
> > When I perform a ceph -s I get :
> > cluster a98faf65-b105-4ec7-913c-f8a33a4db4d1
> > health HEALTH_OK
> > monmap e1: 1 mons at {grenier=172.16.1.11:6789/0}, election epoch 2,
> > quorum 0 grenier
> > osdmap e13: 2 osds: 2 up, 2 in
> > pgmap v47: 192 pgs, 3 pools, 0 bytes data, 0 objects
> > 18400 MB used,
> > 105 GB / 129 GB avail 192 active+clean
> >
> > And here is what I do when trying to add a second monitor on gail:
> > $ sudo mkdir -p /var/lib/ceph/mon/ceph-gail
> > $ ceph mon getmap -o /var/tmp/monmap
> > $ sudo ceph-mon -i gail --mkfs --monmap /var/tmp/monmap --keyring
> > /var/tmp/ceph.mon.keyring
> > which prints:
> > ceph-mon: set fsid to a98faf65-b105-4ec7-913c-f8a33a4db4d1
> > ceph-mon: created monfs at /var/lib/ceph/mon/ceph-gail for mon.gail
> > which seems correct (same uuid as in ceph.conf)
> > $ sudo ceph-mon add gail 172.16.1.12
> > This command prints:
> > 2014-09-01 17:07:26.033688 7f5538ada700 0 monclient: hunting for new mon
> > and hangs
> >
> > Then I would like to do this:
> > $ sudo ceph-mon -i gail --public-addr 172.16.1.12
> > but it is useless as the previous command failed.
> >
> >
> > Would anybody guess what I am doing wrong ?
> >
> > I use ceph 0.80 on an Ubuntu trusty.
> > My ceph.conf is as follows :
> > [global]
> > fsid = a98faf65-b105-4ec7-913c-f8a33a4db4d1
> > mon initial members = grenier
> > mon host = 172.16.1.11
> > public network = 172.16.0.0/16
> > auth cluster required = none
> > auth service required = none
> > auth client required = none
> > osd journal size = 1024
> > filestore xattr use omap = true
> > osd pool default size = 2
> > osd pool default min size = 1
> > osd pool default pg num = 333
> > osd pool default pgp num = 333
> > osd crush chooseleaf type = 1
> > _______________________________________________
> > ceph-users mailing list
> > ceph-users at lists.ceph.com
> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >
