Re: Is mon initial members used after the first quorum?

[Christopher Armstrong <chris@xxxxxxxxxxxx>]

Thanks Greg - I thought the same thing, but confirmed with the user that it appears the radosgw client is indeed using initial members - when he added all of his hosts to initial members, things worked just fine. In either event, all of the monitors were always fully enumerated later in the config file. Is this potentially a bug specific to radosgw? Here's his config file:

[global]
fsid = fc0e2e09-ade3-4ff6-b23e-f789775b2515
mon initial members = nodo-3
auth cluster required = cephx
auth service required = cephx
auth client required = cephx
osd pool default size = 3
osd pool default min_size = 1
osd pool default pg_num = 128
osd pool default pgp_num = 128
osd recovery delay start = 15
log file = /dev/stdout
mon_clock_drift_allowed = 1
 
 
[mon.nodo-1]
host = nodo-1
mon addr = 192.168.2.200:6789
 
[mon.nodo-2]
host = nodo-2
mon addr = 192.168.2.201:6789
 
[mon.nodo-3]
host = nodo-3
mon addr = 192.168.2.202:6789
 
 
 
[client.radosgw.gateway]
host = deis-store-gateway
keyring = /etc/ceph/ceph.client.radosgw.keyring
rgw socket path = /var/run/ceph/ceph.radosgw.gateway.fastcgi.sock
log file = /dev/stdout

On Wed, Dec 10, 2014 at 11:40 AM, Gregory Farnum <greg@xxxxxxxxxxx> wrote:

On Tue, Dec 9, 2014 at 3:11 PM, Christopher Armstrong
<chris@xxxxxxxxxxxx> wrote:
> Hi folks,
>
> I think we have a bit of confusion around how initial members is used. I
> understand that we can specify a single monitor (or a subset of monitors) so
> that the cluster can form a quorum when it first comes up. This is how we're
> using the setting now - so the cluster can come up with just one monitor,
> with the other monitors to follow later.
>
> However, a Deis user reported that when the monitor in his initial members
> list went down, radosgw stopped functioning, even though there are three
> mons in his config file. I would think that the radosgw client would connect
> to any of the nodes in the config file to get the state of the cluster, and
> that the initial members list is only used when the monitors first come up
> and are trying to achieve quorum.
>
> The issue he filed is here: https://github.com/deis/deis/issues/2711
>
> He also found this Ceph issue filed: https://github.com/ceph/ceph/pull/1233

Nope, this has nothing to do with it.

>
> Is that what we're seeing here? Can anyone point us in the right direction?

I didn't see the actual conf file posted anywhere to look at, but my
guess is simply that (since it looks like you're using generated conf
files which can differ across hosts) that the one on the server(s) in
question don't have the monitors listed in them. I'm only skimming the
code, but from it and my recollection, when a Ceph client starts up it
will try to assemble a list of monitors to contact from:
1) the contents of the "mon host" config entry
2) the "mon addr" value in any of the "global", "mon" or "mon.X" sections

The clients don't even look at mon_initial_members that I can see,
actually — so perhaps your client config only lists the initial
monitor, without adding the others?
-Greg

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com