1) But Ceph says "...You can run a cluster with 1 monitor....." (http://ceph.com/docs/master/rados/operations/add-or-rm-mons/), I assume it should work. And brain split is not my current concern 2) I've written object to Ceph, now I just want to get it back Anyway. I tried to reduce the mon number to 1. But after I remove it following the steps, it just cannot start up any more.... 1. [root~] service ceph -a stop mon.serverB 2. [root~] ceph mon remove serverB ## hang here forever 3. #Remove the monitor entry from ceph.conf. 4. Restart ceph service [root@serverA~]# systemctl status ceph.service -l ceph.service - LSB: Start Ceph distributed file system daemons at boot time Loaded: loaded (/etc/rc.d/init.d/ceph) Active: failed (Result: timeout) since Tue 2015-03-31 15:46:25 CST; 3min 15s ago Process: 2937 ExecStop=/etc/rc.d/init.d/ceph stop (code=exited, status=0/SUCCESS) Process: 3670 ExecStart=/etc/rc.d/init.d/ceph start (code=killed, signal=TERM) Mar 31 15:44:26 serverA ceph[3670]: === osd.6 === Mar 31 15:44:56 serverA ceph[3670]: failed: 'timeout 30 /usr/bin/ceph -c /etc/ceph/ceph.conf --name=osd.6 --keyring=/var/lib/ceph/osd/ceph-6/keyring osd crush create-or-move -- 6 3.64 host=serverA root=default' Mar 31 15:44:56 serverA ceph[3670]: === osd.7 === Mar 31 15:45:26 serverA ceph[3670]: failed: 'timeout 30 /usr/bin/ceph -c /etc/ceph/ceph.conf --name=osd.7 --keyring=/var/lib/ceph/osd/ceph-7/keyring osd crush create-or-move -- 7 3.64 host=serverA root=default' Mar 31 15:45:26 serverA ceph[3670]: === osd.8 === Mar 31 15:45:57 serverA ceph[3670]: failed: 'timeout 30 /usr/bin/ceph -c /etc/ceph/ceph.conf --name=osd.8 --keyring=/var/lib/ceph/osd/ceph-8/keyring osd crush create-or-move -- 8 3.64 host=serverA root=default' Mar 31 15:45:57 serverA ceph[3670]: === osd.9 === Mar 31 15:46:25 serverA systemd[1]: ceph.service operation timed out. Terminating. Mar 31 15:46:25 serverA systemd[1]: Failed to start LSB: Start Ceph distributed file system daemons at boot time. Mar 31 15:46:25 serverA systemd[1]: Unit ceph.service entered failed state. /var/log/ceph/ceph.log says: 2015-03-31 15:55:57.648800 mon.0 10.???.78:6789/0 1048 : cluster [INF] osd.21 10.???.78:6855/25598 failed (39 reports from 9 peers after 20.118062 >= grace 20.000000) 2015-03-31 15:55:57.931889 mon.0 10.???.78:6789/0 1055 : cluster [INF] osd.15 10.????.78:6825/23894 failed (39 reports from 9 peers after 20.401379 >= grace 20.000000) Obviously serverB is down, but it should not affect serverA from functioning? Right? ________________________________________ From: Gregory Farnum [greg@xxxxxxxxxxx] Sent: Tuesday, March 31, 2015 11:53 AM To: Lindsay Mathieson; Kai KH Huang Cc: ceph-users@xxxxxxxxxxxxxx Subject: Re: One host failure bring down the whole cluster On Mon, Mar 30, 2015 at 8:02 PM, Lindsay Mathieson <lindsay.mathieson@xxxxxxxxx> wrote: > On Tue, 31 Mar 2015 02:42:27 AM Kai KH Huang wrote: >> Hi, all >> I have a two-node Ceph cluster, and both are monitor and osd. When >> they're both up, osd are all up and in, everything is fine... almost: > > > > Two things. > > 1 - You *really* need a min of three monitors. Ceph cannot form a quorum with > just two monitors and you run a risk of split brain. You can form quorums with an even number of monitors, and Ceph does so — there's no risk of split brain. The problem with 2 monitors is that a quorum is always 2 — which is exactly what you're seeing right now. You can't run with only one monitor up (assuming you have a non-zero number of them). > 2 - You also probably have a min size of two set (the default). This means > that you need a minimum of two copies of each data object for writes to work. > So with just two nodes, if one goes down you can't write to the other. Also this. > > > So: > - Install a extra monitor node - it doesn't have to be powerful, we just use a > Intel Celeron NUC for that. > > - reduce your minimum size to 1 (One). Yep. -Greg _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |