On Mon, Aug 24, 2015 at 11:35 AM, Simon Hallam <sha@xxxxxxxxx> wrote:
> Hi Greg,
>
> The MDS' detect that the other one went down and started the replay.
>
> I did some further testing with 20 client machines. Of the 20 client machines, 5 hung with the following error:
>
> [Aug24 10:53] ceph: mds0 caps stale
> [Aug24 10:54] ceph: mds0 caps stale
> [Aug24 10:58] ceph: mds0 hung
> [Aug24 11:03] ceph: mds0 came back
> [ +8.803334] libceph: mon2 10.15.0.3:6789 socket closed (con state OPEN)
> [ +0.000018] libceph: mon2 10.15.0.3:6789 session lost, hunting for new mon
> [Aug24 11:04] ceph: mds0 reconnect start
> [ +0.084938] libceph: mon2 10.15.0.3:6789 session established
> [ +0.008475] ceph: mds0 reconnect denied
Oh, this might be a kernel bug, failing to ask for mdsmap updates when
the connection goes away. Zheng, does that sound familiar?
-Greg
>
> 10.15.0.3 was the active MDS at the time I unplugged the Ethernet cable.
>
>
> This was the output of ceph -w as I ran the test (I've removed a lot of the pg remapping):
>
> 2015-08-24 11:02:39.547529 mon.1 [INF] mon.ceph2 calling new monitor election
> 2015-08-24 11:02:40.011995 mon.0 [INF] mon.ceph1 calling new monitor election
> 2015-08-24 11:02:45.245869 mon.0 [INF] mon.ceph1@0 won leader election with quorum 0,1
> 2015-08-24 11:02:45.257440 mon.0 [INF] HEALTH_WARN; 1 mons down, quorum 0,1 ceph1,ceph2
> 2015-08-24 11:02:45.535369 mon.0 [INF] monmap e1: 3 mons at {ceph1=10.15.0.1:6789/0,ceph2=10.15.0.2:6789/0,ceph3=10.15.0.3:6789/0}
> 2015-08-24 11:02:45.535444 mon.0 [INF] pgmap v15803: 8256 pgs: 8256 active+clean; 1248 GB data, 2503 GB used, 193 TB / 196 TB avail; 47 B/s wr, 0 op/s
> 2015-08-24 11:02:45.535541 mon.0 [INF] mdsmap e38: 1/1/1 up {0=ceph3=up:active}, 2 up:standby
> 2015-08-24 11:02:45.535629 mon.0 [INF] osdmap e197: 36 osds: 36 up, 36 in
> 2015-08-24 11:03:01.946397 mon.0 [INF] mdsmap e39: 1/1/1 up {0=ceph2=up:replay}, 1 up:standby
> 2015-08-24 11:03:02.993880 mon.0 [INF] mds.0 10.15.0.2:6849/17644 up:reconnect
> 2015-08-24 11:03:02.993930 mon.0 [INF] mdsmap e40: 1/1/1 up {0=ceph2=up:reconnect}, 1 up:standby
> 2015-08-24 11:03:51.461248 mon.0 [INF] mds.0 10.15.0.2:6849/17644 up:rejoin
> 2015-08-24 11:03:55.807131 mon.0 [INF] mds.0 10.15.0.2:6849/17644 up:active
> 2015-08-24 11:03:55.807195 mon.0 [INF] mdsmap e42: 1/1/1 up {0=ceph2=up:active}, 1 up:standby
> 2015-08-24 11:06:48.036736 mon.0 [INF] mds.0 10.15.0.2:6849/17644 up:active
> 2015-08-24 11:06:48.036799 mon.0 [INF] mdsmap e43: 1/1/1 up {0=ceph2=up:active}, 1 up:standby
> *<cable plugged back in>*
> 2015-08-24 11:13:13.230714 mon.0 [INF] osd.32 10.15.0.3:6832/11565 boot
> 2015-08-24 11:13:13.230765 mon.0 [INF] osdmap e212: 36 osds: 25 up, 25 in
> 2015-08-24 11:13:13.230809 mon.0 [INF] mds.? 10.15.0.3:6833/16993 up:boot
> 2015-08-24 11:13:13.230837 mon.0 [INF] mdsmap e47: 1/1/1 up {0=ceph2=up:active}, 2 up:standby
> 2015-08-24 11:13:30.799429 mon.2 [INF] mon.ceph3 calling new monitor election
> 2015-08-24 11:13:30.826158 mon.0 [INF] mon.ceph1 calling new monitor election
> 2015-08-24 11:13:30.926331 mon.0 [INF] mon.ceph1@0 won leader election with quorum 0,1,2
> 2015-08-24 11:13:30.968739 mon.0 [INF] mdsmap e47: 1/1/1 up {0=ceph2=up:active}, 2 up:standby
> 2015-08-24 11:13:28.383203 mds.0 [INF] denied reconnect attempt (mds is up:active) from client.24155 10.10.10.95:0/3238635414 after 625.375507 (allowed interval 45)
> 2015-08-24 11:13:29.721653 mds.0 [INF] denied reconnect attempt (mds is up:active) from client.24146 10.10.10.99:0/3454703638 after 626.713952 (allowed interval 45)
> 2015-08-24 11:13:31.113004 mds.0 [INF] denied reconnect attempt (mds is up:active) from client.24140 10.10.10.60:0/359606080 after 628.105302 (allowed interval 45)
> 2015-08-24 11:13:50.933020 mds.0 [INF] denied reconnect attempt (mds is up:active) from client.24152 10.10.10.67:0/3475305031 after 647.925323 (allowed interval 45)
> 2015-08-24 11:13:51.037681 mds.0 [INF] denied reconnect attempt (mds is up:active) from client.24149 10.10.10.68:0/22416725 after 648.029988 (allowed interval 45)
>
> I did just notice that none of the times match up. So may try again once I fix ntp/chrony and see if that makes a difference.
>
> Cheers,
>
> Simon
>
>> -----Original Message-----
>> From: Gregory Farnum [mailto:gfarnum@xxxxxxxxxx]
>> Sent: 21 August 2015 12:16
>> To: Simon Hallam
>> Cc: ceph-users@xxxxxxxxxxxxxx
>> Subject: Re: Testing CephFS
>>
>> On Thu, Aug 20, 2015 at 11:07 AM, Simon Hallam <sha@xxxxxxxxx> wrote:
>> > Hey all,
>> >
>> >
>> >
>> > We are currently testing CephFS on a small (3 node) cluster.
>> >
>> >
>> >
>> > The setup is currently:
>> >
>> >
>> >
>> > Each server has 12 OSDs, 1 Monitor and 1 MDS running on it:
>> >
>> > The servers are running: 0.94.2-0.el7
>> >
>> > The clients are running: Ceph: 0.80.10-1.fc21, Kernel: 4.0.6-200.fc21.x86_64
>> >
>> >
>> >
>> > ceph -s
>> >
>> > cluster 4ed5ecdd-0c5b-4422-9d99-c9e42c6bd4cd
>> >
>> > health HEALTH_OK
>> >
>> > monmap e1: 3 mons at
>> > {ceph1=10.15.0.1:6789/0,ceph2=10.15.0.2:6789/0,ceph3=10.15.0.3:6789/0}
>> >
>> > election epoch 20, quorum 0,1,2 ceph1,ceph2,ceph3
>> >
>> > mdsmap e12: 1/1/1 up {0=ceph3=up:active}, 2 up:standby
>> >
>> > osdmap e389: 36 osds: 36 up, 36 in
>> >
>> > pgmap v19370: 8256 pgs, 3 pools, 51217 MB data, 14035 objects
>> >
>> > 95526 MB used, 196 TB / 196 TB avail
>> >
>> > 8256 active+clean
>> >
>> >
>> >
>> > Our Ceph.conf is relatively simple at the moment:
>> >
>> >
>> >
>> > cat /etc/ceph/ceph.conf
>> >
>> > [global]
>> >
>> > fsid = 4ed5ecdd-0c5b-4422-9d99-c9e42c6bd4cd
>> >
>> > mon_initial_members = ceph1, ceph2, ceph3
>> >
>> > mon_host = 10.15.0.1,10.15.0.2,10.15.0.3
>> >
>> > mon_pg_warn_max_per_osd = 1000
>> >
>> > auth_cluster_required = cephx
>> >
>> > auth_service_required = cephx
>> >
>> > auth_client_required = cephx
>> >
>> > filestore_xattr_use_omap = true
>> >
>> > osd_pool_default_size = 2
>> >
>> >
>> >
>> > When I pulled the plug on the master MDS last time (ceph1), it stopped all
>> > IO until I plugged it back in. I was under the assumption that the MDS
>> would
>> > fail over the other 2 MDS’s and IO would continue?
>> >
>> >
>> >
>> > Is there something I need to do to allow the MDS’s to failover from each
>> > other without too much interruption? Or is this because the clients ceph
>> > version?
>>
>> That's quite strange. How long did you wait for it to fail over? Did
>> the output of "ceph -s" (or "ceph -w", whichever) change during that
>> time?
>> By default the monitors should have detected the MDS was dead after 30
>> seconds and put one of the other MDS nodes into replay and active.
>>
>> ...I wonder if this is because you lost a monitor at the same time as
>> the MDS. What kind of logging do you have available from during your
>> test?
>> -Greg
>>
>> >
>> >
>> >
>> > Cheers,
>> >
>> >
>> >
>> > Simon Hallam
>> >
>> > Linux Support & Development Officer
>
>
> Please visit our new website at www.pml.ac.uk and follow us on Twitter @PlymouthMarine
>
> Winner of the Environment & Conservation category, the Charity Awards 2014.
>
> Plymouth Marine Laboratory (PML) is a company limited by guarantee registered in England & Wales, company number 4178503. Registered Charity No. 1091222. Registered Office: Prospect Place, The Hoe, Plymouth PL1 3DH, UK.
>
> This message is private and confidential. If you have received this message in error, please notify the sender and remove it from your system. You are reminded that e-mail communications are not secure and may contain viruses; PML accepts no liability for any loss or damage which may be caused by viruses.
>
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
