It appears that the codes are different between what I have (0.77) vs what you've provided. Which version was this patch for? Could you provide one for 0.77 or 0.78?
Regards,
Hong
From: Mohd Bazli Ab Karim <bazli.abkarim@xxxxxxxx>
To: "hjcho616 (hjcho616@xxxxxxxxx)" <hjcho616@xxxxxxxxx>; "Yan, Zheng" <ukernel@xxxxxxxxx>; Sage Weil <sage@xxxxxxxxxxx>
Cc: "ceph-users@xxxxxxxxxxxxxx" <ceph-users@xxxxxxxxxxxxxx>
Sent: Sunday, March 23, 2014 9:32 PM
Subject: RE: MDS crash when client goes to sleep
Hi Hong,
Could you apply the patch and see if it crash after sleep?
This could lead us to find the correct fix to MDS/client too.
As what I can see here, this patch should fix the crash, but how to fix MDS if the crash happens?
It happened to us, when it crashed, it was totally crash, and even restart the ceph-mds service with --reset-journal also not helping.
Anyone can shed some lights on this matter?
p/s: Is there any steps/tools to backup the MDS metadata? Say if MDS crash and refuse to run normally, can we restore the backup metadata? I'm thinking of it as a preventive steps, just in case if it happens again in future.
Many thanks.
Bazli
-----Original Message-----
From: Yan, Zheng [mailto:ukernel@xxxxxxxxx]
Sent: Sunday, March 23, 2014 2:53 PM
To: Sage Weil
Cc: Mohd Bazli Ab Karim; ceph-users@xxxxxxxxxxxxxx
Subject: Re: MDS crash when client goes to sleep
On Sun, Mar 23, 2014 at 11:47 AM, Sage Weil <sage@xxxxxxxxxxx> wrote:
> Hi,
>
> I looked at this a bit earlier and wasn't sure why we would be getting
> a remote_reset event after a sleep/wake cycle. The patch should fix
> the crash, but I'm a bit worried something is not quite right on the
> client side, too...
>
When client wakes up, it first tries reconnecting the old session. MDS refuses the reconnect request and sends a session close message to the client. After receiving the session close message, client closes the old session, then sends a session open message to the MDS. The MDS receives the open request and triggers a remote reset
(Pipe.cc:466)
> sage
>
> On Sun, 23 Mar 2014, Yan, Zheng wrote:
>
>> thank you for reporting this. Below patch should fix this issue
>>
>> ---
>> diff --git a/src/mds/MDS.cc b/src/mds/MDS.cc index 57c7f4a..6b53c14
>> 100644
>> --- a/src/mds/MDS.cc
>> +++ b/src/mds/MDS.cc
>> @@ -2110,6 +2110,7 @@ bool MDS::ms_handle_reset(Connection *con)
>> if (session->is_closed()) {
>> dout(3) << "ms_handle_reset closing connection for session " <<
>> session->info.inst << dendl;
>> messenger->mark_down(con);
>> + con->set_priv(NULL);
>> sessionmap.remove_session(session);
>> }
>> session->put();
>> @@ -2138,6 +2139,7 @@ void MDS::ms_handle_remote_reset(Connection *con)
>> if (session->is_closed()) {
>> dout(3) << "ms_handle_remote_reset closing connection for session "
>> << session->info.inst << dendl;
>> messenger->mark_down(con);
>> + con->set_priv(NULL);
>> sessionmap.remove_session(session);
>> }
>> session->put();
>>
>> On Fri, Mar 21, 2014 at 4:16 PM, Mohd Bazli Ab Karim
>> <bazli.abkarim@xxxxxxxx> wrote:
>> > Hi Hong,
>> >
>> >
>> >
>> > How's the client now? Would it able to mount to the filesystem now?
>> > It looks similar to our case,
>> > http://www.spinics.net/lists/ceph-devel/msg18395.html
>> >
>> > However, you need to collect some logs to confirm this.
>> >
>> >
>> >
>> > Thanks.
>> >
>> >
>> >
>> >
>> >
>> > From: hjcho616 [mailto:hjcho616@xxxxxxxxx]
>> > Sent: Friday, March 21, 2014 2:30 PM
>> >
>> >
>> > To: Luke Jing Yuan
>> > Cc: Mohd Bazli Ab Karim; ceph-users@xxxxxxxxxxxxxx
>> > Subject: Re: MDS crash when client goes to sleep
>> >
>> >
>> >
>> > Luke,
>> >
>> >
>> >
>> > Not sure what flapping ceph-mds daemon mean, but when I connected
>> > to MDS when this happened there no longer was any process with
>> > ceph-mds when I ran one daemon. When I ran three there were one
>> > left but wasn't doing much. I didn't record the logs but behavior
>> > was very similar in 0.72 emperor. I am using debian packages.
>> >
>> >
>> >
>> > Client went to sleep for a while (like 8+ hours). There was no I/O
>> > prior to the sleep other than the fact that cephfs was still mounted.
>> >
>> >
>> >
>> > Regards,
>> >
>> > Hong
>> >
>> >
>> >
>> > ________________________________
>> >
>> > From: Luke Jing Yuan <jyluke@xxxxxxxx>
>> >
>> >
>> > To: hjcho616 <hjcho616@xxxxxxxxx>
>> > Cc: Mohd Bazli Ab Karim <bazli.abkarim@xxxxxxxx>;
>> > "ceph-users@xxxxxxxxxxxxxx" <ceph-users@xxxxxxxxxxxxxx>
>> > Sent: Friday, March 21, 2014 1:17 AM
>> >
>> > Subject: RE: [ceph-users] MDS crash when client goes to sleep
>> >
>> >
>> > Hi Hong,
>> >
>> > That's interesting, for Mr. Bazli and I, we ended with MDS stuck in
>> > (up:replay) and a flapping ceph-mds daemon, but then again we are
>> > using version 0.72.2. Having said so the triggering point seem
>> > similar to us as well, which is the following line:
>> >
>> > -38> 2014-03-20 20:08:44.495565 7fee3d7c4700 0 --
>> > 192.168.1.20:6801/17079
>> >>> 192.168.1.101:0/2113152127 pipe(0x3f03b80 sd=18 :6801 s=0 pgs=0
>> >>> cs=0 l=0
>> > c=0x1f0e2160).accept we reset (peer sent cseq 2), sending
>> > RESETSESSION
>> >
>> > So how long did your client go into sleep? Was there any I/O prior
>> > to the sleep?
>> >
>> > Regards,
>> > Luke
>> >
>> > From: hjcho616 [mailto:hjcho616@xxxxxxxxx]
>> > Sent: Friday, 21 March, 2014 12:09 PM
>> > To: Luke Jing Yuan
>> > Cc: Mohd Bazli Ab Karim; ceph-users@xxxxxxxxxxxxxx
>> > Subject: Re: MDS crash when client goes to sleep
>> >
>> > Nope just these segfaults.
>> >
>> > [149884.709608] ceph-mds[17366]: segfault at 200 ip
>> > 00007f09de9d60b8 sp
>> > 00007f09db461520 error 4 in libgcc_s.so.1[7f09de9c7000+15000]
>> > [211263.265402] ceph-mds[17135]: segfault at 200 ip
>> > 00007f59eec280b8 sp
>> > 00007f59eb6b3520 error 4 in libgcc_s.so.1[7f59eec19000+15000]
>> > [214638.927759] ceph-mds[16896]: segfault at 200 ip
>> > 00007fcb2c89e0b8 sp
>> > 00007fcb29329520 error 4 in libgcc_s.so.1[7fcb2c88f000+15000]
>> > [289338.461271] ceph-mds[20878]: segfault at 200 ip
>> > 00007f4b7211c0b8 sp
>> > 00007f4b6eba7520 error 4 in libgcc_s.so.1[7f4b7210d000+15000]
>> > [373738.961475] ceph-mds[21341]: segfault at 200 ip
>> > 00007f36c3d480b8 sp
>> > 00007f36c07d3520 error 4 in libgcc_s.so.1[7f36c3d39000+15000]
>> >
>> > Regards,
>> > Hong
>> >
>> > ________________________________________
>> > From: Luke Jing Yuan <jyluke@xxxxxxxx>
>> > To: hjcho616 <hjcho616@xxxxxxxxx>
>> > Cc: Mohd Bazli Ab Karim <bazli.abkarim@xxxxxxxx>;
>> > "ceph-users@xxxxxxxxxxxxxx" <ceph-users@xxxxxxxxxxxxxx>
>> > Sent: Thursday, March 20, 2014 10:53 PM
>> > Subject: Re: MDS crash when client goes to sleep
>> >
>> > Did you see any messages in dmesg saying ceph-mds respawnning or
>> > stuffs like that?
>> >
>> > Regards,
>> > Luke
>> >
>> > On Mar 21, 2014, at 11:09 AM, "hjcho616" <hjcho616@xxxxxxxxx> wrote:
>> > On client, I was no longer able to access the filesystem. It would hang.
>> > Makes sense since MDS has crashed. I tried running 3 MDS demon on
>> > the same machine. Two crashes and one appears to be hung up(?).
>> > ceph health says MDS is in degraded state when that happened.
>> >
>> > I was able to recover by restarting every node. I currently have
>> > three machine, one with MDS and MON, and two with OSDs.
>> >
>> > It is failing everytime my client machine goes to sleep. If you
>> > need me to run something let me know what and how.
>> >
>> > Regards,
>> > Hong
>> >
>> > ________________________________________
>> > From: Mohd Bazli Ab Karim <bazli.abkarim@xxxxxxxx>
>> > To: hjcho616 <hjcho616@xxxxxxxxx>; "ceph-users@xxxxxxxxxxxxxx"
>> > <ceph-users@xxxxxxxxxxxxxx>
>> > Sent: Thursday, March 20, 2014 9:40 PM
>> > Subject: RE: MDS crash when client goes to sleep
>> >
>> > Hi Hong,
>> > May I know what has happened to your MDS once it crashed? Was it
>> > able to recover from replay?
>> > We also facing this issue and I am interested to know on how to
>> > reproduce it.
>> >
>> > Thanks.
>> > Bazli
>> >
>> > From: ceph-users-bounces@xxxxxxxxxxxxxx
>> > [mailto:ceph-users-bounces@xxxxxxxxxxxxxx] On Behalf Of hjcho616
>> > Sent: Friday, March 21, 2014 10:29 AM
>> > To: ceph-users@xxxxxxxxxxxxxx
>> > Subject: MDS crash when client goes to sleep
>> >
>> > When CephFS is mounted on a client and when client decides to go to
>> > sleep, MDS segfaults. Has anyone seen this? Below is a part of
>> > MDS log. This happened in emperor and recent 0.77 release. I am
>> > running Debian Wheezy with testing kernels 3.13. What can I do to
>> > not crash the whole system if a client goes to sleep (and looks
>> > like disconnect may do the same)? Let me know if you need any more info.
>> >
>> > Regards,
>> > Hong
>> >
>> > -43> 2014-03-20 20:08:42.463357 7fee3f0cf700 1 --
>> > 192.168.1.20:6801/17079
>> > --> 192.168.1.20:6789/0 -- mdsbeacon(6798/MDS1.2 up:active seq
>> > --> 21120 v6970)
>> > v2 -- ?+0 0x1ee9f080 con 0x2e56580
>> > -42> 2014-03-20 20:08:42.463787 7fee411d4700 1 --
>> > 192.168.1.20:6801/17079 <== mon.0 192.168.1.20:6789/0 21764 ====
>> > mdsbeacon(6798/MDS1.2 up:active seq
>> > 21120 v6970) v2 ==== 108+0+0 (266728949 0 0) 0x1ee88dc0 con 0x2e56580
>> > -41> 2014-03-20 20:08:43.373099 7fee3f0cf700 2 mds.0.cache
>> > check_memory_usage total 665384, rss 503156, heap 24656, malloc
>> > 463874 mmap 0, baseline 16464, buffers 0, max 1048576, 0 / 62380
>> > inodes have caps, 0 caps, 0 caps per inode
>> > -40> 2014-03-20 20:08:44.494963 7fee3d7c4700 1 --
>> > 192.168.1.20:6801/17079
>> >>> :/0 pipe(0x3f03b80 sd=18 :6801 s=0 pgs=0 cs=0 l=0
>> >>> c=0x1f0e2160).accept
>> > sd=18 192.168.1.101:52026/0
>> > -39> 2014-03-20 20:08:44.495033 7fee3d7c4700 0 --
>> > 192.168.1.20:6801/17079
>> >>> 192.168.1.101:0/2113152127 pipe(0x3f03b80 sd=18 :6801 s=0 pgs=0
>> >>> cs=0 l=0
>> > c=0x1f0e2160).accept peer addr is really 192.168.1.101:0/2113152127
>> > (socket is 192.168.1.101:52026/0)
>> > -38> 2014-03-20 20:08:44.495565 7fee3d7c4700 0 --
>> > 192.168.1.20:6801/17079
>> >>> 192.168.1.101:0/2113152127 pipe(0x3f03b80 sd=18 :6801 s=0 pgs=0
>> >>> cs=0 l=0
>> > c=0x1f0e2160).accept we reset (peer sent cseq 2), sending RESETSESSION
>> > -37> 2014-03-20 20:08:44.496015 7fee3d7c4700 2 --
>> > 192.168.1.20:6801/17079
>> >>> 192.168.1.101:0/2113152127 pipe(0x3f03b80 sd=18 :6801 s=4 pgs=0
>> >>> cs=0 l=0
>> > c=0x1f0e2160).fault 0: Success
>> > -36> 2014-03-20 20:08:44.496099 7fee411d4700 5 mds.0.35
>> > ms_handle_reset on 192.168.1.101:0/2113152127
>> > -35> 2014-03-20 20:08:44.496120 7fee411d4700 3 mds.0.35
>> > ms_handle_reset closing connection for session client.6019 192.168.1.101:0/2113152127
>> > -34> 2014-03-20 20:08:44.496207 7fee411d4700 1 --
>> > 192.168.1.20:6801/17079 mark_down 0x1f0e2160 -- pipe dne
>> > -33> 2014-03-20 20:08:44.653628 7fee3d7c4700 1 --
>> > 192.168.1.20:6801/17079
>> >>> :/0 pipe(0x3d8e000 sd=18 :6801 s=0 pgs=0 cs=0 l=0
>> >>> c=0x1f0e22c0).accept
>> > sd=18 192.168.1.101:52027/0
>> > -32> 2014-03-20 20:08:44.653677 7fee3d7c4700 0 --
>> > 192.168.1.20:6801/17079
>> >>> 192.168.1.101:0/2113152127 pipe(0x3d8e000 sd=18 :6801 s=0 pgs=0
>> >>> cs=0 l=0
>> > c=0x1f0e22c0).accept peer addr is really 192.168.1.101:0/2113152127
>> > (socket is 192.168.1.101:52027/0)
>> > -31> 2014-03-20 20:08:44.925618 7fee411d4700 1 --
>> > 192.168.1.20:6801/17079 <== client.6019 192.168.1.101:0/2113152127
>> > 1 ==== client_reconnect(77349
>> > caps) v2 ==== 0+0+11032578 (0 0 3293767716) 0x2e92780 con 0x1f0e22c0
>> > -30> 2014-03-20 20:08:44.925682 7fee411d4700 1 mds.0.server no
>> > longer in reconnect state, ignoring reconnect, sending close
>> > -29> 2014-03-20 20:08:44.925735 7fee411d4700 0 log [INF] :
>> > denied reconnect attempt (mds is up:active) from client.6019
>> > 192.168.1.101:0/2113152127 after 2014-03-20 20:08:44.925679
>> > (allowed interval 45)
>> > -28> 2014-03-20 20:08:44.925748 7fee411d4700 1 --
>> > 192.168.1.20:6801/17079
>> > --> 192.168.1.101:0/2113152127 -- client_session(close) v1 -- ?+0
>> > --> 0x3ea6540
>> > con 0x1f0e22c0
>> > -27> 2014-03-20 20:08:44.927727 7fee3d7c4700 2 --
>> > 192.168.1.20:6801/17079
>> >>> 192.168.1.101:0/2113152127 pipe(0x3d8e000 sd=18 :6801 s=2 pgs=135
>> >>> cs=1
>> > l=0 c=0x1f0e22c0).reader couldn't read tag, Success
>> > -26> 2014-03-20 20:08:44.927797 7fee3d7c4700 2 --
>> > 192.168.1.20:6801/17079
>> >>> 192.168.1.101:0/2113152127 pipe(0x3d8e000 sd=18 :6801 s=2 pgs=135
>> >>> cs=1
>> > l=0 c=0x1f0e22c0).fault 0: Success
>> > -25> 2014-03-20 20:08:44.927849 7fee3d7c4700 0 --
>> > 192.168.1.20:6801/17079
>> >>> 192.168.1.101:0/2113152127 pipe(0x3d8e000 sd=18 :6801 s=2 pgs=135
>> >>> cs=1
>> > l=0 c=0x1f0e22c0).fault, server, going to standby
>> > -24> 2014-03-20 20:08:46.372279 7fee401d2700 10 monclient: tick
>> > -23> 2014-03-20 20:08:46.372339 7fee401d2700 10 monclient:
>> > _check_auth_rotating have uptodate secrets (they expire after
>> > 2014-03-20
>> > 20:08:16.372333)
>> > -22> 2014-03-20 20:08:46.372373 7fee401d2700 10 monclient: renew subs?
>> > (now: 2014-03-20 20:08:46.372372; renew after: 2014-03-20
>> > 20:09:56.370811)
>> > -- no
>> > -21> 2014-03-20 20:08:46.372403 7fee401d2700 10 log_queue is 1
>> > last_log 2 sent 1 num 1 unsent 1 sending 1
>> > -20> 2014-03-20 20:08:46.372421 7fee401d2700 10 will send
>> > 2014-03-20
>> > 20:08:44.925741 mds.0 192.168.1.20:6801/17079 2 : [INF] denied
>> > reconnect attempt (mds is up:active) from client.6019
>> > 192.168.1.101:0/2113152127 after
>> > 2014-03-20 20:08:44.925679 (allowed interval 45)
>> > -19> 2014-03-20 20:08:46.372466 7fee401d2700 10 monclient:
>> > _send_mon_message to mon.MDS1 at 192.168.1.20:6789/0
>> > -18> 2014-03-20 20:08:46.372483 7fee401d2700 1 --
>> > 192.168.1.20:6801/17079
>> > --> 192.168.1.20:6789/0 -- log(1 entries) v1 -- ?+0 0x71d8d80 con
>> > --> 0x2e56580
>> > -17> 2014-03-20 20:08:46.463496 7fee3f0cf700 10 monclient:
>> > _send_mon_message to mon.MDS1 at 192.168.1.20:6789/0
>> > -16> 2014-03-20 20:08:46.463524 7fee3f0cf700 1 --
>> > 192.168.1.20:6801/17079
>> > --> 192.168.1.20:6789/0 -- mdsbeacon(6798/MDS1.2 up:active seq
>> > --> 21121 v6970)
>> > v2 -- ?+0 0x1ee91340 con 0x2e56580
>> > -15> 2014-03-20 20:08:46.499688 7fee411d4700 1 --
>> > 192.168.1.20:6801/17079 <== mon.0 192.168.1.20:6789/0 21765 ====
>> > log(last 2) v1 ==== 24+0+0
>> > (1174756693 0 0) 0x3ea7a40 con 0x2e56580
>> > -14> 2014-03-20 20:08:46.499751 7fee411d4700 10 handle_log_ack
>> > log(last 2)
>> > v1
>> > -13> 2014-03-20 20:08:46.499757 7fee411d4700 10 logged
>> > 2014-03-20
>> > 20:08:44.925741 mds.0 192.168.1.20:6801/17079 2 : [INF] denied
>> > reconnect attempt (mds is up:active) from client.6019
>> > 192.168.1.101:0/2113152127 after
>> > 2014-03-20 20:08:44.925679 (allowed interval 45)
>> > -12> 2014-03-20 20:08:46.499825 7fee411d4700 1 --
>> > 192.168.1.20:6801/17079 <== mon.0 192.168.1.20:6789/0 21766 ====
>> > mdsbeacon(6798/MDS1.2 up:active seq
>> > 21121 v6970) v2 ==== 108+0+0 (51773011 0 0) 0x1ee25600 con 0x2e56580
>> > -11> 2014-03-20 20:08:47.303323 7fee3ddca700 1 --
>> > 192.168.1.20:6801/17079
>> >>> :/0 pipe(0x3e92280 sd=21 :6801 s=0 pgs=0 cs=0 l=0
>> >>> c=0x1f0e3080).accept
>> > sd=21 192.168.1.101:52029/0
>> > -10> 2014-03-20 20:08:47.308830 7fee3ddca700 0 --
>> > 192.168.1.20:6801/17079
>> >>> 192.168.1.101:0/2113152127 pipe(0x3e92280 sd=21 :6801 s=0 pgs=0
>> >>> cs=0 l=0
>> > c=0x1f0e3080).accept peer addr is really 192.168.1.101:0/2113152127
>> > (socket is 192.168.1.101:52029/0)
>> > -9> 2014-03-20 20:08:47.309221 7fee3ddca700 0 --
>> > 192.168.1.20:6801/17079 >> 192.168.1.101:0/2113152127
>> > pipe(0x3e92280 sd=21
>> > :6801 s=0 pgs=0 cs=0 l=0 c=0x1f0e3080).accept connect_seq 0 vs
>> > existing 1 state standby
>> > -8> 2014-03-20 20:08:47.309251 7fee3ddca700 0 --
>> > 192.168.1.20:6801/17079 >> 192.168.1.101:0/2113152127
>> > pipe(0x3e92280 sd=21
>> > :6801 s=0 pgs=0 cs=0 l=0 c=0x1f0e3080).accept peer reset, then
>> > tried to connect to us, replacing
>> > -7> 2014-03-20 20:08:47.309337 7fee411d4700 5 mds.0.35
>> > ms_handle_remote_reset on 192.168.1.101:0/2113152127
>> > -6> 2014-03-20 20:08:47.309357 7fee411d4700 3 mds.0.35
>> > ms_handle_remote_reset closing connection for session client.6019
>> > 192.168.1.101:0/2113152127
>> > -5> 2014-03-20 20:08:47.309385 7fee411d4700 1 --
>> > 192.168.1.20:6801/17079 mark_down 0x1f0e22c0 -- 0x3e92280
>> > -4> 2014-03-20 20:08:47.309478 7fee411d4700 5 mds.0.35
>> > ms_handle_reset on 192.168.1.101:0/2113152127
>> > -3> 2014-03-20 20:08:47.309490 7fee411d4700 3 mds.0.35
>> > ms_handle_reset closing connection for session client.6019 192.168.1.101:0/2113152127
>> > -2> 2014-03-20 20:08:47.309501 7fee411d4700 1 --
>> > 192.168.1.20:6801/17079 mark_down 0x1f0e3080 -- 0x3e92280
>> > -1> 2014-03-20 20:08:47.309485 7fee3ddca700 2 --
>> > 192.168.1.20:6801/17079 >> 192.168.1.101:0/2113152127
>> > pipe(0x3e92280 sd=21
>> > :6801 s=4 pgs=136 cs=1 l=0 c=0x1f0e22c0).accept read error on
>> > newly_acked_seq
>> > 0> 2014-03-20 20:08:47.310724 7fee411d4700 -1 *** Caught signal
>> > (Segmentation fault) ** in thread 7fee411d4700
>> >
>> > ceph version 0.77 (1bca9c5c412b3af722d5250f07fd562a23cf35ff)
>> > 1: /usr/bin/ceph-mds() [0x8baf22]
>> > 2: (()+0xf880) [0x7fee458ef880]
>> > NOTE: a copy of the executable, or `objdump -rdS <executable>` is
>> > needed to interpret this.
>> >
>> > --- logging levels ---
>> > 0/ 5 none
>> > 0/ 1 lockdep
>> > 0/ 1 context
>> > 1/ 1 crush
>> > 1/ 5 mds
>> > 1/ 5 mds_balancer
>> > 1/ 5 mds_locker
>> > 1/ 5 mds_log
>> > 1/ 5 mds_log_expire
>> > 1/ 5 mds_migrator
>> > 0/ 1 buffer
>> > 0/ 1 timer
>> > 0/ 1 filer
>> > 0/ 1 striper
>> > 0/ 1 objecter
>> > 0/ 5 rados
>> > 0/ 5 rbd
>> > 0/ 5 journaler
>> > 0/ 5 objectcacher
>> > 0/ 5 client
>> > 0/ 5 osd
>> > 0/ 5 optracker
>> > 0/ 5 objclass
>> > 1/ 3 filestore
>> > 1/ 3 keyvaluestore
>> > 1/ 3 journal
>> > 0/ 5 ms
>> > 1/ 5 mon
>> > 0/10 monc
>> > 1/ 5 paxos
>> > 0/ 5 tp
>> > 1/ 5 auth
>> > 1/ 5 crypto
>> > 1/ 1 finisher
>> > 1/ 5 heartbeatmap
>> > 1/ 5 perfcounter
>> > 1/ 5 rgw
>> > 1/ 5 javaclient
>> > 1/ 5 asok
>> > 1/ 1 throttle
>> > -2/-2 (syslog threshold)
>> > -1/-1 (stderr threshold)
>> > max_recent 10000
>> > max_new 1000
>> > log_file /var/log/ceph/ceph-mds.MDS1.2.log
>> > --- end dump of recent events ---
>> >
>> >
>> >
>> > ________________________________________
>> > DISCLAIMER:
>> >
>> > This e-mail (including any attachments) is for the addressee(s)
>> > only and may be confidential, especially as regards personal data.
>> > If you are not the intended recipient, please note that any
>> > dealing, review, distribution, printing, copying or use of this
>> > e-mail is strictly prohibited. If you have received this email in
>> > error, please notify the sender immediately and delete the original message (including any attachments).
>> >
>> > MIMOS Berhad is a research and development institution under the
>> > purview of the Malaysian Ministry of Science, Technology and
>> > Innovation. Opinions, conclusions and other information in this
>> > e-mail that do not relate to the official business of MIMOS Berhad
>> > and/or its subsidiaries shall be understood as neither given nor
>> > endorsed by MIMOS Berhad and/or its subsidiaries and neither MIMOS
>> > Berhad nor its subsidiaries accepts responsibility for the same.
>> > All liability arising from or in connection with computer viruses
>> > and/or corrupted e-mails is excluded to the fullest extent permitted by law.
>> >
>> > _______________________________________________
>> > ceph-users mailing list
>> > ceph-users@xxxxxxxxxxxxxx
>> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>> >
>> > ________________________________________
>> > DISCLAIMER:
>> >
>> > This e-mail (including any attachments) is for the addressee(s)
>> > only and may be confidential, especially as regards personal data.
>> > If you are not the intended recipient, please note that any
>> > dealing, review, distribution, printing, copying or use of this
>> > e-mail is strictly prohibited. If you have received this email in
>> > error, please notify the sender immediately and delete the original message (including any attachments).
>> >
>> > MIMOS Berhad is a research and development institution under the
>> > purview of the Malaysian Ministry of Science, Technology and
>> > Innovation. Opinions, conclusions and other information in this
>> > e-mail that do not relate to the official business of MIMOS Berhad
>> > and/or its subsidiaries shall be understood as neither given nor
>> > endorsed by MIMOS Berhad and/or its subsidiaries and neither MIMOS
>> > Berhad nor its subsidiaries accepts responsibility for the same.
>> > All liability arising from or in connection with computer viruses
>> > and/or corrupted e-mails is excluded to the fullest extent permitted by law.
>> > ------------------------------------------------------------------
>> > -
>> > -
>> > DISCLAIMER:
>> >
>> > This e-mail (including any attachments) is for the addressee(s)
>> > only and may contain confidential information. If you are not the
>> > intended recipient, please note that any dealing, review,
>> > distribution, printing, copying or use of this e-mail is strictly
>> > prohibited. If you have received this email in error, please notify
>> > the sender immediately and delete the original message.
>> > MIMOS Berhad is a research and development institution under the
>> > purview of the Malaysian Ministry of Science, Technology and
>> > Innovation. Opinions, conclusions and other information in this e-
>> > mail that do not relate to the official business of MIMOS Berhad
>> > and/or its subsidiaries shall be understood as neither given nor
>> > endorsed by MIMOS Berhad and/or its subsidiaries and neither MIMOS
>> > Berhad nor its subsidiaries accepts responsibility for the same.
>> > All liability arising from or in connection with computer viruses
>> > and/or corrupted e-mails is excluded to the fullest extent
>> > permitted by law.
>> >
>> >
>> >
>> > ________________________________
>> > DISCLAIMER:
>> >
>> >
>> > This e-mail (including any attachments) is for the addressee(s)
>> > only and may be confidential, especially as regards personal data.
>> > If you are not the intended recipient, please note that any
>> > dealing, review, distribution, printing, copying or use of this
>> > e-mail is strictly prohibited. If you have received this email in
>> > error, please notify the sender immediately and delete the original message (including any attachments).
>> >
>> >
>> > MIMOS Berhad is a research and development institution under the
>> > purview of the Malaysian Ministry of Science, Technology and
>> > Innovation. Opinions, conclusions and other information in this
>> > e-mail that do not relate to the official business of MIMOS Berhad
>> > and/or its subsidiaries shall be understood as neither given nor
>> > endorsed by MIMOS Berhad and/or its subsidiaries and neither MIMOS
>> > Berhad nor its subsidiaries accepts responsibility for the same.
>> > All liability arising from or in connection with computer viruses
>> > and/or corrupted e-mails is excluded to the fullest extent permitted by law.
>> >
>> > ------------------------------------------------------------------
>> > -
>> > -
>> > DISCLAIMER:
>> >
>> > This e-mail (including any attachments) is for the addressee(s)
>> > only and may contain confidential information. If you are not the
>> > intended recipient, please note that any dealing, review,
>> > distribution, printing, copying or use of this e-mail is strictly
>> > prohibited. If you have received this email in error, please notify
>> > the sender immediately and delete the original message.
>> > MIMOS Berhad is a research and development institution under the
>> > purview of the Malaysian Ministry of Science, Technology and
>> > Innovation. Opinions, conclusions and other information in this e-
>> > mail that do not relate to the official business of MIMOS Berhad
>> > and/or its subsidiaries shall be understood as neither given nor
>> > endorsed by MIMOS Berhad and/or its subsidiaries and neither MIMOS
>> > Berhad nor its subsidiaries accepts responsibility for the same.
>> > All liability arising from or in connection with computer viruses
>> > and/or corrupted e-mails is excluded to the fullest extent
>> > permitted by law.
>> >
>> >
>> >
>> >
>> > ________________________________
>> > DISCLAIMER:
>> >
>> > This e-mail (including any attachments) is for the addressee(s)
>> > only and may be confidential, especially as regards personal data.
>> > If you are not the intended recipient, please note that any
>> > dealing, review, distribution, printing, copying or use of this
>> > e-mail is strictly prohibited. If you have received this email in
>> > error, please notify the sender immediately and delete the original message (including any attachments).
>> >
>> >
>> > MIMOS Berhad is a research and development institution under the
>> > purview of the Malaysian Ministry of Science, Technology and
>> > Innovation. Opinions, conclusions and other information in this
>> > e-mail that do not relate to the official business of MIMOS Berhad
>> > and/or its subsidiaries shall be understood as neither given nor
>> > endorsed by MIMOS Berhad and/or its subsidiaries and neither MIMOS
>> > Berhad nor its subsidiaries accepts responsibility for the same.
>> > All liability arising from or in connection with computer viruses
>> > and/or corrupted e-mails is excluded to the fullest extent permitted by law.
>> >
>> >
>> > _______________________________________________
>> > ceph-users mailing list
>> > ceph-users@xxxxxxxxxxxxxx
>> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>> >
>> _______________________________________________
>> ceph-users mailing list
>> ceph-users@xxxxxxxxxxxxxx
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>
>>
________________________________
DISCLAIMER:
This e-mail (including any attachments) is for the addressee(s) only and may be confidential, especially as regards personal data. If you are not the intended recipient, please note that any dealing, review, distribution, printing, copying or use of this e-mail is strictly prohibited. If you have received this email in error, please notify the sender immediately and delete the original message (including any attachments).
MIMOS Berhad is a research and development institution under the purview of the Malaysian Ministry of Science, Technology and Innovation. Opinions, conclusions and other information in this e-mail that do not relate to the official business of MIMOS Berhad and/or its subsidiaries shall be understood as neither given nor endorsed by MIMOS Berhad and/or its subsidiaries and neither MIMOS Berhad nor its subsidiaries accepts responsibility for the same. All liability arising from or in connection with computer viruses and/or corrupted e-mails is excluded to the fullest extent permitted by law.
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
