thank you for reporting this. Below patch should fix this issue
---
diff --git a/src/mds/MDS.cc b/src/mds/MDS.cc
index 57c7f4a..6b53c14 100644
--- a/src/mds/MDS.cc
+++ b/src/mds/MDS.cc
@@ -2110,6 +2110,7 @@ bool MDS::ms_handle_reset(Connection *con)
if (session->is_closed()) {
dout(3) << "ms_handle_reset closing connection for session " <<
session->info.inst << dendl;
messenger->mark_down(con);
+ con->set_priv(NULL);
sessionmap.remove_session(session);
}
session->put();
@@ -2138,6 +2139,7 @@ void MDS::ms_handle_remote_reset(Connection *con)
if (session->is_closed()) {
dout(3) << "ms_handle_remote_reset closing connection for session "
<< session->info.inst << dendl;
messenger->mark_down(con);
+ con->set_priv(NULL);
sessionmap.remove_session(session);
}
session->put();
On Fri, Mar 21, 2014 at 4:16 PM, Mohd Bazli Ab Karim
<bazli.abkarim@xxxxxxxx> wrote:
> Hi Hong,
>
>
>
> How's the client now? Would it able to mount to the filesystem now? It looks
> similar to our case, http://www.spinics.net/lists/ceph-devel/msg18395.html
>
> However, you need to collect some logs to confirm this.
>
>
>
> Thanks.
>
>
>
>
>
> From: hjcho616 [mailto:hjcho616@xxxxxxxxx]
> Sent: Friday, March 21, 2014 2:30 PM
>
>
> To: Luke Jing Yuan
> Cc: Mohd Bazli Ab Karim; ceph-users@xxxxxxxxxxxxxx
> Subject: Re: MDS crash when client goes to sleep
>
>
>
> Luke,
>
>
>
> Not sure what flapping ceph-mds daemon mean, but when I connected to MDS
> when this happened there no longer was any process with ceph-mds when I ran
> one daemon. When I ran three there were one left but wasn't doing much. I
> didn't record the logs but behavior was very similar in 0.72 emperor. I am
> using debian packages.
>
>
>
> Client went to sleep for a while (like 8+ hours). There was no I/O prior to
> the sleep other than the fact that cephfs was still mounted.
>
>
>
> Regards,
>
> Hong
>
>
>
> ________________________________
>
> From: Luke Jing Yuan <jyluke@xxxxxxxx>
>
>
> To: hjcho616 <hjcho616@xxxxxxxxx>
> Cc: Mohd Bazli Ab Karim <bazli.abkarim@xxxxxxxx>;
> "ceph-users@xxxxxxxxxxxxxx" <ceph-users@xxxxxxxxxxxxxx>
> Sent: Friday, March 21, 2014 1:17 AM
>
> Subject: RE: MDS crash when client goes to sleep
>
>
> Hi Hong,
>
> That's interesting, for Mr. Bazli and I, we ended with MDS stuck in
> (up:replay) and a flapping ceph-mds daemon, but then again we are using
> version 0.72.2. Having said so the triggering point seem similar to us as
> well, which is the following line:
>
> -38> 2014-03-20 20:08:44.495565 7fee3d7c4700 0 -- 192.168.1.20:6801/17079
>>> 192.168.1.101:0/2113152127 pipe(0x3f03b80 sd=18 :6801 s=0 pgs=0 cs=0 l=0
> c=0x1f0e2160).accept we reset (peer sent cseq 2), sending RESETSESSION
>
> So how long did your client go into sleep? Was there any I/O prior to the
> sleep?
>
> Regards,
> Luke
>
> From: hjcho616 [mailto:hjcho616@xxxxxxxxx]
> Sent: Friday, 21 March, 2014 12:09 PM
> To: Luke Jing Yuan
> Cc: Mohd Bazli Ab Karim; ceph-users@xxxxxxxxxxxxxx
> Subject: Re: MDS crash when client goes to sleep
>
> Nope just these segfaults.
>
> [149884.709608] ceph-mds[17366]: segfault at 200 ip 00007f09de9d60b8 sp
> 00007f09db461520 error 4 in libgcc_s.so.1[7f09de9c7000+15000]
> [211263.265402] ceph-mds[17135]: segfault at 200 ip 00007f59eec280b8 sp
> 00007f59eb6b3520 error 4 in libgcc_s.so.1[7f59eec19000+15000]
> [214638.927759] ceph-mds[16896]: segfault at 200 ip 00007fcb2c89e0b8 sp
> 00007fcb29329520 error 4 in libgcc_s.so.1[7fcb2c88f000+15000]
> [289338.461271] ceph-mds[20878]: segfault at 200 ip 00007f4b7211c0b8 sp
> 00007f4b6eba7520 error 4 in libgcc_s.so.1[7f4b7210d000+15000]
> [373738.961475] ceph-mds[21341]: segfault at 200 ip 00007f36c3d480b8 sp
> 00007f36c07d3520 error 4 in libgcc_s.so.1[7f36c3d39000+15000]
>
> Regards,
> Hong
>
> ________________________________________
> From: Luke Jing Yuan <jyluke@xxxxxxxx>
> To: hjcho616 <hjcho616@xxxxxxxxx>
> Cc: Mohd Bazli Ab Karim <bazli.abkarim@xxxxxxxx>;
> "ceph-users@xxxxxxxxxxxxxx" <ceph-users@xxxxxxxxxxxxxx>
> Sent: Thursday, March 20, 2014 10:53 PM
> Subject: Re: MDS crash when client goes to sleep
>
> Did you see any messages in dmesg saying ceph-mds respawnning or stuffs like
> that?
>
> Regards,
> Luke
>
> On Mar 21, 2014, at 11:09 AM, "hjcho616" <hjcho616@xxxxxxxxx> wrote:
> On client, I was no longer able to access the filesystem. It would hang.
> Makes sense since MDS has crashed. I tried running 3 MDS demon on the same
> machine. Two crashes and one appears to be hung up(?). ceph health says MDS
> is in degraded state when that happened.
>
> I was able to recover by restarting every node. I currently have three
> machine, one with MDS and MON, and two with OSDs.
>
> It is failing everytime my client machine goes to sleep. If you need me to
> run something let me know what and how.
>
> Regards,
> Hong
>
> ________________________________________
> From: Mohd Bazli Ab Karim <bazli.abkarim@xxxxxxxx>
> To: hjcho616 <hjcho616@xxxxxxxxx>; "ceph-users@xxxxxxxxxxxxxx"
> <ceph-users@xxxxxxxxxxxxxx>
> Sent: Thursday, March 20, 2014 9:40 PM
> Subject: RE: MDS crash when client goes to sleep
>
> Hi Hong,
> May I know what has happened to your MDS once it crashed? Was it able to
> recover from replay?
> We also facing this issue and I am interested to know on how to reproduce
> it.
>
> Thanks.
> Bazli
>
> From: ceph-users-bounces@xxxxxxxxxxxxxx
> [mailto:ceph-users-bounces@xxxxxxxxxxxxxx] On Behalf Of hjcho616
> Sent: Friday, March 21, 2014 10:29 AM
> To: ceph-users@xxxxxxxxxxxxxx
> Subject: MDS crash when client goes to sleep
>
> When CephFS is mounted on a client and when client decides to go to sleep,
> MDS segfaults. Has anyone seen this? Below is a part of MDS log. This
> happened in emperor and recent 0.77 release. I am running Debian Wheezy
> with testing kernels 3.13. What can I do to not crash the whole system if a
> client goes to sleep (and looks like disconnect may do the same)? Let me
> know if you need any more info.
>
> Regards,
> Hong
>
> -43> 2014-03-20 20:08:42.463357 7fee3f0cf700 1 -- 192.168.1.20:6801/17079
> --> 192.168.1.20:6789/0 -- mdsbeacon(6798/MDS1.2 up:active seq 21120 v6970)
> v2 -- ?+0 0x1ee9f080 con 0x2e56580
> -42> 2014-03-20 20:08:42.463787 7fee411d4700 1 -- 192.168.1.20:6801/17079
> <== mon.0 192.168.1.20:6789/0 21764 ==== mdsbeacon(6798/MDS1.2 up:active seq
> 21120 v6970) v2 ==== 108+0+0 (266728949 0 0) 0x1ee88dc0 con 0x2e56580
> -41> 2014-03-20 20:08:43.373099 7fee3f0cf700 2 mds.0.cache
> check_memory_usage total 665384, rss 503156, heap 24656, malloc 463874 mmap
> 0, baseline 16464, buffers 0, max 1048576, 0 / 62380 inodes have caps, 0
> caps, 0 caps per inode
> -40> 2014-03-20 20:08:44.494963 7fee3d7c4700 1 -- 192.168.1.20:6801/17079
>>> :/0 pipe(0x3f03b80 sd=18 :6801 s=0 pgs=0 cs=0 l=0 c=0x1f0e2160).accept
> sd=18 192.168.1.101:52026/0
> -39> 2014-03-20 20:08:44.495033 7fee3d7c4700 0 -- 192.168.1.20:6801/17079
>>> 192.168.1.101:0/2113152127 pipe(0x3f03b80 sd=18 :6801 s=0 pgs=0 cs=0 l=0
> c=0x1f0e2160).accept peer addr is really 192.168.1.101:0/2113152127 (socket
> is 192.168.1.101:52026/0)
> -38> 2014-03-20 20:08:44.495565 7fee3d7c4700 0 -- 192.168.1.20:6801/17079
>>> 192.168.1.101:0/2113152127 pipe(0x3f03b80 sd=18 :6801 s=0 pgs=0 cs=0 l=0
> c=0x1f0e2160).accept we reset (peer sent cseq 2), sending RESETSESSION
> -37> 2014-03-20 20:08:44.496015 7fee3d7c4700 2 -- 192.168.1.20:6801/17079
>>> 192.168.1.101:0/2113152127 pipe(0x3f03b80 sd=18 :6801 s=4 pgs=0 cs=0 l=0
> c=0x1f0e2160).fault 0: Success
> -36> 2014-03-20 20:08:44.496099 7fee411d4700 5 mds.0.35 ms_handle_reset
> on 192.168.1.101:0/2113152127
> -35> 2014-03-20 20:08:44.496120 7fee411d4700 3 mds.0.35 ms_handle_reset
> closing connection for session client.6019 192.168.1.101:0/2113152127
> -34> 2014-03-20 20:08:44.496207 7fee411d4700 1 -- 192.168.1.20:6801/17079
> mark_down 0x1f0e2160 -- pipe dne
> -33> 2014-03-20 20:08:44.653628 7fee3d7c4700 1 -- 192.168.1.20:6801/17079
>>> :/0 pipe(0x3d8e000 sd=18 :6801 s=0 pgs=0 cs=0 l=0 c=0x1f0e22c0).accept
> sd=18 192.168.1.101:52027/0
> -32> 2014-03-20 20:08:44.653677 7fee3d7c4700 0 -- 192.168.1.20:6801/17079
>>> 192.168.1.101:0/2113152127 pipe(0x3d8e000 sd=18 :6801 s=0 pgs=0 cs=0 l=0
> c=0x1f0e22c0).accept peer addr is really 192.168.1.101:0/2113152127 (socket
> is 192.168.1.101:52027/0)
> -31> 2014-03-20 20:08:44.925618 7fee411d4700 1 -- 192.168.1.20:6801/17079
> <== client.6019 192.168.1.101:0/2113152127 1 ==== client_reconnect(77349
> caps) v2 ==== 0+0+11032578 (0 0 3293767716) 0x2e92780 con 0x1f0e22c0
> -30> 2014-03-20 20:08:44.925682 7fee411d4700 1 mds.0.server no longer in
> reconnect state, ignoring reconnect, sending close
> -29> 2014-03-20 20:08:44.925735 7fee411d4700 0 log [INF] : denied
> reconnect attempt (mds is up:active) from client.6019
> 192.168.1.101:0/2113152127 after 2014-03-20 20:08:44.925679 (allowed
> interval 45)
> -28> 2014-03-20 20:08:44.925748 7fee411d4700 1 -- 192.168.1.20:6801/17079
> --> 192.168.1.101:0/2113152127 -- client_session(close) v1 -- ?+0 0x3ea6540
> con 0x1f0e22c0
> -27> 2014-03-20 20:08:44.927727 7fee3d7c4700 2 -- 192.168.1.20:6801/17079
>>> 192.168.1.101:0/2113152127 pipe(0x3d8e000 sd=18 :6801 s=2 pgs=135 cs=1
> l=0 c=0x1f0e22c0).reader couldn't read tag, Success
> -26> 2014-03-20 20:08:44.927797 7fee3d7c4700 2 -- 192.168.1.20:6801/17079
>>> 192.168.1.101:0/2113152127 pipe(0x3d8e000 sd=18 :6801 s=2 pgs=135 cs=1
> l=0 c=0x1f0e22c0).fault 0: Success
> -25> 2014-03-20 20:08:44.927849 7fee3d7c4700 0 -- 192.168.1.20:6801/17079
>>> 192.168.1.101:0/2113152127 pipe(0x3d8e000 sd=18 :6801 s=2 pgs=135 cs=1
> l=0 c=0x1f0e22c0).fault, server, going to standby
> -24> 2014-03-20 20:08:46.372279 7fee401d2700 10 monclient: tick
> -23> 2014-03-20 20:08:46.372339 7fee401d2700 10 monclient:
> _check_auth_rotating have uptodate secrets (they expire after 2014-03-20
> 20:08:16.372333)
> -22> 2014-03-20 20:08:46.372373 7fee401d2700 10 monclient: renew subs?
> (now: 2014-03-20 20:08:46.372372; renew after: 2014-03-20 20:09:56.370811)
> -- no
> -21> 2014-03-20 20:08:46.372403 7fee401d2700 10 log_queue is 1 last_log 2
> sent 1 num 1 unsent 1 sending 1
> -20> 2014-03-20 20:08:46.372421 7fee401d2700 10 will send 2014-03-20
> 20:08:44.925741 mds.0 192.168.1.20:6801/17079 2 : [INF] denied reconnect
> attempt (mds is up:active) from client.6019 192.168.1.101:0/2113152127 after
> 2014-03-20 20:08:44.925679 (allowed interval 45)
> -19> 2014-03-20 20:08:46.372466 7fee401d2700 10 monclient:
> _send_mon_message to mon.MDS1 at 192.168.1.20:6789/0
> -18> 2014-03-20 20:08:46.372483 7fee401d2700 1 -- 192.168.1.20:6801/17079
> --> 192.168.1.20:6789/0 -- log(1 entries) v1 -- ?+0 0x71d8d80 con 0x2e56580
> -17> 2014-03-20 20:08:46.463496 7fee3f0cf700 10 monclient:
> _send_mon_message to mon.MDS1 at 192.168.1.20:6789/0
> -16> 2014-03-20 20:08:46.463524 7fee3f0cf700 1 -- 192.168.1.20:6801/17079
> --> 192.168.1.20:6789/0 -- mdsbeacon(6798/MDS1.2 up:active seq 21121 v6970)
> v2 -- ?+0 0x1ee91340 con 0x2e56580
> -15> 2014-03-20 20:08:46.499688 7fee411d4700 1 -- 192.168.1.20:6801/17079
> <== mon.0 192.168.1.20:6789/0 21765 ==== log(last 2) v1 ==== 24+0+0
> (1174756693 0 0) 0x3ea7a40 con 0x2e56580
> -14> 2014-03-20 20:08:46.499751 7fee411d4700 10 handle_log_ack log(last 2)
> v1
> -13> 2014-03-20 20:08:46.499757 7fee411d4700 10 logged 2014-03-20
> 20:08:44.925741 mds.0 192.168.1.20:6801/17079 2 : [INF] denied reconnect
> attempt (mds is up:active) from client.6019 192.168.1.101:0/2113152127 after
> 2014-03-20 20:08:44.925679 (allowed interval 45)
> -12> 2014-03-20 20:08:46.499825 7fee411d4700 1 -- 192.168.1.20:6801/17079
> <== mon.0 192.168.1.20:6789/0 21766 ==== mdsbeacon(6798/MDS1.2 up:active seq
> 21121 v6970) v2 ==== 108+0+0 (51773011 0 0) 0x1ee25600 con 0x2e56580
> -11> 2014-03-20 20:08:47.303323 7fee3ddca700 1 -- 192.168.1.20:6801/17079
>>> :/0 pipe(0x3e92280 sd=21 :6801 s=0 pgs=0 cs=0 l=0 c=0x1f0e3080).accept
> sd=21 192.168.1.101:52029/0
> -10> 2014-03-20 20:08:47.308830 7fee3ddca700 0 -- 192.168.1.20:6801/17079
>>> 192.168.1.101:0/2113152127 pipe(0x3e92280 sd=21 :6801 s=0 pgs=0 cs=0 l=0
> c=0x1f0e3080).accept peer addr is really 192.168.1.101:0/2113152127 (socket
> is 192.168.1.101:52029/0)
> -9> 2014-03-20 20:08:47.309221 7fee3ddca700 0 --
> 192.168.1.20:6801/17079 >> 192.168.1.101:0/2113152127 pipe(0x3e92280 sd=21
> :6801 s=0 pgs=0 cs=0 l=0 c=0x1f0e3080).accept connect_seq 0 vs existing 1
> state standby
> -8> 2014-03-20 20:08:47.309251 7fee3ddca700 0 --
> 192.168.1.20:6801/17079 >> 192.168.1.101:0/2113152127 pipe(0x3e92280 sd=21
> :6801 s=0 pgs=0 cs=0 l=0 c=0x1f0e3080).accept peer reset, then tried to
> connect to us, replacing
> -7> 2014-03-20 20:08:47.309337 7fee411d4700 5 mds.0.35
> ms_handle_remote_reset on 192.168.1.101:0/2113152127
> -6> 2014-03-20 20:08:47.309357 7fee411d4700 3 mds.0.35
> ms_handle_remote_reset closing connection for session client.6019
> 192.168.1.101:0/2113152127
> -5> 2014-03-20 20:08:47.309385 7fee411d4700 1 --
> 192.168.1.20:6801/17079 mark_down 0x1f0e22c0 -- 0x3e92280
> -4> 2014-03-20 20:08:47.309478 7fee411d4700 5 mds.0.35 ms_handle_reset
> on 192.168.1.101:0/2113152127
> -3> 2014-03-20 20:08:47.309490 7fee411d4700 3 mds.0.35 ms_handle_reset
> closing connection for session client.6019 192.168.1.101:0/2113152127
> -2> 2014-03-20 20:08:47.309501 7fee411d4700 1 --
> 192.168.1.20:6801/17079 mark_down 0x1f0e3080 -- 0x3e92280
> -1> 2014-03-20 20:08:47.309485 7fee3ddca700 2 --
> 192.168.1.20:6801/17079 >> 192.168.1.101:0/2113152127 pipe(0x3e92280 sd=21
> :6801 s=4 pgs=136 cs=1 l=0 c=0x1f0e22c0).accept read error on
> newly_acked_seq
> 0> 2014-03-20 20:08:47.310724 7fee411d4700 -1 *** Caught signal
> (Segmentation fault) **
> in thread 7fee411d4700
>
> ceph version 0.77 (1bca9c5c412b3af722d5250f07fd562a23cf35ff)
> 1: /usr/bin/ceph-mds() [0x8baf22]
> 2: (()+0xf880) [0x7fee458ef880]
> NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to
> interpret this.
>
> --- logging levels ---
> 0/ 5 none
> 0/ 1 lockdep
> 0/ 1 context
> 1/ 1 crush
> 1/ 5 mds
> 1/ 5 mds_balancer
> 1/ 5 mds_locker
> 1/ 5 mds_log
> 1/ 5 mds_log_expire
> 1/ 5 mds_migrator
> 0/ 1 buffer
> 0/ 1 timer
> 0/ 1 filer
> 0/ 1 striper
> 0/ 1 objecter
> 0/ 5 rados
> 0/ 5 rbd
> 0/ 5 journaler
> 0/ 5 objectcacher
> 0/ 5 client
> 0/ 5 osd
> 0/ 5 optracker
> 0/ 5 objclass
> 1/ 3 filestore
> 1/ 3 keyvaluestore
> 1/ 3 journal
> 0/ 5 ms
> 1/ 5 mon
> 0/10 monc
> 1/ 5 paxos
> 0/ 5 tp
> 1/ 5 auth
> 1/ 5 crypto
> 1/ 1 finisher
> 1/ 5 heartbeatmap
> 1/ 5 perfcounter
> 1/ 5 rgw
> 1/ 5 javaclient
> 1/ 5 asok
> 1/ 1 throttle
> -2/-2 (syslog threshold)
> -1/-1 (stderr threshold)
> max_recent 10000
> max_new 1000
> log_file /var/log/ceph/ceph-mds.MDS1.2.log
> --- end dump of recent events ---
>
>
>
> ________________________________________
> DISCLAIMER:
>
> This e-mail (including any attachments) is for the addressee(s) only and may
> be confidential, especially as regards personal data. If you are not the
> intended recipient, please note that any dealing, review, distribution,
> printing, copying or use of this e-mail is strictly prohibited. If you have
> received this email in error, please notify the sender immediately and
> delete the original message (including any attachments).
>
> MIMOS Berhad is a research and development institution under the purview of
> the Malaysian Ministry of Science, Technology and Innovation. Opinions,
> conclusions and other information in this e-mail that do not relate to the
> official business of MIMOS Berhad and/or its subsidiaries shall be
> understood as neither given nor endorsed by MIMOS Berhad and/or its
> subsidiaries and neither MIMOS Berhad nor its subsidiaries accepts
> responsibility for the same. All liability arising from or in connection
> with computer viruses and/or corrupted e-mails is excluded to the fullest
> extent permitted by law.
>
> _______________________________________________
> ceph-users mailing list
> ceph-users@xxxxxxxxxxxxxx
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
> ________________________________________
> DISCLAIMER:
>
> This e-mail (including any attachments) is for the addressee(s) only and may
> be confidential, especially as regards personal data. If you are not the
> intended recipient, please note that any dealing, review, distribution,
> printing, copying or use of this e-mail is strictly prohibited. If you have
> received this email in error, please notify the sender immediately and
> delete the original message (including any attachments).
>
> MIMOS Berhad is a research and development institution under the purview of
> the Malaysian Ministry of Science, Technology and Innovation. Opinions,
> conclusions and other information in this e-mail that do not relate to the
> official business of MIMOS Berhad and/or its subsidiaries shall be
> understood as neither given nor endorsed by MIMOS Berhad and/or its
> subsidiaries and neither MIMOS Berhad nor its subsidiaries accepts
> responsibility for the same. All liability arising from or in connection
> with computer viruses and/or corrupted e-mails is excluded to the fullest
> extent permitted by law.
> ------------------------------------------------------------------
> -
> -
> DISCLAIMER:
>
> This e-mail (including any attachments) is for the addressee(s)
> only and may contain confidential information. If you are not the
> intended recipient, please note that any dealing, review,
> distribution, printing, copying or use of this e-mail is strictly
> prohibited. If you have received this email in error, please notify
> the sender immediately and delete the original message.
> MIMOS Berhad is a research and development institution under
> the purview of the Malaysian Ministry of Science, Technology and
> Innovation. Opinions, conclusions and other information in this e-
> mail that do not relate to the official business of MIMOS Berhad
> and/or its subsidiaries shall be understood as neither given nor
> endorsed by MIMOS Berhad and/or its subsidiaries and neither
> MIMOS Berhad nor its subsidiaries accepts responsibility for the
> same. All liability arising from or in connection with computer
> viruses and/or corrupted e-mails is excluded to the fullest extent
> permitted by law.
>
>
>
> ________________________________
> DISCLAIMER:
>
>
> This e-mail (including any attachments) is for the addressee(s) only and may
> be confidential, especially as regards personal data. If you are not the
> intended recipient, please note that any dealing, review, distribution,
> printing, copying or use of this e-mail is strictly prohibited. If you have
> received this email in error, please notify the sender immediately and
> delete the original message (including any attachments).
>
>
> MIMOS Berhad is a research and development institution under the purview of
> the Malaysian Ministry of Science, Technology and Innovation. Opinions,
> conclusions and other information in this e-mail that do not relate to the
> official business of MIMOS Berhad and/or its subsidiaries shall be
> understood as neither given nor endorsed by MIMOS Berhad and/or its
> subsidiaries and neither MIMOS Berhad nor its subsidiaries accepts
> responsibility for the same. All liability arising from or in connection
> with computer viruses and/or corrupted e-mails is excluded to the fullest
> extent permitted by law.
>
> ------------------------------------------------------------------
> -
> -
> DISCLAIMER:
>
> This e-mail (including any attachments) is for the addressee(s)
> only and may contain confidential information. If you are not the
> intended recipient, please note that any dealing, review,
> distribution, printing, copying or use of this e-mail is strictly
> prohibited. If you have received this email in error, please notify
> the sender immediately and delete the original message.
> MIMOS Berhad is a research and development institution under
> the purview of the Malaysian Ministry of Science, Technology and
> Innovation. Opinions, conclusions and other information in this e-
> mail that do not relate to the official business of MIMOS Berhad
> and/or its subsidiaries shall be understood as neither given nor
> endorsed by MIMOS Berhad and/or its subsidiaries and neither
> MIMOS Berhad nor its subsidiaries accepts responsibility for the
> same. All liability arising from or in connection with computer
> viruses and/or corrupted e-mails is excluded to the fullest extent
> permitted by law.
>
>
>
>
> ________________________________
> DISCLAIMER:
>
> This e-mail (including any attachments) is for the addressee(s) only and may
> be confidential, especially as regards personal data. If you are not the
> intended recipient, please note that any dealing, review, distribution,
> printing, copying or use of this e-mail is strictly prohibited. If you have
> received this email in error, please notify the sender immediately and
> delete the original message (including any attachments).
>
>
> MIMOS Berhad is a research and development institution under the purview of
> the Malaysian Ministry of Science, Technology and Innovation. Opinions,
> conclusions and other information in this e-mail that do not relate to the
> official business of MIMOS Berhad and/or its subsidiaries shall be
> understood as neither given nor endorsed by MIMOS Berhad and/or its
> subsidiaries and neither MIMOS Berhad nor its subsidiaries accepts
> responsibility for the same. All liability arising from or in connection
> with computer viruses and/or corrupted e-mails is excluded to the fullest
> extent permitted by law.
>
>
> _______________________________________________
> ceph-users mailing list
> ceph-users@xxxxxxxxxxxxxx
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
