On Fri, Oct 21, 2016 at 10:56 PM, Nick Fisk <nick@xxxxxxxxxx> wrote:
> -----Original Message-----
> From: ceph-users [mailto:ceph-users-bounces@lists.ceph.com ] On Behalf Of Haomai Wang
> Sent: 21 October 2016 15:40
> To: Nick Fisk <nick@xxxxxxxxxx>
> Cc: ceph-users@xxxxxxxxxxxxxx
> Subject: Re: Ceph and TCP States
>
>
>
> On Fri, Oct 21, 2016 at 10:31 PM, Nick Fisk <mailto:nick@xxxxxxxxxx> wrote:
> > -----Original Message-----
> > From: ceph-users [mailto:mailto:ceph-users-bounces@xxxxxxxxxxxxxx ] On Behalf Of Haomai Wang
> > Sent: 21 October 2016 15:28
> > To: Nick Fisk <mailto:nick@xxxxxxxxxx>
> > Cc: mailto:ceph-users@xxxxxxxxxx.com
> > Subject: Re: Ceph and TCP States
> >
> >
> >
> > On Fri, Oct 21, 2016 at 10:19 PM, Nick Fisk <mailto:mailto:nick@xxxxxxxxxx> wrote: Yeah I think that’s where the problem lies. Most Firewalls tend to silently drop denied packets without sending RST's, so Ceph effectively just thinks that its experiencing packet loss and will never retry until the 15 minute timeout period is up. Am I right in thinking I can't tune down this parameter for a CephFS kernel client as it doesn't use the ceph.conf file?
> > Hi,
> >
> > I'm just testing out using a Ceph client in a DMZ behind a FW from the main Ceph cluster. One thing I have noticed is that if the
> > state table on the FW is emptied maybe by restarting it or just clearing the state table...etc. Then the Ceph client will hang for a
> > long time as the TCP session can no longer pass through the FW and just gets blocked instead.
> >
> > This "FW" is linux firewall or hardware FW?
>
> PFSense running on dedicated HW. Eventually they will be in a HA pair so states should persist, but trying to work around this for now.
> Bit annoying having CephFS lock hard for 15 minutes even though the network connection only went down for a few seconds.
>
> hmm, I'm not familiar with this fw. And from my view, whether RST packet sent is decided by FW. But I think you can try
> "/proc/sys/net/ipv4/tcp_keepalive_time", if FW reset tcp session, tcp keepalive should detect and send a rst.
I think cephfs kernel client doesn't have any timeout behavior. 15mins timeout is triggered by server side I guess. So you can turn down this in server side. Keep in mind that a very low value will cause frequency connection lossy.
>
> >
> >
> > I believe this behaviour can be adjusted by the "ms tcp read timeout" setting to limit its impact, but wondering if anybody has any
> > other ideas. I'm also thinking of experimenting with either stateless FW rules for Ceph or getting the FW to send back RST packets
> > instead of silently dropping packets.
> >
> > hmm, I think it depends on FW
> >
> >
> > Thanks,
> > Nick
> >
> > _______________________________________________ > > mailto:mailto:ceph-users@
> > ceph-users mailing list
lists.ceph.com
> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph. com
>
>
> _______________________________________________ > mailto:ceph-users@xxxxxxxxxx.
> ceph-users mailing list
com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph. com
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
