Dear CephFS experts...
We are using Ceph and CephFS 9.2.0. CephFS clients are being mounted via ceph-fuse.
We recently noticed the firewall from certain CephFS clients dropping connections with OSDs as SRC. This is something which is not systematic but we noticed happening at least once. Here is an example of the firewall log.
IPTABLES-IN IN=eth0 OUT= MAC=<SOME_MAC_ADDRESS> SRC="" DST=<SOME_CLIENT_IP> LEN=52 TOS=0x00 PREC=0x00 TTL=63 ID=42315 DF PROTO=TCP SPT=6802 DPT=57950 WINDOW=243 RES=0x00 ACK URGP=0
and of the netstat output executed in the OSD at the same time
tcp 0 0 <SOME_OSD_IP>:6800 <SOME_CLIENT_IP>:37893 ESTABLISHED 23987/ceph-osd
tcp 0 0 <SOME_OSD_IP>:6810 <SOME_CLIENT_IP>:50875 ESTABLISHED 23971/ceph-osd
tcp 1 1 <SOME_OSD_IP>:6802 <SOME_CLIENT_IP>:57950 CLOSING - <---------
(...)
In normal situations, we always have seen that it is the ceph-fuse client that starts the connections to OSDs, and the connection back from OSDs is accepted by the client firewall because it is a related,established connection. The fact that we saw drops in our client firewalls tell us that OSDs are trying to connect to the clients as a new connection.
Up to now I haven't seen the situation again but the infrastructure is not being used at the moment while when it happened it was under (some) load. My basic dd bare tests do not show anything abnormal and I have not been able to replicate the event.
We do not understand our observation and we wonder if this is
1) something expected (and in this case we will open our clients firewalls)
2) an unexpected behaviour (such as ceph-fuse breaking the established connections and the OSD is not aware of it)
Yeah, it doesn't make much sense that the OSD would be opening a connection -- I can't think of any way this could happen. Is it possible your firewall software is referring to packets rather than streams when it identifies source and destination? (I don't do much network watching so that dump output means very little to me.)
-Greg
Any comment is welcome.
TIA
G.
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
