Hi cephers, Recently there has been a big problem in our production ceph cluster.It has been running very well for one and a half years. RBD client network and ceph public network are different, communicating through a router. Our ceph version is 0.94.5. Our IO transport is using Simple Messanger. Yesterday some of our VM (using qemu librbd) can not send IO to ceph cluster. Ceph status is healthy and no osd up/down and no pg inactive and down. When we export an rbd image through rbd export ,we find the rbd client can not connect to one osd just to say osd.34. We find thant osd.34 up and running ,but in the log we find some errors as follows: accepter no incoming connection? sd =-1 ,errer 24, too many open files. accepter no incoming connection? sd =-1 ,errer 24, too many open files. accepter no incoming connection? sd =-1 ,errer 24, too many open files. accepter no incoming connection? sd =-1 ,errer 24, too many open files. accepter no incoming connection? sd =-1 ,errer 24, too many open files. accepter no incoming connection? sd =-1 ,errer 24, too many open files. accepter no incoming connection? sd =-1 ,errer 24, too many open files. accepter no incoming connection? sd =-1 ,errer 24, too many open files. accepter no incoming connection? sd =-1 ,errer 24, too many open files. accepter no incoming connection? sd =-1 ,errer 24, too many open files. We find that our max open files is set to 200000, but filestore fd cache size is too big like 500000. I think we have some wrong fd configurations.But when there are some errors in Accepter::entry() ,it's better to assert the osd process so that new rbd client can connect to the ceph cluster and when there are some network probem, the old rbd client can also reconnect to the cluster. I do not know if there has been some fixes in upper version. Best regards, brandy _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |