On Thu, Jan 24, 2019 at 8:16 PM Martin Palma <martin@xxxxxxxx> wrote: > > We are experiencing the same issues on clients with CephFS mounted > using the kernel client and 4.x kernels. > > The problem shows up when we add new OSDs, on reboots after > installing patches and when changing the weight. > > Here the logs of a misbehaving client; > > [6242967.890611] libceph: mon4 10.8.55.203:6789 session established > [6242968.010242] libceph: osd534 10.7.55.23:6814 io error > [6242968.259616] libceph: mon1 10.7.55.202:6789 io error > [6242968.259658] libceph: mon1 10.7.55.202:6789 session lost, hunting > for new mon > [6242968.359031] libceph: mon4 10.8.55.203:6789 session established > [6242968.622692] libceph: osd534 10.7.55.23:6814 io error > [6242968.692274] libceph: mon4 10.8.55.203:6789 io error > [6242968.692337] libceph: mon4 10.8.55.203:6789 session lost, hunting > for new mon > [6242968.694216] libceph: mon0 10.7.55.201:6789 session established > [6242969.099862] libceph: mon0 10.7.55.201:6789 io error > [6242969.099888] libceph: mon0 10.7.55.201:6789 session lost, hunting > for new mon > [6242969.224565] libceph: osd534 10.7.55.23:6814 io error > > Additional to the MON io error we also got some OSD io errors. This isn't surprising -- the kernel client can receive osdmaps from both monitors and OSDs. > > Moreover when the error occurs several clients causes a > "MDS_CLIENT_LATE_RELEASE" error on the MDS server. > > We are currently running on Luminous 12.2.10 and have around 580 OSDs > and 5 monitor nodes. The cluster is running on CentOS 7.6. > > The ‘osd_map_message_max’ setting is set to the default value of 40. > But we are still getting these errors. My advise is the same: set it to 20 or even 10. The problem is this setting is in terms of the number of osdmaps instead of the size of the resulting message. I've filed http://tracker.ceph.com/issues/38040 Thanks, Ilya _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |