Re: cephfs kernel client instability

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



On Thu, Jan 24, 2019 at 8:16 PM Martin Palma <martin@xxxxxxxx> wrote:
>
> We are experiencing the same issues on clients with CephFS mounted
> using the kernel client and 4.x kernels.
>
> The problem  shows up when we add new OSDs, on reboots after
> installing patches and when changing the weight.
>
> Here the logs of a misbehaving client;
>
> [6242967.890611] libceph: mon4 10.8.55.203:6789 session established
> [6242968.010242] libceph: osd534 10.7.55.23:6814 io error
> [6242968.259616] libceph: mon1 10.7.55.202:6789 io error
> [6242968.259658] libceph: mon1 10.7.55.202:6789 session lost, hunting
> for new mon
> [6242968.359031] libceph: mon4 10.8.55.203:6789 session established
> [6242968.622692] libceph: osd534 10.7.55.23:6814 io error
> [6242968.692274] libceph: mon4 10.8.55.203:6789 io error
> [6242968.692337] libceph: mon4 10.8.55.203:6789 session lost, hunting
> for new mon
> [6242968.694216] libceph: mon0 10.7.55.201:6789 session established
> [6242969.099862] libceph: mon0 10.7.55.201:6789 io error
> [6242969.099888] libceph: mon0 10.7.55.201:6789 session lost, hunting
> for new mon
> [6242969.224565] libceph: osd534 10.7.55.23:6814 io error
>
> Additional to the MON io error we also got some OSD io errors.

This isn't surprising -- the kernel client can receive osdmaps from
both monitors and OSDs.

>
> Moreover when the error occurs several clients causes a
> "MDS_CLIENT_LATE_RELEASE" error on the MDS server.
>
> We are currently running on Luminous 12.2.10 and have around 580 OSDs
> and 5 monitor nodes. The cluster is running on CentOS 7.6.
>
> The ‘osd_map_message_max’ setting is set to the default value of 40.
> But we are still getting these errors.

My advise is the same: set it to 20 or even 10.  The problem is this
setting is in terms of the number of osdmaps instead of the size of the
resulting message.  I've filed

  http://tracker.ceph.com/issues/38040

Thanks,

                Ilya
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com




[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux