Re: CephFS clients waiting for lock when one of them goes slow

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hi,

1,5 GB MDS cache for 20 clients is not enough, I would start with increasing that to 4 or 8 GB and see if the problems still occur.

Regards,
Eugen


Zitat von Petr Belyaev <p.belyaev@xxxxxxxxx>:

Hi all,

I have a very small Ceph setup - 3 OSDs, 3 MDS, 3 MON, CephFS with ~20 ceph-fuse clients connected. All run version 14.2.9 (both servers and clients).

Some clients work with a relatively large CephFS directory, few tens of thousands of files, with lots of files being added and deleted all the time. Almost every day I see exactly the same pattern when system goes under load:

- One of the clients starts running slow (e.g. high CPU or out-of-memory, sometimes with OOM killer intervention) - A number of requests from other clients gets blocked for ~15 seconds waiting for the (r/w/x)lock on other files in the same directory
- Objecter requests empty
- No requests visibly running in the dump_ops_in_flight output, only those waiting for the lock

MDS cache size is 1.5 gb, total capacity - around 1TB, usual load is ~300 [rw]ops, 3-4 mb/s, most of the config is the default one.

Have somebody seen similar issues before?

Best regards,
Petr Belyaev
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx


_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx



[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux