Re: cephfs hangs on writes

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Some new information:

It looks like the hang happens at the 4194304 byte boundary. For example:
`dd if=/dev/zero of=zero bs=4194304 count=1` succeeds, but
`dd if=/dev/zero of=zero bs=4194305 count=1` hangs with an empty file.

If bs is set lower, e.g. 8192, then 4194304 bytes are written to a file and then the hang happens. The problem is transient. Sometimes it goes away sometimes it comes back. Tested on 3.10.0-1160.62.1.el7 kernel.

I've resolved slow metadata io issues and metadata is now stored on 12 NVMe flash OSDs, so that wasn't the cause.


Vlad


On 4/24/22 16:25, Vladimir Brik wrote:
Hello

We are experiencing an issue where, sometimes, when users write to cephfs an empty file is created and then the application hangs, seemingly indefinitely. I am sometimes able to reproduce with dd.

Does anybody know what might be going on?

Some details:
- ceph health complains about 100+ slow metadata IOs
- CPU utilization of ceph-mds is low
- We have almost 200 kernel cephfs clients
- Cephfs metadata is stored on 3 OSDs that use NVMe flash AICs


Vlad
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx



[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux