It's the latest CentOS 7.6 kernel. Known pain there? The user was running a 1.95TiB ior benchmark -- so, trying to do parallel writes to one single 1.95TiB file. We have max_file_size 2199023255552 (exactly 2 TiB) so it should fit. Thanks! Dan On Mon, Apr 1, 2019 at 1:06 PM Paul Emmerich <paul.emmerich@xxxxxxxx> wrote: > > Which kernel version are you using? We've had lots of problems with > random deadlocks in kernels with cephfs but 4.19 seems to be pretty > stable. > > > Paul > > -- > Paul Emmerich > > Looking for help with your Ceph cluster? Contact us at https://croit.io > > croit GmbH > Freseniusstr. 31h > 81247 München > www.croit.io > Tel: +49 89 1896585 90 > > On Mon, Apr 1, 2019 at 12:45 PM Dan van der Ster <dan@xxxxxxxxxxxxxx> wrote: > > > > Hi all, > > > > We have been benchmarking a hyperconverged cephfs cluster (kernel > > clients + osd on same machines) for awhile. Over the weekend (for the > > first time) we had one cephfs mount deadlock while some clients were > > running ior. > > > > All the ior processes are stuck in D state with this stack: > > > > [<ffffffffafdb53a3>] wait_on_page_bit+0x83/0xa0 > > [<ffffffffafdb54d1>] __filemap_fdatawait_range+0x111/0x190 > > [<ffffffffafdb5564>] filemap_fdatawait_range+0x14/0x30 > > [<ffffffffafdb79e6>] filemap_write_and_wait_range+0x56/0x90 > > [<ffffffffc0f11575>] ceph_fsync+0x55/0x420 [ceph] > > [<ffffffffafe76247>] do_fsync+0x67/0xb0 > > [<ffffffffafe76530>] SyS_fsync+0x10/0x20 > > [<ffffffffb0372d5b>] system_call_fastpath+0x22/0x27 > > [<ffffffffffffffff>] 0xffffffffffffffff > > > > We tried restarting the co-located OSDs, and tried evicting the > > client, but the processes stay deadlocked. > > > > We've seen the recent issue related to co-location > > (https://bugzilla.redhat.com/show_bug.cgi?id=1665248) but we don't > > have the `usercopy` warning in dmesg. > > > > Are there other known issues related to co-locating? > > > > Thanks! > > Dan > > _______________________________________________ > > ceph-users mailing list > > ceph-users@xxxxxxxxxxxxxx > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |