Hi,
There have been various stability issues with the MDS that I reported a
while ago and most of them have been addressed and fixes will be
available in upcoming patch releases. However, there also seem to be
problems on the client side, which I have not reported so far.
Note: This report is in part inspired by a previous mail to the list
about CephFS deletion performance
(http://lists.ceph.com/pipermail/ceph-users-ceph.com/2019-September/036842.html
), but since I am not quite sure if we are actually talking about the
very same issue, I decided to start a new thread.
I tried copying 70TB of data (mostly small files like 1TB of Git
repositories) using parallel rsync jobs. I did this first using fpsync,
but after a while the client started locking up with ever increasing
load. No IO from or to the mount was possible anymore and `sync` hung
indefinitely. Even after force-unmounting the FS, I still had kernel
processes using 100% of about half my CPU cores. Remounting the FS was
not possible until forcefully rebooting the entire node.
I then tried parsyncfp, which is more considerate regarding the load and
I was able to sync the whole tree without issues after setting
`vm.dirty_background_bytes` and `vm.dirty_bytes` via `sysctl` to 1GB and
4GB (the defaults of 10 and 20% of total RAM are way too much for a
machine with 128GB of memory and write-heavy workloads). Right now, I am
running another single rsync pass, since the parallel versions cannot do
`--delete`. To ensure this one isn't locking up my system either, I use
the same sysctl settings and periodically run `sync` in the background.
So far the job has been running for a day with an average 15m load of
2.5 on a 32-thread machine).
I am not entirely sure if this is a general kernel bug or a cephfs bug.
I believe it may be possible to produce similar issues with other
kernel-space remote file systems like NFS (I had that in the past), but
generally, it seems to be much more of an issue with the cephfs kernel
driver (at least from my experience).
I am using Nautilus 14.2.3 and a single MDS with optimized recall and
cache trimming settings to avoid cache inflation issues caused by the
housekeeping thread not being able to catch up (fixed in future
releases). Switching to multiple MDSs does not seem to have an impact on
the problem).
Cheers
Janek
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
