Hi all, we are observing a dramatic performance drop on our ceph file system and are wondering if this could be related to ceph fs snapshots. We are taking rotating snapshots in 2 directories and have 11 snapshots in each (ls below) as of today. We observe the performance drop with an rsync process that writes to ceph fs to another folder *without* snapshots. The performance reduction is a factor of 3 or even higher. Could this possibly be caused by the snapshots being present? Has anyone else seen something like this? The reason we consider snapshots is that not much else changed on the cluster except that we started taking rolling snapshots on the 23rd of February. In addition, the kernel symbols ceph_update_snap_trace, rebuild_snap_realms and build_snap_context show up really high in a perf report. The performance reduction seems to be present since at least 3 days. The ceph version is mimic 13.2.10. The kernel version of the rsync server is 3.10.0-1127.10.1.el7.x86_64. $ ls home/.snap 2021-02-23_183554+0100_weekly 2021-03-06_000611+0100_daily 2021-03-09_000611+0100_daily 2021-03-01_000911+0100_weekly 2021-03-07_000611+0100_daily 2021-03-10_000611+0100_daily 2021-03-04_000611+0100_daily 2021-03-08_000611+0100_daily 2021-03-11_000611+0100_daily 2021-03-05_000611+0100_daily 2021-03-08_000911+0100_weekly $ ls groups/.snap 2021-02-23_183554+0100_weekly 2021-03-06_000611+0100_daily 2021-03-09_000611+0100_daily 2021-03-01_000912+0100_weekly 2021-03-07_000611+0100_daily 2021-03-10_000612+0100_daily 2021-03-04_000611+0100_daily 2021-03-08_000611+0100_daily 2021-03-11_000612+0100_daily 2021-03-05_000611+0100_daily 2021-03-08_000911+0100_weekly Many thanks for any pointers and best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |