Dear Ilya, Am 22.12.22 um 16:25 schrieb Ilya Dryomov:
On Thu, Dec 22, 2022 at 3:41 PM Roose, Marco <marco.roose@xxxxxxxxxxxxx> wrote:
thanks for providing the revert. Using that commit all is fine: ~# uname -a Linux S1020-CephTest 6.1.0+ #1 SMP PREEMPT_DYNAMIC Thu Dec 22 14:30:22 CET 2022 x86_64 x86_64 x86_64 GNU/Linux ~# rsync -ah --progress /root/test-file_1000MB /mnt/ceph/test-file_1000MB sending incremental file list test-file_1000MB 1.00G 100% 90.53MB/s 0:00:10 (xfr#1, to-chk=0/1) I attach some ceph reports taking before, during and after an rsync on a bad kernel (5.6.0) for debugging.I see two CephFS data pools and one of them is nearfull: "pool": 10, "pool_name": "cephfs_data", "create_time": "2020-11-22T08:19:53.701636+0100", "flags": 1, "flags_names": "hashpspool", "pool": 11, "pool_name": "cephfs_data_ec", "create_time": "2020-11-22T08:22:01.779715+0100", "flags": 2053, "flags_names": "hashpspool,ec_overwrites,nearfull", How is this CephFS filesystem is configured? If you end up writing to cephfs_data_ec pool there, the slowness is expected. nearfull makes the client revert to synchronous writes so that it can properly return ENOSPC error when nearfull develops into full. That is the whole point of the commit that you landed upon when bisecting so of course reverting it helps: - if (ceph_osdmap_flag(&fsc->client->osdc, CEPH_OSDMAP_NEARFULL)) + if ((map_flags & CEPH_OSDMAP_NEARFULL) || + (pool_flags & CEPH_POOL_FLAG_NEARFULL)) iocb->ki_flags |= IOCB_DSYNC;
Well, that effect is not documented in the commit message, and for the user it’s a regression, that the existing (for the user working) configuration performs worse after updating the Linux kernel. That violates Linux’ no-regression policy, and at least needs to be better documented and explained.
Kind regards, Paul
