On Thu, Dec 22, 2022 at 3:41 PM Roose, Marco <marco.roose@xxxxxxxxxxxxx> wrote:
>
> Hi Ilya,
> thanks for providing the revert. Using that commit all is fine:
>
> ~# uname -a
> Linux S1020-CephTest 6.1.0+ #1 SMP PREEMPT_DYNAMIC Thu Dec 22 14:30:22 CET
> 2022 x86_64 x86_64 x86_64 GNU/Linux
>
> ~# rsync -ah --progress /root/test-file_1000MB /mnt/ceph/test-file_1000MB
> sending incremental file list
> test-file_1000MB
> 1.00G 100% 90.53MB/s 0:00:10 (xfr#1, to-chk=0/1)
>
> I attach some ceph reports taking before, during and after an rsync on a bad
> kernel (5.6.0) for debugging.
I see two CephFS data pools and one of them is nearfull:
"pool": 10,
"pool_name": "cephfs_data",
"create_time": "2020-11-22T08:19:53.701636+0100",
"flags": 1,
"flags_names": "hashpspool",
"pool": 11,
"pool_name": "cephfs_data_ec",
"create_time": "2020-11-22T08:22:01.779715+0100",
"flags": 2053,
"flags_names": "hashpspool,ec_overwrites,nearfull",
How is this CephFS filesystem is configured? If you end up writing to
cephfs_data_ec pool there, the slowness is expected. nearfull makes
the client revert to synchronous writes so that it can properly return
ENOSPC error when nearfull develops into full. That is the whole point
of the commit that you landed upon when bisecting so of course
reverting it helps:
- if (ceph_osdmap_flag(&fsc->client->osdc, CEPH_OSDMAP_NEARFULL))
+ if ((map_flags & CEPH_OSDMAP_NEARFULL) ||
+ (pool_flags & CEPH_POOL_FLAG_NEARFULL))
iocb->ki_flags |= IOCB_DSYNC;
Thanks,
Ilya
