On 3/19/21 7:20 PM, Andrej Filipcic wrote:
Hi,
I am testing 15.2.10 on a large cluster (RH8). cephfs pool (size=1) with
122 nvme OSDs works fine till the number of clients is relatively low.
Writing from 400 kernel clients (ior benchmark), 8 streams each, causes
issues. Writes are initially fast at 100GB/s but then they drop to
<1GB/s after few minutes, while the OSDs use 300% cpu each.
My guess is that the OSDs are overloaded with requests from too many
clients, since it does not happen till there are ~3-4 streams OSD. The
OSDs log do not show anything problematic.
tried to increase osd_op_num_threads_per_shard_ssd, did not help.
Restarting OSDs recovers the situation for few minutes.
Writing to HDD pool with 1500 HDDs does not have any issues at all under
same conditions.
Any hints, settings to improve this?
Not yet, just questions. How many PG's per NVMe do you have? How much
memory per OSD (osd_memory_target) is configured? Do you have enabled
"bluefs_buffered_io" on the OSDs?
Gr. Stefan
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
