500K object size > Op 13 jun 2024 om 21:11 heeft Anthony D'Atri <aad@xxxxxxxxxxxxxx> het volgende geschreven: > > How large are the objects you tested with? > >> On Jun 13, 2024, at 14:46, sinan@xxxxxxxx wrote: >> >> I have doing some further testing. >> >> My RGW pool is placed on spinning disks. >> I created a 2nd RGW data pool, placed on flash disks. >> >> Benchmarking on HDD pool: >> Client 1 -> 1 RGW Node: 150 obj/s >> Client 1-5 -> 1 RGW Node: 150 ob/s (30 obj/s each client) >> Client 1 -> HAProxy -> 3 RGW Nodes: 150 obj/s >> Client 1-5 -> HAProxy -> 3 RGW Nodes: 150 obj/s (30 obj/s each client) >> >> I did the same tests towards the RGW pool on flash disks: same results >> >> So, it doesn't matter if my pool is hosted on HDD or SSD. >> It doesn't matter if I am using 1 RGW or 3 RGW nodes. >> It doesn't matter if I am using 1 client or 5 clients. >> >> I am constantly limited at around 140-160 objects/s. >> >> I see some TCP Retransmissions on the RGW Node, but maybe thats 'normal'. >> >> Any ideas/suggestions? >> >> On 2024-06-11 22:08, Anthony D'Atri wrote: >>>> I am not sure adding more RGW's will increase the performance. >>> That was a tangent. >>>> To be clear, that means whatever.rgw.buckets.index ? >>>>>> No, sorry my bad. .index is 32 and .data is 256. >>>>> Oh, yeah. Does `ceph osd df` show you at the far right like 4-5 PG replicas on each OSD? You want (IMHO) to end up with 100-200, keeping each pool's pg_num to a power of 2 ideally. >>>> No, my RBD pool is larger. My average PG per OSD is round 60-70. >>> Ah. Aim for 100-200 with spinners. >>>>> Assuming all your pools span all OSDs, I suggest at a minimum 256 for .index and 8192 for .data, assuming you have only RGW pools. And would be included to try 512 / 8192. Assuming your other minor pools are at 32, I'd bump .log and .non-ec to 128 or 256 as well. >>>>> If you have RBD or other pools colocated, those numbers would change. >>>>> ^ above assume disabling the autoscaler >>>> I bumped my .data pool from 256 to 1024 and .index from 32 to 128. >>> Your index pool still only benefits from half of your OSDs with a value of 128. >>>> Also doubled the .non-e and .log pools. Performance wise I don't see any improvement. If I would see 10-20% improvement, I definitely would increase it to 512 / 8192. >>>> With 0.5MB object size I am still limited at about 150 up to 250 objects/s. >>>> The disks aren't saturated. The wr await is mostly around 1ms and does not get higher when benchmarking with S3. >>> Trust iostat about as far as you can throw it. >>>> Other suggestions, or does anyone else has suggestions? >> _______________________________________________ >> ceph-users mailing list -- ceph-users@xxxxxxx >> To unsubscribe send an email to ceph-users-leave@xxxxxxx > _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |