>>>> >>> You are right here, but we use Ceph mainly for RBD. It performs 'good enough' for our RBD load. >> You use RBD for archival? > > No, storage for (light-weight) virtual machines. I'm surprised that it's enough, I've seen HDDs fail miserably in that role. > The (CPU) load on the OSD nodes is quite low. Our MON/MGR/RGW aren't hosted on the OSD nodes and are running on modern hardware. >> You didn't list additional nodes so I assumed. You might still do well to have a larger number of RGWs, wherever they run. RGWs often scale better horizontally than vertically. > > Good to know. I'll check if adding more RGW nodes is possible. To be clear, you don't need more nodes. You can add RGWs to the ones you already have. You have 12 OSD nodes - why not put an RGW on each? > >> There are still serializations in the OSD and PG code. You have 240 OSDs, does your index pool have *at least* 256 PGs? > > Index as the data pool has 256 PG's. To be clear, that means whatever.rgw.buckets.index ? > >>>> You might also disable Nagle on the RGW nodes. >>> I need to lookup what that exactly is and does. >>>>> It depends on the concurrency setting of Warp. >>>>> It look likes the objects/s is the bottleneck, not the throughput. >>>>> Max memory usage is about 80-90GB per node. CPU's are quite idling. >>>>> Is it reasonable to expect more IOps / objects/s for RGW with my setup? At this moment I am not able to find the bottleneck what is causing the low obj/s. >>>> HDDs are a false economy. >>> Got it :) >>>>> Ceph version is 15.2. >>>>> Thanks! >>>>> _______________________________________________ >>>>> ceph-users mailing list -- ceph-users@xxxxxxx >>>>> To unsubscribe send an email to ceph-users-leave@xxxxxxx >>> _______________________________________________ >>> ceph-users mailing list -- ceph-users@xxxxxxx >>> To unsubscribe send an email to ceph-users-leave@xxxxxxx > _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |