Hi Steven, When using synchronous (=non-persistent) notifications, the overall rate is dependent on the latency between the RGW and the endpoint to which you are sending the notifications. The protocols for sending the notifications (kafka/amqp) are using batches and are usually very efficient. However, if there is latency, this would slow down the RGW. When using asynchronous (=persistent) notifications, they are written to a RADOS-backed queue by the RGW that received the request, and then pulled from that queue by some other (or sometimes the same) RGW that sends the notifications to the endpoint. Pulling from the queue, and sending the notifications is usually very fast, however, writing to the notification queue is a RADOS operation. The amount of information written to the queue is usually small but still has the RADOS overhead, as the notifications are written one-by-one. So, in this case, the limiting factor would be the RADOS IOpS. Please let me know if this clarifies the behavior you observe? Yuval On Thu, Nov 24, 2022 at 1:27 PM Steven Goodliff <sgoodliff@xxxxxxxxx> wrote: > Hi, > > I'm really struggling with persistent bucket notifications running 17.2.3. > I can't get much more than 600 notifications a second but when changing to > async then i see higher rates using the following metric > > sum(rate(ceph_rgw_pubsub_push_ok[$__rate_interval])) > > I believe this is mainly down to being throttled by using 1 rgw rather than > all the rgw's the async method allows. > > We would prefer to use persistent but can't get the throughput we need, any > suggestions would be much appreciated. > > Thanks > > Steven > _______________________________________________ > ceph-users mailing list -- ceph-users@xxxxxxx > To unsubscribe send an email to ceph-users-leave@xxxxxxx > > _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |