Let me just start off by saying, I really appreciate all your input so far. Its been a huge help! Even if it can scale to 10-20 per second that would make things far more viable. Sounds like it shouldn't be much of a issue with the changes you mentioned. As it sits we have roughly 1300 (and growing) LVM Logical Volumes that we would like to move over to Ceph and replicate. These are currently running on one large Nimble HPE volume. For testing I currently have 2x of the OSD nodes I plan on using in production. SuperMicro SYS-1029U-TN10RT 2x Platinum 8268 2.9Ghz CPU 10x Micron 9300 PRO 15.36TB Drives 386G Ram 1x Micron 5300 PRO 960G for OS I virtualized each "sites" cluster using Proxmox. Proxmox is also the OS of the OSD/MON VM's. The full production setup will have a dedicated 100G network for the private network and a dedicated 10G network for the public network. If things go well with this testing setup, I hope to have the production hardware by June. Possibly moving LV's to RBD's by next year at this time. I am ready/eager to help the Ceph project in anyway I can. From: "Jason Dillaman" <jdillama@xxxxxxxxxx> To: "adamb" <adamb@xxxxxxxxxx> Cc: "ceph-users" <ceph-users@xxxxxxx> Sent: Thursday, January 21, 2021 2:18:06 PM Subject: Re: Re: RBD-Mirror Snapshot Scalability On Thu, Jan 21, 2021 at 2:00 PM Adam Boyhan <adamb@xxxxxxxxxx> wrote: > > Looks like a script and cron will be a solid work around. > > Still interested to know if there are any options to make it so rbd-mirror can take more than 1 mirror snap per second. > > > > From: "adamb" <adamb@xxxxxxxxxx> > To: "ceph-users" <ceph-users@xxxxxxx> > Sent: Thursday, January 21, 2021 11:18:36 AM > Subject: RBD-Mirror Snapshot Scalability > > I have noticed that RBD-Mirror snapshot mode can only manage to take 1 snapshot per second. For example I have 21 images in a single pool. When the schedule is triggered it takes the mirror snapshot of each image 1 at a time. It doesn't feel or look like a performance issue as the OSD's are Micron 9300 PRO NVMe's and each server has 2x Intel Platinum 8268 CPU's. The creation of snapshot ids is limited by the MONs quorum process. It can issue multiple ids in a single batch, but they all need to be queued up. The most recent version of the MGR's RBD mirror snapshot scheduler works asynchronously so it can start multiple snapshots concurrently. It's much better but it still won't scale to hundreds of snapshots per second (*not that your cluster could even keep up regardless even if the MONs could). > I was hoping that adding more RDB-Mirror instance would help, but that only seems to help with overall throughput. As it sits I have 3 RBD-Mirror instances running on each cluster. > > We run a 30 minute snapshot schedule to our remote site as it is, based on that I can only squeeze 1800 mirror snaps every 30 minutes. Honestly, you might be at the bleeding edge here with an attempt to replicate >1,800 images. Getting feedback from deployments like yours can help us improve the software since we, realistically, don't have the compute resources to easily test at large scale. > I was hoping there might be something I am missing with RBD-Mirror as far as scaling goes. > > Maybe multiple pools would be a solution and have other benefits? > > > > > _______________________________________________ > ceph-users mailing list -- ceph-users@xxxxxxx > To unsubscribe send an email to ceph-users-leave@xxxxxxx > _______________________________________________ > ceph-users mailing list -- ceph-users@xxxxxxx > To unsubscribe send an email to ceph-users-leave@xxxxxxx > -- Jason _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |