Hey folks: I was wondering if the community can provide any advice — over time and due to some external issues, we have managed to accumulate thousands of snapshots of RBD images, which are now in need of cleaning up. I have recently attempted to roll through a “for" loop to perform a “rbd snap rm” on each snapshot, sequentially, waiting until the rbd command finishes before moving onto the next one, of course. I noticed that shortly after starting this, I started seeing thousands of slow ops and a few of our guest VMs became unresponsive, naturally. My questions are: - Is this expected behavior? - Is the background cleanup asynchronous from the “rbd snap rm” command? - If so, are there any OSD parameters I can set to reduce the impact on production? - Would “rbd snap purge” be any different? I expect not, since fundamentally, rbd is performing the same action that I do via the loop. Relevant details are as follows, though I’m not sure cluster size *really* has any effect here: - Ceph: version 10.2.5 (c461ee19ecbc0c5c330aca20f7392c9a00730367) - 5 storage nodes, each with: - 10x 2TB 7200 RPM SATA Spindles (for a total of 50 OSDs) - 2x Samsung MZ7LM240 SSDs (used as journal for the OSDs) - 64GB RAM - 2x Intel(R) Xeon(R) CPU E5-2609 v3 @ 1.90GHz - 20GBit LACP Port Channel via Intel X520 Dual Port 10GbE NIC Let me know if I’ve missed something fundamental. Thanks, -- Kenneth Van Alstyne Systems Architect Knight Point Systems, LLC Service-Disabled Veteran-Owned Business 1775 Wiehle Avenue Suite 101 | Reston, VA 20190 c: 228-547-8045 f: 571-266-3106 www.knightpoint.com DHS EAGLE II Prime Contractor: FC1 SDVOSB Track GSA Schedule 70 SDVOSB: GS-35F-0646S GSA MOBIS Schedule: GS-10F-0404Y ISO 20000 / ISO 27001 / CMMI Level 3 Notice: This e-mail message, including any attachments, is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, copy, use, disclosure, or distribution is STRICTLY prohibited. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies of the original message. _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |