Hi, Last week I upgraded a 250 OSD cluster from Dumpling 0.67.10 to Firefly 0.80.7 and after the upgrade there was a severe performance drop on the cluster. It started raining slow requests after the upgrade and most of them included a 'snapc' in the request. That lead me to investigate the RBD snapshots and I found that a rogue process had created ~1800 snapshots spread out over 200 volumes. One image even had 181 snapshots! As the snapshots weren't used I removed them all and after the snapshots were removed the performance of the cluster came back to normal level again. I'm wondering what changed between Dumpling and Firefly which caused this? I saw OSDs spiking to 100% disk util constantly under Firefly where this didn't happen with Dumpling. Did something change in the way OSDs handle RBD snapshots which causes them to create more disk I/O? -- Wido den Hollander 42on B.V. Ceph trainer and consultant Phone: +31 (0)20 700 9902 Skype: contact42on -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
