Odd, sounds like it might be rbd client side? -Sam On Thu, Jan 1, 2015 at 1:30 AM, Stefan Priebe <s.priebe@xxxxxxxxxxxx> wrote: > hi, > > Am 31.12.2014 um 17:21 schrieb Wido den Hollander: >> >> Hi, >> >> Last week I upgraded a 250 OSD cluster from Dumpling 0.67.10 to Firefly >> 0.80.7 and after the upgrade there was a severe performance drop on the >> cluster. >> >> It started raining slow requests after the upgrade and most of them >> included a 'snapc' in the request. >> >> That lead me to investigate the RBD snapshots and I found that a rogue >> process had created ~1800 snapshots spread out over 200 volumes. >> >> One image even had 181 snapshots! >> >> As the snapshots weren't used I removed them all and after the snapshots >> were removed the performance of the cluster came back to normal level >> again. >> >> I'm wondering what changed between Dumpling and Firefly which caused >> this? I saw OSDs spiking to 100% disk util constantly under Firefly >> where this didn't happen with Dumpling. >> >> Did something change in the way OSDs handle RBD snapshots which causes >> them to create more disk I/O? > > > I saw the same and addionally a slowdown in librbd too, that's why i'm still > on dumpling and won't upgrade until hammer. > > Stefan > > -- > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in > the body of a message to majordomo@xxxxxxxxxxxxxxx > More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
