That may not be related. -Sam On Fri, Jan 2, 2015 at 10:43 AM, Stefan Priebe <s.priebe@xxxxxxxxxxxx> wrote: > Am 02.01.2015 um 17:49 schrieb Samuel Just: >> >> Odd, sounds like it might be rbd client side? >> -Sam > > > That one was already on list: > https://www.mail-archive.com/ceph-devel@xxxxxxxxxxxxxxx/msg19091.html > > Sadly there was no result as it was unseen for 2 weeks and i didn't had the > test equipment anymore. > > Greets, > Stefan > > >> On Thu, Jan 1, 2015 at 1:30 AM, Stefan Priebe <s.priebe@xxxxxxxxxxxx> >> wrote: >>> >>> hi, >>> >>> Am 31.12.2014 um 17:21 schrieb Wido den Hollander: >>>> >>>> >>>> Hi, >>>> >>>> Last week I upgraded a 250 OSD cluster from Dumpling 0.67.10 to Firefly >>>> 0.80.7 and after the upgrade there was a severe performance drop on the >>>> cluster. >>>> >>>> It started raining slow requests after the upgrade and most of them >>>> included a 'snapc' in the request. >>>> >>>> That lead me to investigate the RBD snapshots and I found that a rogue >>>> process had created ~1800 snapshots spread out over 200 volumes. >>>> >>>> One image even had 181 snapshots! >>>> >>>> As the snapshots weren't used I removed them all and after the snapshots >>>> were removed the performance of the cluster came back to normal level >>>> again. >>>> >>>> I'm wondering what changed between Dumpling and Firefly which caused >>>> this? I saw OSDs spiking to 100% disk util constantly under Firefly >>>> where this didn't happen with Dumpling. >>>> >>>> Did something change in the way OSDs handle RBD snapshots which causes >>>> them to create more disk I/O? >>> >>> >>> >>> I saw the same and addionally a slowdown in librbd too, that's why i'm >>> still >>> on dumpling and won't upgrade until hammer. >>> >>> Stefan >>> >>> -- >>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in >>> the body of a message to majordomo@xxxxxxxxxxxxxxx >>> More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
