Good to know. Thanks for sharing! > On 09 Dec 2014, at 10:21, Wido den Hollander <wido@xxxxxxxx> wrote: > > Hi, > > Last sunday I got a call early in the morning that a Ceph cluster was > having some issues. Slow requests and OSDs marking each other down. > > Since this is a 100% SSD cluster I was a bit confused and started > investigating. > > It took me about 15 minutes to see that fstrim was running and was > utilizing the SSDs 100%. > > On Ubuntu 14.04 there is a weekly CRON which executes fstrim-all. It > detects all mountpoints which can be trimmed and starts to trim those. > > On the Intel SSDs used here it caused them to become 100% busy for a > couple of minutes. That was enough for them to no longer respond on > heartbeats, thus timing out and being marked down. > > Luckily we had the "out interval" set to 1800 seconds on that cluster, > so no OSD was marked as "out". > > fstrim-all does not execute fstrim with a ionice priority. From what I > understand, but haven't tested yet, is that running fstrim with ionice > -c Idle should solve this. > > It's weird that this issue didn't come up earlier on that cluster, but > after killing fstrim all problems we resolved and the cluster ran > happily again. > > So watch out for fstrim on early Sunday mornings on Ubuntu! > > -- > Wido den Hollander > 42on B.V. > Ceph trainer and consultant > > Phone: +31 (0)20 700 9902 > Skype: contact42on > _______________________________________________ > ceph-users mailing list > ceph-users@xxxxxxxxxxxxxx > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com Cheers. –––– Sébastien Han Cloud Architect "Always give 100%. Unless you're giving blood." Phone: +33 (0)1 49 70 99 72 Mail: sebastien.han@xxxxxxxxxxxx Address : 11 bis, rue Roquépine - 75008 Paris Web : www.enovance.com - Twitter : @enovance
Attachment:
signature.asc
Description: Message signed with OpenPGP using GPGMail
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
