It was during a random write (fio benchmark). I can't reproduce it now,I'll try to do tests again this week. ----- Mail original ----- De: "Mark Nelson" <mark.nelson@xxxxxxxxxxx> À: "Alexandre DERUMIER" <aderumier@xxxxxxxxx> Cc: "David Blundell" <David.Blundell@xxxxxxxxxxxxxxxx>, ceph-devel@xxxxxxxxxxxxxxx Envoyé: Jeudi 5 Juillet 2012 19:58:27 Objet: Re: Slow request warnings on 0.48 On 07/04/2012 11:58 AM, Alexandre DERUMIER wrote: > Hi, I see same messages here after upgrade to 0.48. > > with random write benchmark. > > I have more lags than before with 0.47 (but disks are at 100% usage, so can't tell if it's normal or not) > > > ----- Mail original ----- > > De: "David Blundell"<David.Blundell@xxxxxxxxxxxxxxxx> > À: ceph-devel@xxxxxxxxxxxxxxx > Envoyé: Mercredi 4 Juillet 2012 18:53:02 > Objet: Slow request warnings on 0.48 > > I have three servers running mon and osd using Ubuntu 12.04 that I have been testing with RADOS storing RBD KVM instances > > 0.47.3 worked extremely well (once I got over a few btrfs issues). The same servers running 0.48 give a large number of "[WRN] slow request" messages whenever I generate a lot of random IO in the KVM instances using iozone. The slow responses eventually leads to disk timeouts on the KVM instances. > > I have erased the osds and recreated on new btrfs volumes with the same result. > > I have also tried switching to xfs using mkfs.xfs -n size=64k with noatime, inode64,delaylog,logbufs=8,logbsize=256k > > Xfs gives the same result - the iozone tests run fine until the random IO starts and then there are lots of slow request warnings. > > Does anyone have any ideas about the best place to start troubleshooting / debugging? > > Thanks, > > David Hi David and Alexandre, Does this only happen with random writes or also sequential writes? If it happens with sequential writes as well, does it happen with rados bench? -- Mark Nelson Performance Engineer Inktank -- -- Alexandre D e rumier Ingénieur Systèmes et Réseaux Fixe : 03 20 68 88 85 Fax : 03 20 68 90 88 45 Bvd du Général Leclerc 59100 Roubaix 12 rue Marivaux 75002 Paris -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
