It would be really nice if somebody from inktank can comment this whole sitation. Thanks! Stefan Am 29.05.2012 05:54, schrieb Alexandre DERUMIER: >>> This happens with ext4 or btrfs too. > > maybe this is related to io scheduler ? > > did you have compared cfq,deadline,noop scheduler ? > > noop should be fast with ssd. > > > also what's is your sas/sata controller ? > > ----- Mail original ----- > > De: "Stefan Priebe" <s.priebe@xxxxxxxxxxxx> > À: "Alexandre DERUMIER" <aderumier@xxxxxxxxx> > Cc: ceph-devel@xxxxxxxxxxxxxxx, "Mark Nelson" <mark.nelson@xxxxxxxxxxx> > Envoyé: Lundi 28 Mai 2012 21:48:34 > Objet: Re: poor OSD performance using kernel 3.4 > > Am 28.05.2012 08:52, schrieb Alexandre DERUMIER: >>> I think filestore journal parallel works only with btrfs. >>> Other filesystem are writeahead. >>>> ... you might be right but i can't change ceph's implementation. >> >> See my schema, >> I think you see parallel writes, because you see flush write of first wave to disk, in the same time >> of second wave write to journal. > Yes i fulllý understand and agree - but still this should at least > result in a constant bandwidth near max of underlying disk. > >>>> I totally aggree with you but this is just a test setup AND if you have >>>> a big log file to copy let's say 100GB your journal will never be big >>>> enough and the speed should never drop to 0MB/s. Also i see the correct >>>> behaviour with 3.0.X where the speed is maxed to the underlying device. >>>> So i still see no reason that with 3.4 the speed drops to 0MB/s and is >>>> mostly 10-20MB/s instead of 130MB/s. >> >> Maybe something is wrong with 3.4, then your disk write more slowly. (xfs bug, sata driver controller bug, ...) > > This happens with ext4 or btrfs too. > > Squential write speed to FS is exactly the same under 3.0 and 3.4 using > oflag=direct. > > 3.4: > 10000+0 records in > 10000+0 records out > 10485760000 bytes (10 GB) copied, 41,4899 s, 253 MB/s > > 3.0: > 10000+0 records in > 10000+0 records out > 10485760000 bytes (10 GB) copied, 40,861 s, 257 MB/s > >> maybe some local benchmark of your ssd with 3.4 can give some tips ? > >>>> How many disks (7,2K) do you have by osd ? >>>>> One intel 520 SSD per OSD. >> >> I see some benchmark on internet about 150-300MB/s (depend of the blocksize). > bench OSD shows around 260MB/s > > ceph osd tell X bench shows me a speed of 260MB/s under both kernels > which corresponds to the dd from above. > >> Something must be wrong, Doing local benchmark can really help I think. >> You can use sysbench-tools >> https://github.com/tsuna/sysbench-tools >> It make bench compare with nice graphs. > Thx hopefully i'll find something. > > Stefan > > > -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
