Some more tests... ~journal in XFS: [root@p05151113777233 fio]# time dd if=/dev/zero of=/var/lib/ceph/osd/osd.1045/testtest bs=512 count=1 conv=fsync 1+0 records in 1+0 records out 512 bytes (512 B) copied, 0.0423365 s, 12.1 kB/s real 0m0.044s user 0m0.000s sys 0m0.001s ~directio via the FS [root@p05151113777233 fio]# time dd if=/dev/zero of=/var/lib/ceph/osd/osd.1045/testtest bs=512 count=1 oflag=direct,dsync 1+0 records in 1+0 records out 512 bytes (512 B) copied, 0.0426422 s, 12.0 kB/s real 0m0.044s user 0m0.000s sys 0m0.001s ~directio to the bdev seems to be faster (a few attempts below): [root@p05151113760120 ~]# time dd if=/dev/zero of=/dev/sdo1 bs=512 count=1 seek=$(($RANDOM*1000)) oflag=direct,dsync 1+0 records in 1+0 records out 512 bytes (512 B) copied, 0.0113362 s, 45.2 kB/s real 0m0.013s user 0m0.001s sys 0m0.001s [root@p05151113760120 ~]# time dd if=/dev/zero of=/dev/sdo1 bs=512 count=1 seek=$(($RANDOM*1000)) oflag=direct,dsync 1+0 records in 1+0 records out 512 bytes (512 B) copied, 0.0129222 s, 39.6 kB/s real 0m0.014s user 0m0.001s sys 0m0.000s [root@p05151113760120 ~]# time dd if=/dev/zero of=/dev/sdo1 bs=512 count=1 seek=$(($RANDOM*1000)) oflag=direct,dsync 1+0 records in 1+0 records out 512 bytes (512 B) copied, 0.00484823 s, 106 kB/s real 0m0.006s user 0m0.001s sys 0m0.001s [root@p05151113760120 ~]# [root@p05151113760120 ~]# time dd if=/dev/zero of=/dev/sdo1 bs=512 count=1 seek=$(($RANDOM*1000)) oflag=direct,dsync 1+0 records in 1+0 records out 512 bytes (512 B) copied, 0.00441663 s, 116 kB/s real 0m0.006s user 0m0.001s sys 0m0.001s So it looks like I really should try those O_DIRECT journals on a separate partition. Cheers, Dan On Tue, Sep 24, 2013 at 11:13 AM, Dan van der Ster <dan@xxxxxxxxxxxxxx> wrote: > On Mon, Sep 23, 2013 at 8:18 PM, Sage Weil <sage@xxxxxxxxxxx> wrote: >> You >> might try measuring that directly and comparing it to the 33ms >> append+fsync that you previously saw. > > dd with fsync is quite slow... > > [root@p05151113777233 fio]# time dd if=/dev/zero > of=/var/lib/ceph/osd/osd.1045/testtest bs=5 count=1 > 1+0 records in > 1+0 records out > 5 bytes (5 B) copied, 8.884e-05 s, 56.3 kB/s > > real 0m0.001s > user 0m0.000s > sys 0m0.001s > > > [root@p05151113777233 fio]# time dd if=/dev/zero > of=/var/lib/ceph/osd/osd.1045/testtest bs=5 count=1 conv=fsync > 1+0 records in > 1+0 records out > 5 bytes (5 B) copied, 0.0435103 s, 0.1 kB/s > > real 0m0.045s > user 0m0.001s > sys 0m0.001s -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
