On Fri, Nov 12, 2010 at 5:43 AM, Sebastian Kayser <sebastian@xxxxxxxxxx> wrote:
> * Shawn Lewis <shawnlewis@xxxxxxxxxx> wrote:
>> On Thu, Nov 11, 2010 at 8:22 AM, Sebastian Kayser <sebastian@xxxxxxxxxx> wrote:
>> > Shawn Lewis sent me an email off-list and suggested that the disk itself
>> > will still do write-caching (despite O_SYNC) unless this has been
>> > explicitly disabled. I dug into the storage configuration and found an
>> > option which sounded promising ("Drive Delayed Write", not to be confused
>> > with the RAID controller write cache which was disabled from the start).
>> > Disabled the delayed writes and the IOPS went down from ~240 to
>> >
>> > * ~100 for a filesystem test (size=100g out of 2TB)
>> > * ~110 for a raw device test
>>
>> Did any of your tests use the full device or were you still setting
>> the size= parameter? Set size=2T or get rid of the size= parameter
>> entirely for the raw device test and you should see more like 70 or 80
>> iops.
>
> Shibby. :) Raw device tests without size= deliver an average of 71 IOPS,
> thus the disk actually delivers less than what the theoretical datasheet
> calculation [1] suggests. Or the non-available specs for our desktop range
> harddisk are simply a bit inferior.
>
> root@ubuntu-804-x64:~# ./fio --section=iscsi-raw patterns.fio
> iscsi-raw: (g=0): rw=randwrite, bs=4K-4K/4K-4K, ioengine=sync, iodepth=1
> Starting 1 process
> Jobs: 1 (f=1): [w] [100.0% done] [0K/311K /s] [0/76 iops] [eta 00m:00s]
> iscsi-raw: (groupid=0, jobs=1): err= 0: pid=5309
> write: io=515496KB, bw=293258B/s, iops=71, runt=1800008msec
> ...
>
>> I believe part of the reason the gain is so huge with the drive cache
>> is that you were also only using the outer diameter of the disk. If
>> you use the full disk but enable drive cache I think you'll see about
>> 140 or 150iops.
>
> That's gold! Enabled the disk write cache again, tested without size=
> and got 160-170 IOPS. Less caching effect than with size=100g.
>
> So the moral of the story is "IOPS tests which don't utilize the full
> device won't show its worst case IOPS performance and are pretty much
> misleading (unless the application which will later be put onto the
> device also only uses a similar subset of the device)"?!
>
> I wonder how many times I saw someone - including myself - fire up
> bonnie++ or iozone with a rather small test file compared to the full
> disk size ... Thanks very much everyone! This was an (overdue) eye
> opener.
In addition, testing with a file is dependent on the state of the file
system. For example, if you're using a 100GB file and there's not enough
contiguous free space to lay it out in one chunk you could have regions on
different parts of the disk. Further, even there is enough space, that space
could be near the inner or outer diameter which would also affect
performance.
There is a way to get a file address to lba map (which would give you some
insight into how a file is laid out). I think there's a syscall. Jens should
know.
>
> Sebastian
>
> [1] http://www.spinics.net/lists/fio/msg00558.html
> --
> To unsubscribe from this list: send the line "unsubscribe fio" in
> the body of a message to majordomo@xxxxxxxxxxxxxxx
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
--
To unsubscribe from this list: send the line "unsubscribe fio" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
