Hi,
Well things stall occasionally (when apps are dependent on
input/output). This didn't happen as much with IET (of course if there's
resource contention things might slow down). Occasionally I see write
latency peaks of over 500ms whilst at the same time read latencies are
still below 5ms which I find very odd - especially since there's 16GB of
RAM, it should have quite some buffer space. Then again - I might just
be unlucky in monitoring and it might just be requesting blocks that are
in cache.
On average it seems to outperform IET though, for example extracting
the MySQL installer earlier went with 90MB/s+ consistently which is
pretty nice considering it's connected over gigabit (2 separate wires /
separate subnets so it sees 2 links (1 per adapter) instead of 4 (1 ->
1, 1 -> 2, 2 -> 1, 2 -> 2 if they were in the same subnet or could route
between it). We could get these speeds with IET too but they weren't as
sustainable (the rate would usually drop pretty quick). It's configured
using round-robin (default # of I/O's are IIRC 1000 to hop to the next -
didn't tune that yet) to distribute load a bit.
But occasionally things just hang for a short time (1 - 2 sec).
Unfortunately customers notice that much more than the increased
throughput on overall as they work on terminal servers and it's pretty
noticeable if things freeze for a short while.
Maybe I should try a new kernel - but since it runs production with
~20VM's on it taking it down requires quite a bit of planning or
additional storage. Usually we have some overcapacity so I could just
svmotion but due to a SAN crash at a customer they have our overcapacity
until their new SAN comes in. Very bad timing :/.
Might try turning off round-robin for some time and see if that makes a
difference. See more or less the same on 3 hosts though.
Thanks for the re'.
On 2013-04-11 18:59, Marc Fleischmann wrote:
Hi Ferry,
Thank you for your posting. The latency fluctuation is a bit strange
indeed, but hard to assess without more information.
May I ask you: "Since a couple of months we're running on LIO [...]
and we notice occasional hick-ups [...]"
What kind of hick-ups are you finding with LIO? Can you please give
us a bit more details?
Thanks very much,
Marc
-----Original Message-----
From: target-devel-owner@xxxxxxxxxxxxxxx
[mailto:target-devel-owner@xxxxxxxxxxxxxxx] On Behalf Of Ferry
Sent: Thursday, April 11, 2013 7:58 AM
To: target-devel@xxxxxxxxxxxxxxx
Subject: Buffered fileio I/O write latency over LIO
Hi there,
we had 2 SAN's (8 disk SATA in RAID-10 (mdadm) only 4GB RAM) running
IET. Whilst it was on IET the read latency was *always* above the
write latency. We used buffered fileio on IET as well. This is quite
expected in my limited view as the writes would go to RAM (buffer) on
the SAN and were written to disk a little later (but this isn't
visible from vmware oc). Reads most of the time actually have to come
from the disk, so the rust has to move and that takes time.
Since a couple of months we're running on LIO (on Ubuntu 12.10 with 2
x
12 disk SAS in RAID-10 with 16GB RAM) and we notice occasional
hick-ups as well as the write latency being pretty high too. Write
latency frequently peaks to >100ms, which I don't really get as
there's 16GB mem in the servers and it should have plenty of buffer
space with the loads we currently have on it. With IET the write
latency didn't go above 1-2ms until the buffers were full (ie when we
started writing/copying a 50GB vm for example).
For example, looking at the performance on one of the hosts for the
entire 2 SAN's:
SAN1 read latency (avg 30 mins) 4.261ms - max (30 mins) 57ms,
write latency (avg 30 mins) 7.194ms - max (30 mins) 83ms (this
SAN is the least loaded btw)
SAN2 read latency (avg 30 mins) 5.756ms - max (30 mins) 54ms
write latency (avg 30 mins) 14.744ms - max (30 mins) 106
During normal loads on the previous setup the read latencies were
*always* higher than the write latencies. The opposite is true now
(most of the time anyways).
Any ideas what might cause this? As vmware does sync writes only
these latencies seem to hinder performance a lot. Whilst this
hardware
is not new and approx the same age as the SATA disks the density is
lower and there's more platters to spread the load over. Yet it
performs worse (over iSCSI tests before production showed higher
throughput on the machine locally) in writes and oddly enough is
faster in reads (there's more memory though so it will have more in
cache).
Any ideas on what might be causing these write latencies? No clue on
where to look for it. One thing worth mentioning is that the OS now
runs from USB stick. I don't see any correlation with the USB stick
performance and LIO/mdadm, but if there is any that might explain a
lot as the stick is horribly slow due to the USB 1.1 (the chipset
reports USB 2.0 support but I think HP found it too expensive to
actually wire that to the port - tried all ports - none function as
USB 2.0 unfortunately).
Btw if I monitor the disks with iostat every 2 seconds (iostat -x 2
/dev/md0) whilst pushing lots of data to it one usually sees,
nothing, nothing, nothing, 480-520MB/s, nothing, nothing, nothing,
480-520MB/s, nothing, etc. So buffers seem to be working just fine.
Hardly ever see access to the USB stick the OS is on, but if it does
happen await times are horrible (1100ms+).
Is it possible to tune the cache/buffer by the way? I like write
cache to speed things up - but it's also a risk so I don't want too
much in cache. Preferably not more than 1G is used for write cache
and
whatever it can take for read cache. Seeing the pretty constant rate
at which it writes in iostat it seems to flush every 8-10 secs or so
it probably doesn't matter much though.
Kind regards,
--
To unsubscribe from this list: send the line "unsubscribe
target-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe target-devel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
