Hello again!
I have closed all the load generating services, waited a few minutes for
the server load to reach a clean 0.00 and then I have re-performed the
dd tests with various bs sizes. I was not able to setup correctly fio
with a compile error but I'll get it done.
One more thing before the results: I omitted to answer something earlier
today. CentOS was installed due to fact that cPanel is not installable
on many OSes (CentOS, RHEL and I think that's about it). So I picked
CentOS. The installation was done remotely over KVM with a minimal
CentOS CD (datacenter does not offer any server related services so we
had to do it ourselves over a Raritan KVM).
Tests were done roughly 1 minute apart.
1. First test (bs=1G): same as always.
root [~]# dd if=testfile.tar.gz of=test oflag=sync bs=1G
547682517 bytes (548 MB) copied, 53.3767 s, 10.3 MB/s
2. With a bs of 4MB: niceeee! Best result ever. I am not sure what
happened this time. However it's short lived.
root [~]# dd if=testfile.tar.gz of=test2 oflag=sync bs=4M
547682517 bytes (548 MB) copied, 4.43305 s, 124 MB/s
3. bs=2MB, starting to decay.
root [~]# dd if=testfile.tar.gz of=test3 oflag=sync bs=2M
547682517 bytes (548 MB) copied, 20.3647 s, 26.9 MB/s
4. bs=4MB again. Back to square 1.
root [~]# dd if=testfile.tar.gz of=test4 oflag=sync bs=4M
547682517 bytes (548 MB) copied, 56.7124 s, 9.7 MB/s
As services were shut down prior to the test, the biggest load it
reached was about 2.
5. Finally I restarted the services and redone the bs=4MB test (going
from a load of 0.23):
root [~]# dd if=testfile.tar.gz of=test6 oflag=sync bs=4M
547682517 bytes (548 MB) copied, 116.469 s, 4.7 MB/s
Again, I don't think my problem is related to any concurrent I/O
starvation. These SSDs or this mdraid or I don't know what simply can't
take any sustained write task. And this is not due to the server load.
Even during very low server loads it's enough to write about 1GB of data
within a short time frame (minutes) to bring the I/O system to it's
knees for a considerable time (at least tens of minutes).
4.7MB per second for writing a 548MB file starting from a load of 0.23
during off peak hours on SSDs. Nice!!!
Thanks!
On 22/04/2013 2:17 AM, Stan Hoeppner wrote:
On 4/21/2013 3:46 PM, Andrei Banu wrote:
Hello,
At this point I probably should state that I am not an experienced
sysadmin.
Things are becoming more clear now.
Knowing this, I do have a server management company but they
said they don't know what to do
So you own this hardware and it is colocated, correct?
so now I am trying to fix things myself
but I am something of a noob. I normally try to keep my actions to
cautious config changes and testing.
Why did you choose Centos? Was this installed by the company?
I have never done a kernel update.
Any easy way to do this?
It may not be necessary, at least to solve any SSD performance problems
anyway. Reexamining your numbers shows you hit 262MB/s to /dev/sda.
That's 65% of SATA2 interface bandwidth, so this kernel probably does
have the patch. Your problem lie elsewhere.
Regarding your second advice (to purchase a decent HBA) I have already
thought about it but I guess it comes with it's own drivers that need to
be compiled into initramfs etc.
The default CentOS (RHEL) initramfs should include mptsas, which
supports all the LSI HBAs. The LSI caching RAID cards are supported as
well with megaraid_sas.
The question is, do you really need more than the ~260MB/s of peak
throughput you currently have? And is it worth the hassle?
So I am trying to replace the baseboard
with one with SATA3 support to avoid any configuration changes (the old
board has the C202 chipset and the new one has C204 so I guess this
replacement is as simple as it gets - just remove the old board and plug
the new one without any software changes or recompiles). Again I need to
say this server is in production and I can't move the data or the users.
I can have a few hours downtime during the night but that's about all.
It's not clear your problem is hardware bandwidth. In fact it seems the
problem lie elsewhere. It may simply be that you're running these tests
while other substantial IO is occurring. Actually, your numbers show
this is exactly the case. What they don't show is how much other IO is
hitting the SSDs while you're running your tests.
Regarding the kernel upgrade, do we need to compile one from source or
there's an easier way?
I don't believe at this point you need a new kernel to fix the problem
you have. If this patch was not present you'd not be able to get
260MB/s from SATA2. Your problem lie elsewhere.
In the future, instead of making a post saying "md is slow, my SSDs are
slow" and pasting test data which appears to back that claim, you'd be
better served by describing a general problem, such as "users say the
system is slow and I think it may be md or SSD related". This way we
don't waste time following a troubleshooting path based on incorrect
assumptions, as we've done here. Or at least as I've done here, as I'm
the only one assisting.
Boot all users off the system, shut down any daemons that may generate
any meaningful load on the disks or CPUs. Disable any encryption or
compression. Then rerun your tests while completely idle. Then we'll
go from there.
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
