Hi All,
Back in February/March of this year, I started a very long thread
(http://marc.info/?l=linux-raid&m=136021974127295&w=2). During that
thread, I got lots of very helpful advice, and made some significant
architecture changes to the systems, as well as purchasing lots of
additional hardware.
Unfortunately, I don't have a complete summary of every change that has
happened, and I still don't have a perfectly working system, however, I
thought I might provide some information in case it is helpful (or
interesting) to anyone.
My current hardware:
2 x Storage servers each with
2 x Quad port 1Gbps ethernet cards (Intel)
2 x Onboard 1Gbps ethernet cards
5 x 480GB SSD (Intel 520s)
The storage servers have one ethernet crossover to each other, one to
the "user LAN" and the other 8 in a Linux bond using balance-alb and mtu
9000
The 5 x 480GB SSD is configured as a single RAID5 array:
md1 : active raid5 sdb1[7] sdd1[9] sde1[5] sdc1[8] sda1[6]
1863535104 blocks super 1.2 level 5, 64k chunk, algorithm 2 [5/5]
[UUUUU]
bitmap: 4/4 pages [16KB], 65536KB chunk
That is used by DRBD to join the two servers together, using the
crossover cable to keep the second server in-sync
I then use LVM2 on top to split it up into "drives" for each server
Finally, iSCSI (ietd) is used to server these LV's to the network (over
the 8 x 1Gbps ethernet).
Raw throughput from fio at the time was seeing performance of approx
2.5GB/s read and 1.6GB/s write performance, though at the time the
secondary server was disconnected from DRBD and using 4 x 2TB drives in
RAID10.
I then have 8 x Xen hosts, each with a dual port 1Gbps ethernet card
(Intel) plus an onboard ethernet.
These servers have the onboard ethernet connected to a switch for the
"User LAN", and the other two are configured with individual IP's, both
on the same subnet, and also on the same subnet as the bond for the
storage servers.
So, this has been working .... for some definition of that, for a number
of months, but I shifted my attention away from the RAID layer and back
to the network layer when I finally noticed periods of complete packet
loss to the various windows VM's running on the Xen machines. Finally,
at the beginning of this month, I (not really me, but the auther of the
GPLPV drivers for windows) found the necesary bug and released a fix.
Since then, I've had 0 packet loss (on the local LAN) to the various VM's.
I want to repeat that, for anyone still running Windows XP/2003 in Xen,
please make sure you get the latest GPLPV to solve networking issues
(they only happen under certain types of load, which my bunch of users
seem to be good at generating).
In addition, (after a one hour outage on Sunday afternoon) found a bug
in the switch firmware which causes it to reboot randomly (well, once in
8 months isn't too bad, but I expect 100% perfection :). So tonight I'll
be upgrading that.
However, I've still got a performance issue, and it still looks like a
disk performance issue, but I'll come back to that hopefully later
tonight after I do the firmware upgrade on the switch, then install the
absolute latest GPLPV drivers, and then start performance testing all
over again... I'll cross my fingers and hope I don't break everything
too badly. If I'm lucky, I'll be able to pinpoint the cause now that so
many of my other problems have been fixed.
Regards,
Adam
--
Adam Goryachev Website Managers www.websitemanagers.com.au
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
