Tyan Thunder K8SE S2892 Report

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]



I had my eye on the Tyan dual-Opteron mobos for awhile. I tried to find
a posting *anywhere* sharing experiences with these boards under Linux.
No such luck. So placing myself under the heading "Where Angles Fear to
Tread," I went ahead and built a system anyway. Here's what I've learned.

The specs:
Tyan Thunder K8SE S2892, BIOS 1.01
2x Opteron 270, 2Ghz Dual-Core, retail package with fan
2GB RAM (4x SuperTalent 512MB P/N D32RB12P3)
3Ware 9500S-4LP 4 port SATA controller
4x 200GB Segate ST3200826AS 7200.8 SATA (attached to 3Ware controller)
1x 80GB Wester Digital WD800JD-00LSA0 SATA (attached to onboard controller)
Antec Truepower 2.0 550W PSU
Chenming 901A-0-0 RTL Case
Supermicro 5-drive SATA HD Cage
CentOS 4.1 x86_64

I thought I bought a plenty large case. Turns out the CPU heat sink at
the front of the motherboard sticks up far enough to interfere with the
drive bays at the bottom of the case. I had carefully mounted the CPUs,
heat sinks and memory before mounting the motherboard. When I tried to
insert the motherboard, the front heat sink hit the shelf that supports
one of the two internal hard drive cages that come with the Chenming
case. I had to peel off the heat sink to get the board in then reinstall
it -- something I didn't want to do. If the case was 9" wide instead of
8", this probably wouldn't be an issue.

The heat sink sticks up far enough to keep me from reinstalling the
internal hard drive cages. Not a problem since I'm using the Supermicro
SATA cage, but it bugs me to lose expansion possibilities. Can anyone
recommend a "low-rise" Opteron heat sink?

I hadn't yet purchased an optical drive and so was planning on
booting the install program from a USB key. However I could not get the
BIOS to recognize the key. I put 'removable drives' at the top of the
boot preferences but it refused to recognize the key. I ended up doing a
PXE boot install instead.

The install hung once forcing me to restart. But after that the install
process was the fastest I've ever seen, taking about seven minutes to
copy everything.

Booting the first time, I was happy to see that everything was
recognized by the system, all three network ports, the Nvidia SATA
controller and even the 3Ware controller.

However, the boot messages showed:

	warning: many lost ticks.
	Your time source seems to be instable or some driver is hogging 	
interrupts

and several repeating messages:

	powernow-k8: error - out of sync...

Also, /proc/cpuinfo showed each cpu as running at about 1004Mhz with
about 900 bogomips. Not what I expected.

I decided to press ahead and added a couple of drives the the 3Ware,
created a unit, formatted it and began benchmarking the array. The
"powernow" messages continued to occur and then the 3Ware driver started
issuing error messages. Eventually the system crashed and stopped
responding.

Googling on the powernow error message let me know that the kernel
wizards had started to fix some powernow bugs starting with about kernel
2.6.10. Of course the stock CentOS 4.1 kernel is 2.6.9-11.

So I downloaded, compiled and installed kernel 2.6.12.3. On rebooting,
the error messages went away and the 3Ware worked without complaint. Now
/proc/cpuinfo showed each cpu running at 2009.267 Mhz with 4014.08
bogomips. Ah, much better! In addition, the system came up in with NUMA
enabled. Looks like the Red Hat kernel had it turned off by default.

I benchmarked disk array performance using Bonnie++ version 1.03a. I ran
six benchmarks using 50GB of data, three using 16k blocks and three
using 64k blocks. During the raid 0 testing I had two instances of
Setiathome running. During the raid 5 testing I had three instances of
Setiathome running. Here are the results:

Raid 0, 64k Stripes:

bonnie++ 1.03a      ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec  %CP K/sec %CP K/sec %CP K/sec  %CP  /sec %CP
Tyan        50G:16k 49317  99 138307  62 +++++ +++ 46765  80 183180  22  88.9   0
Tyan        50G:16k 48911  98 148421  67 77099  21 46018  79 182136  21  89.8   0
Tyan        50G:16k 49188  98 143615  65 77381  21 46158  78 181181  21  90.5   0
Tyan        50G:64k 49372  99 146417  67 77185  21 45828  78 179758  21  76.7   0
Tyan        50G:64k 48585  98 145376  66 76580  21 45609  78 171888  21  76.3   0
Tyan        50G:64k 46093  92 134903  58 67851  18 45200  77 172103  21  75.6   0
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files:max:min        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
Tyan             16  2524  97 +++++ +++ +++++ +++  2542  97 +++++ +++  8670  99
Tyan             16  2385  98 +++++ +++ +++++ +++  2579  97 +++++ +++  8408  98
Tyan             16  2627  97 +++++ +++ +++++ +++  2437  92 +++++ +++  8671 100
Tyan             16  1582  98 +++++ +++ +++++ +++  1647  98 +++++ +++  8488 100
Tyan             16  1399  85 +++++ +++ +++++ +++  1632  98 +++++ +++  8476  99
Tyan             16  1654  95 +++++ +++ +++++ +++  1636  98 +++++ +++  8405  99

Writes at about 140MB/Sec, reads at about 180MB/Sec. Very nice.

Raid 5, 64k Stripes:

bonnie++ 1.03a      ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
Tyan        50G:16k 33394  68 35889  16 26513   7 44096  75 157248  19  84.1   0
Tyan        50G:16k 32601  67 36058  16 26623   7 43964  75 156970  19  83.3   0
Tyan        50G:16k 37892  77 35702  16 26960   7 44390  75 157084  19  83.5   0
Tyan        50G:64k 36168  74 35560  17 27003   8 43463  75 155179  20  69.1   0
Tyan        50G:64k 32713  68 36187  16 26580   7 43922  76 156752  20  69.2   0
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files:max:min        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
Tyan             16  1561  98 +++++ +++ +++++ +++  1694  97 +++++ +++  8428  97
Tyan             16  1672  98 +++++ +++ +++++ +++  1712  97 +++++ +++  8751  99
Tyan             16  1542  98 +++++ +++ +++++ +++  1355  81 +++++ +++  8986 100
Tyan             16  2534  96 +++++ +++ +++++ +++  2408  96 +++++ +++  7945  98
Tyan             16  2382  97 +++++ +++ +++++ +++  2380  97 +++++ +++  7989  99

Whoa! Writes drop to about 35MB/Sec under raid 5, about 25% of the raid 0
performance. In fact, it was taking so long that I aborted the last test. Just to be
sure that setiathome wasn't interfering, I killed seti and reran a smaller test.
Writes increased to about 50MB/Sec without seti running. I was expecting some hit for
raid 5 but not this much. Guess I'll stick with the hot rsync backups to other hosts
scheme that I'm using now.

As a final test, I restarted the 50GB benchmark under raid5 and popped out one of the
drives. There was a pause while I guess the 3Ware controller decided that the drive
actually went off-line. The writing then continued to the remaining three disks.
Checking the controller showed that both the drive and the unit were 'DEGRADED.'

After popping the disk back in, I had to remove and re-add the drive to the array
configuration then start the rebuild process. At this point the drive showed
'DEGRADED' and the unit showed 'REBUILDING.' The 3Ware drive logged appropriate
messages when the disk was removed and replaced. I didn't bother finishing the
rebuild, but it looked like it would take a couple of hours to finish if I let the
benchmark continue to run.

The 3Ware controller is pretty cool. As I said, the driver (3w-9xxx) is
included in the 2.6 kernel. 3Ware provides a simple CLI utility for
management. They also have a GUI tool but I didn't bother running it.
You can create, remove and verify 'units' on a running system. The
associated /dev entries are dynamically added and removed as you make
changes. Very nice.

In summary, the Tyan S2892 plus the 3Ware controller runs well under CentOS 4.1
although you will have to update to a more current kernel than the one provided by
Red Hat. Watch your clearance at the front of the motherboard when selecting a case.

Comments and corrections are encouraged.

Kirk Bocek













[Index of Archives]     [CentOS]     [CentOS Announce]     [CentOS Development]     [CentOS ARM Devel]     [CentOS Docs]     [CentOS Virtualization]     [Carrier Grade Linux]     [Linux Media]     [Asterisk]     [DCCP]     [Netdev]     [Xorg]     [Linux USB]
  Powered by Linux