On Fri, 2005-08-26 at 15:52, Bryan J. Smith wrote:
> Evan Bursey <ehbursey@xxxxxxx> wrote:
> > I've just started caring for a dual Opteron machine. It
> > has a 3Ware 9500S-12 RAID controller, a Tyan S2882G3NR
> > motherboard, 10 IBM 400GB disk drives, and a 650 watt power
> > supply.
>
> Ouch, that's pushing it bad. I use a 550W with an old
> dual-P3 (sub-20W/each) and 3Ware Escalade 7800 with 8 x PATA
> at home.
>
> Most of my other installations that have 8-12 drives are
> typically 6U or pedestal with 850W or 1000W 2+1 2xSSI power
> supplies.
>
> > (actually, I did get some joy out of recompiling the
> kernel.
> > Its been several years since I last compiled a kernel and
> > the process has changed for the better).
>
> Actually, you shouldn't have to rebuild the kernel.
> You just:
> 1. Run make in the 3Ware source directory (the Makefile
> 3Ware includes does the job)
> 2. Copy the new 3w-9xxx.ko module in the appropriate
> /lib/modules/x.x.x-x directory
> 3. Run "depmod -a" to rebuild dependencies
> 4. Run "mkinitrd /boot/x.x.x-x-new.img x.x.x-x" to build a
> new initrd
> 5. Edit /etc/grub.conf to add a new entry with the new
> initrd
>
> > I've been in contact with 3Ware. From the error logs I
> > supplied to them, it looks like the issue is an
> insufficient
> > power supply.
>
> Would not surprise me one bit. Although the 3Ware can stage
> spin-ups at boot, if you have enough power usage, the drives
> can still stall due to lack of power in full seek.
>
> > Their rule of thumb is 300W for the motherboard
>
> You'll easily need 400+W with dual-Opterons. I'm sure your
> voltage is dropping on the +12V due to inadequate current.
>
> > and 35 W for each disk drive.
>
> Yep, sounds typical at 3A@12V ~ 36W.
>
> > That means that I would need (at a theoretical minimum) a
> > 650 W power supply - which I have.
>
> I'd say you're way over budget.
>
> > I only notice the problem in SMP mode, not in uniprocessor
> > mode.
>
> Correct because even though the CPU is plugged in, it's
> probably idle and not sucking up any juice.
>
> > That suggests to me that (1) I have a
> > boarderline-sufficient power supply that can support one
> > but not two CPUs
>
> I concur, it's definitely power. Unless you're staging your
> drive startup, I'm surprised you don't have an issue at boot.
>
> > or (2) there is a problem with the combination of
> > Opteron/3Ware 9500S-12 hardware and the x86_64 SMP
> > kernel. I suppose there are even worse possibilities,
> > but I'd rather not think about those right now.
> > What do you think?
>
> Definitely power. You are way over budget.
>
> You should either consider a 2+1 PS design (3 units, 2
> required to run), or use a separate power supply for the
> drives. What kind of enclosure do you have currently?
>
> > Does anyone have any experience with this sort of hardware
> > on x86_64 kernels?
>
> I don't think it's a kernel issue. I think the x86-64 kernel
> is making far greater use of your Opteron's capabilities. I
> mean, there are literally units that get "turned on" when the
> x86-64 kernel is running -- things the i686/Athlon kernel
> does not enable at all.
>
> > Does the 3w_9xxx driver work well on a x86_64 machine?
> > Does anyone have a suggestion as to what capacity power
> > supply I should have?
>
> Yes, 850W, easily. Power PC & Cooling has a non-redundant
> SSI one for just over $400. I know there are a few other
> vendors that have one for about $300.
>
> I'd be interested in what your enclosure is right now.
>
Hi Bryan, Thanks very much for your reply. The machine uses what looks
to me to be a large PC case. Its a SuperMicro 942i single power supply
tower. It has two SuperMicro 5 bay SATA hot swap modules.
Evan
