Re: Megaraid and Dell PERC 4 controllers

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Seokmann,
This sounds identical to a crash that I had on Saturday.
I have a server that has a dual Opteron/244 with 2GB of memory (4x512MB
400MHz, Registered ECC, Corsair CM72SD512RLP-3) on a Tyan Opteron 8131
motherboard.  The controller is the LSI MegaRAID SATA II 300-8X PCI-X
(P/N LSI00005 with the LSI00012 battery backup).  The system is fairly new,
it was manufactured on 06/22/05 and put in service about a mounth later.
The MegaRAID controller has 8 Seagate ST3250823AS 250GB SATA drives with 
NCQ.  
The RAID array is a RAID5 array with a global spare.  It is divided 
into two nearly equal sized logical disks.  The controller parameters 
are set to:
FlexRAID PowerFail = ENABLED
Command Que = Enabled

both logical drives are set to:
RAID = 5
Size = 712392MB 
StripeSize = 64KB 
{Write Policy = WRTHRU
Read Policy = NORMAL
Cache Policy = DirectIO
#Stripes = 7
State = OPTIMAL

The system is running Red Hat Enterprise Linux AS release 4 (Nahant Update 1)
With an updated kernel (I am booting off of a SATA disk on the 
Silicon Image, Inc. SiI 3114 controller which was only fixed in recent
kernels and firmware):
Kernel 2.6.11.12 on a 2-processor i686

The system is being used primarily as an NFS server. It also serves as
the head node for a small cluster.  It does the Ganglia data collection
task for the cluster.  Looking at the Ganglia data does not indicate
that there was much of a load on the system just before the crash.  
Although Ganglia is not recording disk I/O's I do not see much indirect 
evidence that there was heavy disk I/O: the CPUs are steady state--
around 97% idle, and no particular peaks or valleys.  Same with the 
number of packets and network bytes transmitted/received, and memory 
usage.  It all seems normal, with no particular peaks just before
I rebooted it (as with the original case--the system kept running,
although it was logging lots of disk I/O failed messages becuse the 
controller had been off-lined.

I am attaching a file that has the log records from the last 
reboot (we had moved it to a UPS just under 4 days before the 
controller locked up) showing the megaraid initialization,
and the sequence of error (condensed) messages from the controller 
up to the point where it off-lined the array(s).

Other than this incident the system has been running fine since it was
installed.  I hope that this helps.  If you have any suggestions 
please tell me as I am worried that this may happen again.

Thank you,
	steve.

On Mon, Aug 29, 2005 at 04:25:52PM -0400, Ju, Seokmann wrote:
> FYI - Resending due to failure on previous sending.  
> 
> > -----Original Message-----
> > From: Ju, Seokmann 
> > Sent: Friday, August 26, 2005 11:00 AM
> > To: 'Jonathan Fischer'
> > Cc: Kolli, Neela Syam
> > Subject: RE: Megaraid and Dell PERC 4 controllers
> > 
> > Hi Jonathan,
> > 
> > On Tuesday, August 23, 2005 4:52 PM, Jonathan Fischer wrote:
> > > I think next up I'm trying writethru mode, instead of write 
> > back, but
> > > has anyone seen anything like this, or have any insight they might
> > > offer?  I'm quickly getting to the point of being stumped.
> > Can you please specify detail system configuration? (memory 
> > size, # of cpus)
> > And, what kind of load are you putting on the system when it locks up.
> > Also, I assuem that the system doesn't have any monitoring 
> > applications running for those PERC controllers. Please confirm this.
> > From the message, the controller takes more than 3 minutes to 
> > return certain I/O requests and it leads system to lock up.
> > 
> > Thank you.
> > 
> > Seokmann
> > 
> > > -----Original Message-----
> > > From: Jonathan Fischer [mailto:jfischer@xxxxxxxxx] 
> > > Sent: Tuesday, August 23, 2005 4:52 PM
> > > To: linux-scsi@xxxxxxxxxxxxxxx
> > > Subject: Megaraid and Dell PERC 4 controllers
> > > 
> > > I apologize if this is the wrong list to ask this kind of 
> > question on;
> > > I've posted on Dell's PowerEdge list and Red Hat's lists as 
> > > well, but I
> > > figure the people here might know better what to try for 
> > this problem.
> > > 
> > > I have 2 Dell PowerEdge 2850's, one with a PERC 4e/DC raid 
> > controller,
> > > and the other with a PERC 4e/Di.  On both of these systems, I can
> > > reliably cause the controllers to lock up under heavy load.  This is
> > > using a fully up-to-date Red Hat 4 EL (non x86_64) 
> > > installation on both
> > > computers.  The controllers use the megaraid_mbox driver.
> > > 
> > > During a period of high load, the controller suddenly seems to stop
> > > responding to the driver, causing the driver to go into a 
> > waiting loop
> > > for it.  It waits 3 minutes for the controller to respond, which it
> > > never does, and then takes the controller offline, pretty 
> > much yanking
> > > the filesystem out from underneath the OS.
> > > 
> > > Some things keep running alright, so (working with Red Hat's 
> > > support) I
> > > got the thing set up to netdump to another server to see if we could
> > > figure out what was going wrong.  The kernel never actually 
> > > crashes, so
> > > netdump doesn't produce a vmcore to look through, but syslog keeps
> > > spouting out information, so I've got that.
> > > 
> > > Every time this lockup occurs, the log file looks like this:
> > > 
> > > megaraid: aborting-29762 cmd=2a <c=2 t=0 l=0>
> > > megaraid abort: 29762:21[255:128], fw owner
> > > megaraid: aborting-29763 cmd=2a <c=2 t=0 l=0>
> > > megaraid abort: 29763:39[255:128], fw owner
> > > megaraid: aborting-29764 cmd=2a <c=2 t=0 l=0>
> > > megaraid abort: 29764:16[255:128], fw owner
> > > megaraid: aborting-29768 cmd=2a <c=2 t=0 l=0>
> > > megaraid abort: 29768:53[255:128], fw owner
> > > 
> > > 	This part repeats 64 times, then...
> > > 
> > > megaraid: aborting-29831 cmd=2a <c=2 t=0 l=0>
> > > megaraid abort: 29831:8[255:128], fw owner
> > > megaraid: resetting the host...
> > > megaraid: 64 outstanding commands. Max wait 180 sec
> > > megaraid mbox: Wait for 64 commands to complete:180
> > > megaraid mbox: Wait for 64 commands to complete:175
> > > 	
> > > 	megaraid mbox counts down to 0, and then...
> > > 
> > > megaraid mbox: critical hardware error!
> > > megaraid: resetting the host...
> > > megaraid: hw error, cannot reset
> > > megaraid: resetting the host...
> > > megaraid: hw error, cannot reset
> > > SCSI error : <0 2 0 0> return code = 0x6000000
> > > end_request: I/O error, dev sda, sector 242938701
> > > Buffer I/O error on device dm-4, logical block 9893952 lost 
> > page write
> > > due to I/O error on dm-4
> > > scsi0 (0:0): rejecting I/O to offline device
> > > 
> > > The commands that the driver are waiting for are always the 
> > > same, except
> > > for the sequence number (the number right after "aborting-" 
> > > and  "abort:
> > > ").  And there are always 64 commands backed up that the driver is
> > > waiting for.
> > > 
> > > Both machines in question pass memtest86 and Dell's 
> > > diagnostic sets, and
> > > since the failure is identical in both I don't believe it's bad
> > > hardware.  We've got the latest BIOS, RAID firmware, and backplane
> > > firmware on the machines.
> > > 
> > > I've also tried:
> > > - the RHEL 4 Update 2 Beta kernel (at Red Hat's suggestion)
> > > - RHEL 4 x86_64
> > > - RHEL 3 x86_64
> > > - Fedora Core 4 x86
> > > - disabling Patrol Read in the RAID bios
> > > - disabling read-ahead in the RAID bios
> > > - changing the writeback cache flush to every 2 seconds, 
> > > instead of the
> > > default 4
> > > 
> > > I think next up I'm trying writethru mode, instead of write 
> > back, but
> > > has anyone seen anything like this, or have any insight they might
> > > offer?  I'm quickly getting to the point of being stumped.
> > > 
> > > Jonathan Fischer
> > > Operating Systems Analyst - CSU San Marcos
> > > jfischer@xxxxxxxxx
> > > 
> > > -
> > > : send the line "unsubscribe 
> > > linux-scsi" in
> > > the body of a message to majordomo@xxxxxxxxxxxxxxx
> > > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > > 
> > 
> -
> : send the line "unsubscribe linux-scsi" in
> the body of a message to majordomo@xxxxxxxxxxxxxxx
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
Red Hat Enterprise Linux AS release 4 (Nahant Update 1)
Kernel 2.6.11.12 on a 2-processor i686

Aug 23 19:49:03 brule kernel: megaraid cmm: 2.20.2.5 (Release Date: Fri Jan 21 00:01:03 EST 2005)
Aug 23 19:49:03 brule kernel: megaraid: 2.20.4.5 (Release Date: Thu Feb 03 12:27:22 EST 2005)
Aug 23 19:49:03 brule kernel: megaraid: probe new device 0x1000:0x0409:0x1000:0x3008: bus 2:slot 14:func 0
Aug 23 19:49:03 brule kernel: ACPI: PCI interrupt 0000:02:0e.0[C] -> GSI 28 (level, low) -> IRQ 28
Aug 23 19:49:03 brule kernel: megaraid: fw version:[813i] bios version:[H430]
Aug 23 19:49:03 brule kernel: scsi0 : LSI Logic MegaRAID driver
Aug 23 19:49:03 brule kernel: scsi[0]: scanning scsi channel 0 [Phy 0] for non-raid devices
Aug 23 19:49:03 brule kernel: scsi[0]: scanning scsi channel 1 [virtual] for logical drives
Aug 23 19:49:03 brule kernel:   Vendor: MegaRAID  Model: LD 0 RAID5  712G  Rev: 813i
Aug 23 19:49:03 brule kernel:   Type:   Direct-Access                      ANSI SCSI revision: 02
Aug 23 19:49:03 brule kernel:   Vendor: MegaRAID  Model: LD 1 RAID5  712G  Rev: 813i
Aug 23 19:49:03 brule kernel:   Type:   Direct-Access                      ANSI SCSI revision: 02
Aug 23 19:49:03 brule kernel: ACPI: PCI interrupt 0000:04:05.0[A] -> GSI 19 (level, low) -> IRQ 19
Aug 23 19:49:03 brule kernel: ata1: SATA max UDMA/100 cmd 0xF8806C80 ctl 0xF8806C8A bmdma 0xF8806C00 irq 19
Aug 23 19:49:03 brule kernel: ata2: SATA max UDMA/100 cmd 0xF8806CC0 ctl 0xF8806CCA bmdma 0xF8806C08 irq 19
Aug 23 19:49:03 brule kernel: ata3: SATA max UDMA/100 cmd 0xF8806E80 ctl 0xF8806E8A bmdma 0xF8806E00 irq 19
Aug 23 19:49:03 brule kernel: ata4: SATA max UDMA/100 cmd 0xF8806EC0 ctl 0xF8806ECA bmdma 0xF8806E08 irq 19
Aug 23 19:49:03 brule kernel: ata1: dev 0 ATA, max UDMA/133, 234441648 sectors: lba48
Aug 23 19:49:03 brule kernel: ata1: dev 0 configured for UDMA/100
Aug 23 19:49:03 brule kernel: scsi1 : sata_sil
Aug 23 19:49:03 brule kernel: ata2: no device found (phy stat 00000000)
Aug 23 19:49:03 brule kernel: scsi2 : sata_sil
Aug 23 19:49:03 brule kernel: ata3: no device found (phy stat 00000000)
Aug 23 19:49:03 brule kernel: scsi3 : sata_sil
Aug 23 19:49:03 brule kernel: ata4: no device found (phy stat 00000000)
Aug 23 19:49:03 brule kernel: scsi4 : sata_sil
Aug 23 19:49:03 brule kernel:   Vendor: ATA       Model: ST3120026AS       Rev: 3.05
Aug 23 19:49:03 brule kernel:   Type:   Direct-Access                      ANSI SCSI revision: 05
Aug 23 19:49:03 brule kernel: SCSI device sda: 1458978816 512-byte hdwr sectors (746997 MB)
Aug 23 19:49:03 brule kernel: sda: asking for cache data failed
Aug 23 19:49:03 brule kernel: sda: assuming drive cache: write through
Aug 23 19:49:04 brule kernel: SCSI device sda: 1458978816 512-byte hdwr sectors (746997 MB)
Aug 23 19:49:04 brule kernel: sda: asking for cache data failed
Aug 23 19:49:04 brule kernel: sda: assuming drive cache: write through
Aug 23 19:49:04 brule kernel:  sda: sda1 sda2 sda3 sda4 < sda5 sda6 sda7 sda8 sda9 sda10 sda11 sda12 sda13 sda14 >
Aug 23 19:49:04 brule kernel: Attached scsi disk sda at scsi0, channel 1, id 0, lun 0
Aug 23 19:49:04 brule kernel: SCSI device sdb: 1458978816 512-byte hdwr sectors (746997 MB)
Aug 23 19:49:04 brule kernel: sdb: asking for cache data failed
Aug 23 19:49:04 brule kernel: sdb: assuming drive cache: write through
Aug 23 19:49:04 brule kernel: SCSI device sdb: 1458978816 512-byte hdwr sectors (746997 MB)
Aug 23 19:49:04 brule kernel: sdb: asking for cache data failed
Aug 23 19:49:04 brule kernel: sdb: assuming drive cache: write through
Aug 23 19:49:04 brule kernel:  sdb: sdb1 sdb2 sdb3 sdb4
Aug 23 19:49:04 brule kernel: Attached scsi disk sdb at scsi0, channel 1, id 1, lun 0
Aug 23 19:49:04 brule kernel: SCSI device sdc: 234441648 512-byte hdwr sectors (120034 MB)
Aug 23 19:49:04 brule kernel: SCSI device sdc: drive cache: write back
Aug 23 19:49:04 brule kernel: SCSI device sdc: 234441648 512-byte hdwr sectors (120034 MB)
Aug 23 19:49:04 brule kernel: SCSI device sdc: drive cache: write back
Aug 23 19:49:04 brule kernel:  sdc: sdc1 sdc2 sdc3 < sdc5 sdc6 sdc7 sdc8 > sdc4
Aug 23 19:49:04 brule kernel: Attached scsi disk sdc at scsi1, channel 0, id 0, lun 0
Aug 23 19:49:04 brule kernel: Attached scsi generic sg0 at scsi0, channel 1, id 0, lun 0,  type 0
Aug 23 19:49:04 brule kernel: Attached scsi generic sg1 at scsi0, channel 1, id 1, lun 0,  type 0
Aug 23 19:49:04 brule kernel: Attached scsi generic sg2 at scsi1, channel 0, id 0, lun 0,  type 0
... the disk ran fine for nearly 4 days

Aug 27 16:19:56 brule kernel: megaraid: aborting-35347365 cmd=2a <c=1 t=0 l=0>
Aug 27 16:19:56 brule kernel: megaraid abort: 35347365:95[255:128], fw owner
Aug 27 16:19:56 brule kernel: megaraid: aborting-35347366 cmd=2a <c=1 t=0 l=0>
Aug 27 16:19:56 brule kernel: megaraid abort: 35347366:121[255:128], fw owner
Aug 27 16:19:56 brule kernel: megaraid: aborting-35347367 cmd=2a <c=1 t=0 l=0>
...
Aug 27 16:19:57 brule kernel: megaraid: aborting-35347510 cmd=2a <c=1 t=0 l=0>
Aug 27 16:19:57 brule kernel: megaraid abort: 35347510:112[255:128], fw owner
Aug 27 16:19:57 brule kernel: megaraid: reseting the host...
Aug 27 16:19:57 brule kernel: megaraid: 64 outstanding commands. Max wait 180 sec
Aug 27 16:19:57 brule kernel: megaraid mbox: Wait for 64 commands to complete:180
Aug 27 16:20:01 brule kernel: megaraid mbox: Wait for 64 commands to complete:175
Aug 27 16:20:06 brule kernel: megaraid mbox: Wait for 1 commands to complete:170
Aug 27 16:20:11 brule kernel: megaraid mbox: Wait for 1 commands to complete:165
Aug 27 16:20:16 brule kernel: megaraid mbox: Wait for 1 commands to complete:160
...
Aug 27 16:22:51 brule kernel: megaraid mbox: Wait for 1 commands to complete:5
Aug 27 16:22:56 brule kernel: megaraid mbox: Wait for 1 commands to complete:0
Aug 27 16:23:01 brule kernel: megaraid mbox: Wait for 1 commands to complete:-5
...
Aug 27 16:24:46 brule kernel: megaraid mbox: Wait for 1 commands to complete:-110
Aug 27 16:24:51 brule kernel: megaraid mbox: Wait for 1 commands to complete:-115
Aug 27 16:24:56 brule kernel: megaraid mbox: critical hardware error!
Aug 27 16:24:56 brule kernel: megaraid: reseting the host...
Aug 27 16:24:56 brule kernel: megaraid: hw error, cannot reset
Aug 27 16:24:56 brule kernel: megaraid: reseting the host...
Aug 27 16:24:56 brule kernel: megaraid: hw error, cannot reset
Aug 27 16:24:56 brule kernel: scsi: Device offlined - not ready after error recovery: host 0 channel 1 id 0 lun 0

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[Index of Archives]     [SCSI Target Devel]     [Linux SCSI Target Infrastructure]     [Kernel Newbies]     [IDE]     [Security]     [Git]     [Netfilter]     [Bugtraq]     [Yosemite News]     [MIPS Linux]     [ARM Linux]     [Linux Security]     [Linux RAID]     [Linux ATA RAID]     [Linux IIO]     [Samba]     [Device Mapper]
  Powered by Linux