On 12/03/2012 02:52 PM, Ric Wheeler wrote: > I jumped into this thread late - can you repost detail on the specific > drive and HBA used here? In any case, it sounds like this is a better > topic for the linux-scsi or linux-ide list where most of the low level > storage people lurk :) Okay, expanding the receiver list. :) To recap: I'm running 2.6.27 with LVM over software RAID 1 over a pair of SAS disks. Disks are WD9001BKHG, controller is Intel C600. Recently we started seeing messages of the following pattern, and we don't know what's causing them: Nov 28 08:57:10 kernel: end_request: I/O error, dev sda, sector 1758169523 Nov 28 08:57:10 kernel: md: super_written gets error=-5, uptodate=0 Nov 28 08:57:10 kernel: raid1: Disk failure on sda2, disabling device. Nov 28 08:57:10 kernel: raid1: Operation continuing on 1 devices. We've been assuming it's a software issue since it's reproducible on multiple systems, although so far we've only seen the problem with these particular disks. We've seen the problems with disk write cache enabled and disabled. It looks like it may be related to being in the middle of a background short self-test at the time we see the error. The disks are still in-service at this point--is this supported behaviour or would it be expected to cause errors? (The self-test works fine with other disks, and worked fine with these disks until recently, but we haven't made any changes to the block I/O code.) Here's the smartctl output from right after a failure. The self-tests are frequent as a stress-test, normally they're done once per day: root@typhoon-base-unit0:/root> ./smartctl --all /dev/sda smartctl version 5.38 [i686-wrs-linux-gnu] Copyright (C) 2002-8 Bruce Allen Home page is http://smartmontools.sourceforge.net/ Device: WD WD9001BKHG-02D22 Version: SR03 Serial number: WX21EB1ANU78 Device type: disk Transport protocol: SAS Local Time is: Fri Nov 23 00:35:03 2012 HKT Device supports SMART and is Enabled Temperature Warning Enabled SMART Health Status: OK Current Drive Temperature: 39 C Drive Trip Temperature: 69 C Manufactured in week 01 of year 2010 Recommended maximum start stop count: 1048576 times Current start stop count: 26 times Elements in grown defect list: 0 Error counter log: Errors Corrected by Total Correction Gigabytes Total ECC rereads/ errors algorithm processed uncorrected fast | delayed rewrites corrected invocations [109 bytes] errors read: 21187 2 2 21189 2 4950.446 0 write: 89 4 0 93 4 1317.938 0 verify: 103 0 0 103 0 0.000 0 Non-medium error count: 169436 SMART Self-test log Num Test Status segment LifeTime LBA_first_err [SK ASC ASQ] Description number (hours) # 1 Background short Self test in progress ... 4 NOW - [- - -] # 2 Background short Completed - 1377 - [- - -] # 3 Background short Completed - 1377 - [- - -] # 4 Background short Completed - 1377 - [- - -] # 5 Background short Completed - 1377 - [- - -] # 6 Background short Completed - 1377 - [- - -] # 7 Background short Completed - 1377 - [- - -] # 8 Background short Completed - 1377 - [- - -] # 9 Background short Completed - 1377 - [- - -] #10 Background short Completed - 1377 - [- - -] #11 Background short Completed - 1377 - [- - -] #12 Background short Completed - 1377 - [- - -] #13 Background short Completed - 1377 - [- - -] #14 Background short Completed - 1377 - [- - -] #15 Background short Completed - 1377 - [- - -] #16 Background short Completed - 1377 - [- - -] #17 Background short Completed - 1377 - [- - -] #18 Background short Completed - 1377 - [- - -] #19 Background short Completed - 1377 - [- - -] #20 Background short Completed - 1377 - [- - -] Long (extended) Self Test duration: 6362 seconds [106.0 minutes] I also have this from ten minutes later with a newer version of smartctl: root@typhoon-base-unit0:/root> ./smartctl.eric -x /dev/sda smartctl 5.40 2010-10-16 r3189 [i686-pc-linux-gnu] (local build) Copyright (C) 2002-10 by Bruce Allen, http://smartmontools.sourceforge.net Device: WD WD9001BKHG-02D22 Version: SR03 Serial number: WX21EB1ANU78 Device type: disk Transport protocol: SAS Local Time is: Fri Nov 23 00:45:08 2012 HKT Device supports SMART and is Enabled Temperature Warning Enabled SMART Health Status: OK Current Drive Temperature: 39 C Drive Trip Temperature: 69 C Manufactured in week 01 of year 2010 Specified cycle count over device lifetime: 1048576 Accumulated start-stop cycles: 26 Specified load-unload count over device lifetime: 1114112 Accumulated load-unload cycles: 0 Elements in grown defect list: 0 Error counter log: Errors Corrected by Total Correction Gigabytes Total ECC rereads/ errors algorithm processed uncorrected fast | delayed rewrites corrected invocations [109 bytes] errors read: 21189 2 2 21191 2 4950.446 0 write: 89 4 0 93 4 1317.939 0 verify: 103 0 0 103 0 0.000 0 Non-medium error count: 169436 SMART Self-test log Num Test Status segment LifeTime LBA_first_err [SK ASC ASQ] Description number (hours) # 1 Background short Completed - 1378 - [- - -] # 2 Background short Completed - 1378 - [- - -] # 3 Background short Completed - 1378 - [- - -] # 4 Background short Completed - 1377 - [- - -] # 5 Background short Completed - 1377 - [- - -] # 6 Background short Completed - 1377 - [- - -] # 7 Background short Completed - 1377 - [- - -] # 8 Background short Completed - 1377 - [- - -] # 9 Background short Completed - 1377 - [- - -] #10 Background short Completed - 1377 - [- - -] #11 Background short Completed - 1377 - [- - -] #12 Background short Completed - 1377 - [- - -] #13 Background short Completed - 1377 - [- - -] #14 Background short Completed - 1377 - [- - -] #15 Background short Completed - 1377 - [- - -] #16 Background short Completed - 1377 - [- - -] #17 Background short Completed - 1377 - [- - -] #18 Background short Completed - 1377 - [- - -] #19 Background short Completed - 1377 - [- - -] #20 Background short Completed - 1377 - [- - -] Long (extended) Self Test duration: 6362 seconds [106.0 minutes] Background scan results log Status: no scans active Accumulated power on time, hours:minutes 1378:08 [82688 minutes] Number of background scans performed: 0, scan progress: 0.00% Number of background medium scans performed: 0 Protocol Specific port log page for SAS SSP relative target port id = 1 generation code = 0 number of phys = 1 phy identifier = 0 attached device type: end device attached reason: unknown reason: unknown negotiated logical link rate: phy enabled; 3 Gbps attached initiator port: ssp=1 stp=1 smp=1 attached target port: ssp=0 stp=0 smp=0 SAS address = 0x50014ee3556977a6 attached SAS address = 0x5fcfffff00000001 attached phy identifier = 0 Invalid DWORD count = 0 Running disparity error count = 0 Loss of DWORD synchronization = 3 Phy reset problem = 0 Phy event descriptors: Transmitted SSP frame error count: 0 Received SSP frame error count: 0 relative target port id = 2 generation code = 0 number of phys = 1 phy identifier = 1 attached device type: no device attached attached reason: unknown reason: unknown negotiated logical link rate: phy enabled; unknown attached initiator port: ssp=0 stp=0 smp=0 attached target port: ssp=0 stp=0 smp=0 SAS address = 0x50014ee3556977a7 attached SAS address = 0x0 attached phy identifier = 0 Invalid DWORD count = 0 Running disparity error count = 0 Loss of DWORD synchronization = 0 Phy reset problem = 0 Phy event descriptors: Transmitted SSP frame error count: 0 Received SSP frame error count: 0 -- To unsubscribe from this list: send the line "unsubscribe linux-ide" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
