Hi James, 2010/5/5 James Smart <james.smart@xxxxxxxxxx>: > > > brem belguebli wrote: >> >> Hi james, >> >> We haven't yet been able to ask our Telco to switch back the DWDM >> links to original situation. >> >> However, since logging was activated on the server I'm having a lot of >> messages : >> >> lpfc 0000:10:00.1: 1:(0):0730 FCP command x26 failed: x2 SNS x70000500 >> x20000000 Data: xa x200 x10 x0 x0 >> >> for which I couldn't find no explanation >> (http://www-dl.emulex.com/support/linux/820482p/linux.pdf) >> >> Do you have any information on this ? > > This is saying that SCSI command opcode 0x26 (Vendor-specific opcode ??) > failed, with Status code x2 (Check Condition) followed by the SCSI sense > data, w/ Sense Key 5 (ILLEGAL REQUEST). > > I don't know who would be issuing this command (opcode 0x26), most likely > some utility/daemon using sgio, but the target is rejecting the command (not > valid for the vendor). Very reasonable. > I could finally find the 730 messages explanation in your docs, and we have tracked the faulty program. It is hpasm which is shipped with the Proliant Support Pack, that we invoque in the monitoring of the hardware RAID of the servers. Actually the same program runs on similar (OS, HBA's, etc...) machines without querying the opcode 0x26, and on 2 servers it does. Further investigation on this pointed out that on these 2 servers, we did install extra Emulex packages, elxocmlibhbaapi, elxocmlibhbaapi-32bit and elxocmcore that install various libraries ( /usr/lib/libemsdm.so, /usr/lib/libdfc.so,/usr/lib/libnl.so.1) that certainly contained symbols that are, thru the linux-gate.so, matched in these 3 libs, making the above program (hpasm) querying opcode 0x26 on all the storage controllers on the system. > >> Also, there are other lpfc parameters that could be tweaked if I >> understand well their meaning: >> >> lpfc_hba_queue_depth currently set to 1024 : Does it represent the >> number of [IOs/Exchanges] the HBA will queue untill the remote port >> acks them or untill it is considered down ? > > This is the total number of i/o's outstanding on the wire, to all > targets/luns, at any point in time. This is typically the capacity of the > adapter, which is used in a FIFO basis as I/O is received from the midlayer. > The default value of the attribute takes the maximum from the adapter. On > your adapter, the value is 1024. On most newer adapters, it is 2x this or > more. The only time I've seen this value tweaked is when our adapter is > connected to a single target (array), and overruns or fully utilizes the > capacity of the target, causing the target to work harder, and actually > accomplish less, than it could at say an 80% utilization level (note: > capacity level is target-specific). (another reason per-target queue_depth > handling was put in - see next comment). > > >> >> lpfc_max_scsicmpl_time set to 0 : Does 0 represent some infinite >> value, meaning it won't timeout any IO for which the driver did not >> receive any completion ack ? > > No, unrelated. This is relative to target queue depth mgmt. The midlayer > doesn't do queue depth management by target - only per sdev (lun). Our > driver does though. Target queue depth is the sum of all i/o to all luns on > the same target, with a threshold that may or may not be capped based on > the array type, and which scales/ramps down to the existing outstanding i/o > count when the target reports QUEUE_FULL/TASK_SET_FULL. This behavior is > valid only on targets that have a shared i/o queue for all luns. This value > controls the per-target ramp-up processing. If 0, we use a constant > compiled-in interval which ramps our target queue depth back up by x%. When > non-zero, it specifies a shost-specific time interval for the ramp up (it's > actually a little trickier than this as it's tailored on some arrays that > really depended upon not being overrun beyond their capacity levels). > Thanks for the explanation. However, we do not have anymore x26 opcode error messages, though I wasn't sure this was the root cause of the problem we had during the DWDM ring failover, I increased the logging (0xffff) on the HBA's of the nodes (total 4 nodes, 2 that were reporting the x26 opcode error say Group A, and the 2 that never did, say Group B). These 4 nodes form a cluster accessing the same LUNS thru the same controllers the very same way, and I get errors relative to INQUIRY on Group A: lpfc 0000:10:00.1: 1:(0):0730 FCP command x12 failed: x0 SNS x0 x0 Data: x8 x3c x0 x0 x0 lpfc 0000:10:00.1: 1:(0):0716 FCP Read Underrun, expected 96, residual 60 Data: x3c x12 x0 lpfc 0000:10:00.1: 1:0336 Rsp Ring 0 error: IOCB Data: xff000018 xe99fc48 x0 x0 x3c x0 x1d70c8e xa29b16 lpfc 0000:10:00.1: 1:0729 FCP cmd x12 failed <0/0> status: x1 result: x3c Data: x1d7 xc8e lpfc 0000:10:00.0: 0:(0):0730 FCP command x12 failed: x0 SNS x0 x0 Data: x8 x3c x0 x0 x0 lpfc 0000:10:00.0: 0:(0):0716 FCP Read Underrun, expected 96, residual 60 Data: x3c x12 x0 lpfc 0000:10:00.1: 1:0336 Rsp Ring 0 error: IOCB Data: xff000018 xe9960c0 x0 x0 x3c x0 x3360c67 xa29b16 lpfc 0000:10:00.1: 1:0729 FCP cmd x12 failed <0/0> status: x1 result: x3c Data: x336 xc67 On both HBA's and concerning the 13 paths seen thru target 0 (<0/0>, <0/1>...) Group B doesn't show no error. I'm going to get on one of Group B node a HBA's change to make sure it is not a hardware issue, and I'll keep you informed. > > -- james s > > Regards Brem -- To unsubscribe from this list: send the line "unsubscribe linux-scsi" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
