On initiator side, there were also a lot of messages: May 21 11:01:52 poc1 kernel: [ 3374.393864] host7: fcp: 00061e: Returning DID_ERROR to scsi-ml due to FC_DATA_UNDRUN (scsi) May 21 11:01:52 poc1 kernel: [ 3374.396149] host7: xid fc1: Exchange timer canceled May 21 11:01:52 poc1 kernel: [ 3374.396155] host7: fcp: 00061e: Returning DID_ERROR to scsi-ml due to FC_DATA_UNDRUN (scsi) May 21 11:01:52 poc1 kernel: [ 3374.397069] host7: xid 602: Exchange timer canceled May 21 11:01:52 poc1 kernel: [ 3374.397075] host7: fcp: 00061e: Returning DID_ERROR to scsi-ml due to FC_DATA_UNDRUN (scsi) May 21 11:01:52 poc1 kernel: [ 3374.397482] host7: xid a46: Exchange timer armed : 8000 msecs May 21 11:01:52 poc1 kernel: [ 3374.398443] host7: xid 602: Exchange timer armed : 8000 msecs May 21 11:01:52 poc1 kernel: [ 3374.398498] host7: xid 6ce: Exchange timer armed : 8000 msecs May 21 11:01:52 poc1 kernel: [ 3374.398863] host7: xid 6ce: Exchange timer canceled May 21 11:01:52 poc1 kernel: [ 3374.398869] host7: fcp: 00061e: Returning DID_ERROR to scsi-ml due to FC_DATA_UNDRUN (scsi) May 21 11:01:52 poc1 kernel: [ 3374.399449] host7: xid 6a2: Exchange timer armed : 8000 msecs May 21 11:01:52 poc1 kernel: [ 3374.399476] host7: xid 3e1: Exchange timer armed : 8000 msecs May 21 11:01:52 poc1 kernel: [ 3374.399486] host7: xid dcd: Exchange timer armed : 8000 msecs May 21 11:01:52 poc1 kernel: [ 3374.399866] host7: xid dcd: Exchange timer canceled May 21 11:01:52 poc1 kernel: [ 3374.399872] host7: fcp: 00061e: Returning DID_ERROR to scsi-ml due to FC_DATA_UNDRUN (scsi) May 21 11:01:52 poc1 kernel: [ 3374.400472] host7: xid d6c: Exchange timer armed : 8000 msecs May 21 11:01:52 poc1 kernel: [ 3374.401442] host7: xid 585: Exchange timer canceled May 21 11:01:52 poc1 kernel: [ 3374.401453] host7: fcp: 00061e: Returning DID_ERROR to scsi-ml due to FC_DATA_UNDRUN (scsi) May 21 11:01:52 poc1 kernel: [ 3374.402457] host7: xid 585: Exchange timer armed : 8000 msecs May 21 11:01:52 poc1 kernel: [ 3374.402480] host7: xid 742: Exchange timer armed : 8000 msecs May 21 11:01:52 poc1 kernel: [ 3374.403602] host7: xid 4a7: Exchange timer canceled May 21 11:01:52 poc1 kernel: [ 3374.403607] host7: fcp: 00061e: Returning DID_ERROR to scsi-ml due to FC_DATA_UNDRUN (scsi) May 21 11:01:52 poc1 kernel: [ 3374.403674] host7: xid 907: Exchange timer canceled May 21 11:01:52 poc1 kernel: [ 3374.403678] host7: fcp: 00061e: Returning DID_ERROR to scsi-ml due to FC_DATA_UNDRUN (scsi) May 21 11:01:52 poc1 kernel: [ 3374.404276] host7: fcp: 00061e: xid 0084-08c2: DDP I/O in fc_fcp_recv_data set ERROR May 21 11:01:52 poc1 kernel: [ 3374.404281] host7: xid 84: f_ctl 90000 seq 1 May 21 11:01:52 poc1 kernel: [ 3374.404492] host7: xid dac: Exchange timer armed : 8000 msecs May 21 11:01:52 poc1 kernel: [ 3374.404506] host7: xid 70a: Exchange timer armed : 8000 msecs May 21 11:01:52 poc1 kernel: [ 3374.405384] host7: xid 585: Exchange timer canceled May 21 11:01:52 poc1 kernel: [ 3374.405390] host7: fcp: 00061e: Returning DID_ERROR to scsi-ml due to FC_DATA_UNDRUN (scsi) More and more "fc_fcp_recv_data set ERROR" messages show up in the file later: May 21 11:01:53 poc1 kernel: [ 3374.614142] host7: fcp: 00061e: xid 01ed-0925: DDP I/O in fc_fcp_recv_data set ERROR May 21 11:01:53 poc1 kernel: [ 3374.614147] host7: xid 1ed: f_ctl 90000 seq 1 May 21 11:01:53 poc1 kernel: [ 3374.616658] host7: xid 4e7: Exchange timer armed : 8000 msecs May 21 11:01:53 poc1 kernel: [ 3374.617657] host7: xid 54a: Exchange timer armed : 8000 msecs May 21 11:01:53 poc1 kernel: [ 3374.617958] host7: xid 54a: Exchange timer canceled May 21 11:01:53 poc1 kernel: [ 3374.617965] host7: fcp: 00061e: Returning DID_ERROR to scsi-ml due to FC_DATA_UNDRUN (scsi) May 21 11:01:53 poc1 kernel: [ 3374.619656] host7: xid ded: Exchange timer armed : 8000 msecs May 21 11:01:53 poc1 kernel: [ 3374.620627] host7: xid 489: Exchange timer armed : 8000 msecs May 21 11:01:53 poc1 kernel: [ 3374.620744] host7: fcp: 00061e: xid 01cd-038e: DDP I/O in fc_fcp_recv_data set ERROR May 21 11:01:53 poc1 kernel: [ 3374.620747] host7: xid 1cd: f_ctl 90000 seq 1 May 21 11:01:53 poc1 kernel: [ 3374.621078] host7: xid 1cd: BLS rctl 85 - BLS reject received May 21 11:01:53 poc1 kernel: [ 3374.621389] host7: xid 3c1: Exchange timer canceled May 21 11:01:53 poc1 kernel: [ 3374.621395] host7: fcp: 00061e: Returning DID_ERROR to scsi-ml due to FC_DATA_UNDRUN (scsi) May 21 11:01:53 poc1 kernel: [ 3374.621622] host7: xid a64: Exchange timer armed : 8000 msecs May 21 11:01:53 poc1 kernel: [ 3374.622056] host7: fcp: 00061e: xid 0107-06e4: DDP I/O in fc_fcp_recv_data set ERROR May 21 11:01:53 poc1 kernel: [ 3374.622060] host7: xid 107: f_ctl 90000 seq 1 May 21 11:01:53 poc1 kernel: [ 3374.622370] host7: xid cc: exch: BLS rctl 84 - BLS accept May 21 11:01:53 poc1 kernel: [ 3374.622381] host7: fcp: 00061e: Returning DID_ERROR to scsi-ml due to FC_CMD_ABORTED May 21 11:01:53 poc1 kernel: [ 3374.622491] host7: fcp: 00061e: xid 00a5-0862: DDP I/O in fc_fcp_recv_data set ERROR May 21 11:01:53 poc1 kernel: [ 3374.622496] host7: xid a5: f_ctl 90000 seq 1 May 21 11:01:53 poc1 kernel: [ 3374.622866] host7: xid b86: Exchange timer canceled May 21 11:01:53 poc1 kernel: [ 3374.622870] host7: xid 1e6: f_ctl 90000 seq 1 May 21 11:01:53 poc1 kernel: [ 3374.622889] host7: xid dcd: Exchange timer canceled May 21 11:01:53 poc1 kernel: [ 3374.622897] host7: fcp: 00061e: Returning DID_ERROR to scsi-ml due to FC_DATA_UNDRUN (scsi) May 21 11:01:53 poc1 kernel: [ 3374.623491] host7: xid 1e6: BLS rctl 85 - BLS reject received May 21 11:01:53 poc1 kernel: [ 3374.623637] host7: xid 4a0: Exchange timer canceled May 21 11:01:53 poc1 kernel: [ 3374.623719] host7: fcp: 00061e: xid 012d-0869: DDP I/O in fc_fcp_recv_data set ERROR May 21 11:01:53 poc1 kernel: [ 3374.623724] host7: xid 12d: f_ctl 90000 seq 1 May 21 11:01:53 poc1 kernel: [ 3374.624626] host7: xid fa4: Exchange timer armed : 8000 msecs May 21 11:01:53 poc1 kernel: [ 3374.624824] host7: xid fa4: Exchange timer canceled May 21 11:01:53 poc1 kernel: [ 3374.624829] host7: fcp: 00061e: Returning DID_ERROR to scsi-ml due to FC_DATA_UNDRUN (scsi) May 21 11:01:53 poc1 kernel: [ 3374.625133] host7: fcp: 00061e: xid 00a4-0e4f: DDP I/O in fc_fcp_recv_data set ERROR May 21 11:01:53 poc1 kernel: [ 3374.625137] host7: xid a4: f_ctl 90000 seq 1 May 21 11:01:53 poc1 kernel: [ 3374.625404] host7: xid a4: BLS rctl 85 - BLS reject received May 21 11:01:53 poc1 kernel: [ 3374.625536] host7: fcp: 00061e: xid 01e7-05e7: DDP I/O in fc_fcp_recv_data set ERROR May 21 11:01:53 poc1 kernel: [ 3374.625540] host7: xid 1e7: f_ctl 90000 seq 1 May 21 11:01:53 poc1 kernel: [ 3374.625637] host7: xid fa4: Exchange timer armed : 8000 msecs May 21 11:01:53 poc1 kernel: [ 3374.625842] host7: xid 1e7: BLS rctl 85 - BLS reject received May 21 11:01:53 poc1 kernel: [ 3374.626210] host7: fcp: 00061e: xid 01a7-0666: DDP I/O in fc_fcp_recv_data set ERROR May 21 11:01:53 poc1 kernel: [ 3374.626213] host7: xid 1a7: f_ctl 90000 seq 1 May 21 11:01:53 poc1 kernel: [ 3374.626581] host7: xid 102: exch: BLS rctl 84 - BLS accept May 21 11:01:53 poc1 kernel: [ 3374.626590] host7: fcp: 00061e: Returning DID_ERROR to scsi-ml due to FC_CMD_ABORTED May 21 11:01:53 poc1 kernel: [ 3374.626602] host7: fcp: 00061e: xid 01c7-0862: DDP I/O in fc_fcp_recv_data set ERROR On Wed, May 21, 2014 at 2:03 PM, Jun Wu <jwu@xxxxxxxxxxxx> wrote: > I enabled the Tx offload and set the debug_loggings as suggested. A > few minutes run generated a 1.5GB messages file. > > The file has repeated patterns of the following: > > 18630713 May 21 11:01:53 poc2 kernel: [ 1528.182334] host7: xid b48: > f_ctl 800000 seq 1 > 18630714 May 21 11:01:53 poc2 kernel: [ 1528.182345] host7: xid b48: > f_ctl 880008 seq 2 > 18630715 May 21 11:01:53 poc2 kernel: [ 1528.182601] host7: xid 38e: > f_ctl 800000 seq 1 > 18630716 May 21 11:01:53 poc2 kernel: [ 1528.182621] host7: xid 38e: > f_ctl 880008 seq 2 > 18630717 May 21 11:01:53 poc2 kernel: [ 1528.182771] host7: xid 74a: > f_ctl 800000 seq 1 > 18630718 May 21 11:01:53 poc2 kernel: [ 1528.182785] host7: xid 74a: > f_ctl 880008 seq 2 > 18630719 May 21 11:01:53 poc2 kernel: [ 1528.183161] host7: xid e4f: > f_ctl 800000 seq 1 > 18630720 May 21 11:01:53 poc2 kernel: [ 1528.183181] host7: xid e4f: > f_ctl 880008 seq 2 > 18630721 May 21 11:01:53 poc2 kernel: [ 1528.184285] host7: xid 666: > f_ctl 800000 seq 1 > 18630722 May 21 11:01:53 poc2 kernel: [ 1528.184301] host7: xid 666: > f_ctl 880008 seq 2 > 18630723 May 21 11:01:53 poc2 kernel: [ 1528.184550] host7: xid c20: > f_ctl 800000 seq 1 > 18630724 May 21 11:01:53 poc2 kernel: [ 1528.184589] host7: xid c20: > f_ctl 880008 seq 2 > 18630725 May 21 11:01:53 poc2 kernel: [ 1528.185198] host7: xid 607: > f_ctl 800000 seq 1 > 18630726 May 21 11:01:53 poc2 kernel: [ 1528.185213] host7: xid 607: > f_ctl 880008 seq 2 > 18630727 May 21 11:01:53 poc2 kernel: [ 1528.185659] host7: xid 925: > f_ctl 800000 seq 1 > 18630728 May 21 11:01:53 poc2 kernel: [ 1528.185662] host7: xid b48: > f_ctl 800000 seq 1 > 18630729 May 21 11:01:53 poc2 kernel: [ 1528.185672] host7: xid b48: > f_ctl 880008 seq 2 > 18630730 May 21 11:01:53 poc2 kernel: [ 1528.185680] host7: xid 925: > f_ctl 880008 seq 2 > 18630731 May 21 11:01:53 poc2 kernel: [ 1528.185751] host7: xid b61: > f_ctl 800000 seq 1 > 18630732 May 21 11:01:53 poc2 kernel: [ 1528.185765] host7: xid b61: > f_ctl 880008 seq 2 > 18630733 May 21 11:01:53 poc2 kernel: [ 1528.186413] host7: xid 829: > f_ctl 800000 seq 1 > 18630734 May 21 11:01:53 poc2 kernel: [ 1528.186425] host7: xid 829: > f_ctl 880008 seq 2 > 18630735 May 21 11:01:53 poc2 kernel: [ 1528.186785] host7: xid 6e4: > f_ctl 800000 seq 1 > 18630736 May 21 11:01:53 poc2 kernel: [ 1528.186817] host7: xid 6e4: > f_ctl 880008 seq 2 > 18630737 May 21 11:01:53 poc2 kernel: [ 1528.186932] host7: xid dab: > f_ctl 800000 seq 1 > 18630738 May 21 11:01:53 poc2 kernel: [ 1528.186946] host7: xid dab: > f_ctl 880008 seq 2 > 18630739 May 21 11:01:53 poc2 kernel: [ 1528.187907] host7: xid 8ac: > f_ctl 800000 seq 1 > 18630740 May 21 11:01:53 poc2 kernel: [ 1528.187920] host7: xid 8ac: > f_ctl 880008 seq 2 > 18630741 May 21 11:01:53 poc2 kernel: [ 1528.188656] host7: xid 38e: > f_ctl 800000 seq 1 > 18630742 May 21 11:01:53 poc2 kernel: [ 1528.188675] host7: xid 38e: > f_ctl 880008 seq 2 > 18630743 May 21 11:01:53 poc2 kernel: [ 1528.188889] host7: xid b61: > f_ctl 800000 seq 1 > 18630744 May 21 11:01:53 poc2 kernel: [ 1528.188899] host7: xid b61: > f_ctl 880008 seq 2 > 18630745 May 21 11:01:53 poc2 kernel: [ 1528.189281] host7: xid 88d: > f_ctl 800000 seq 1 > 18630746 May 21 11:01:53 poc2 kernel: [ 1528.189301] host7: xid 88d: > f_ctl 880008 seq 2 > 18630747 May 21 11:01:53 poc2 kernel: [ 1528.189378] host7: xid c20: > f_ctl 800000 seq 1 > 18630748 May 21 11:01:53 poc2 kernel: [ 1528.189392] host7: xid c20: > f_ctl 880008 seq 2 > 18630749 May 21 11:01:53 poc2 kernel: [ 1528.189836] host7: xid 862: > f_ctl 800000 seq 1 > 18630750 May 21 11:01:53 poc2 kernel: [ 1528.189850] host7: xid 862: > f_ctl 880008 seq 2 > 18630751 May 21 11:01:53 poc2 kernel: [ 1528.191740] host7: xid 6e4: > Exchange timer armed : 0 msecs > 18630752 May 21 11:01:53 poc2 kernel: [ 1528.191747] host7: xid 6e4: > f_ctl 800000 seq 1 > 18630753 May 21 11:01:53 poc2 kernel: [ 1528.191756] host7: xid 6e4: > f_ctl 800000 seq 2 > 18630754 May 21 11:01:53 poc2 kernel: [ 1528.191763] host7: xid 6e4: > Exchange timed out > 18630755 May 21 11:01:53 poc2 kernel: [ 1528.191777] ft_queue_data_in: > Failed to send frame ffff8805bf245e00, xid <0x6e4>, remaining 458752, > lso_max <0x10000> > 18630756 May 21 11:01:53 poc2 kernel: [ 1528.191782] ft_queue_data_in: > Failed to send frame ffff8805bf245e00, xid <0x6e4>, remaining 393216, > lso_max <0x10000> > 18630757 May 21 11:01:53 poc2 kernel: [ 1528.191786] ft_queue_data_in: > Failed to send frame ffff8805bf245e00, xid <0x6e4>, remaining 327680, > lso_max <0x10000> > 18630758 May 21 11:01:53 poc2 kernel: [ 1528.191790] ft_queue_data_in: > Failed to send frame ffff8805bf245e00, xid <0x6e4>, remaining 262144, > lso_max <0x10000> > 18630759 May 21 11:01:53 poc2 kernel: [ 1528.191794] ft_queue_data_in: > Failed to send frame ffff8805bf245e00, xid <0x6e4>, remaining 196608, > lso_max <0x10000> > 18630760 May 21 11:01:53 poc2 kernel: [ 1528.191798] ft_queue_data_in: > Failed to send frame ffff8805bf245e00, xid <0x6e4>, remaining 131072, > lso_max <0x10000> > 18630761 May 21 11:01:53 poc2 kernel: [ 1528.191801] ft_queue_data_in: > Failed to send frame ffff8805bf245e00, xid <0x6e4>, remaining 65536, > lso_max <0x10000> > 18630762 May 21 11:01:53 poc2 kernel: [ 1528.191805] ft_queue_data_in: > Failed to send frame ffff8805bf245e00, xid <0x6e4>, remaining 0, > lso_max <0x10000> > 18630763 May 21 11:01:53 poc2 kernel: [ 1528.192163] host7: xid b48: > f_ctl 800000 seq 1 > 18630764 May 21 11:01:53 poc2 kernel: [ 1528.192166] host7: xid 607: > f_ctl 800000 seq 1 > 18630765 May 21 11:01:53 poc2 kernel: [ 1528.192176] host7: xid 607: > f_ctl 880008 seq 2 > 18630766 May 21 11:01:53 poc2 kernel: [ 1528.192180] host7: xid b48: > f_ctl 880008 seq 2 > 18630767 May 21 11:01:53 poc2 kernel: [ 1528.192266] host7: xid 666: > f_ctl 800000 seq 1 > > Above is the first time ft_queue_data_in message shows up in the file. > Here is another instance: > > 18631537 May 21 11:01:53 poc2 kernel: [ 1528.333876] host7: xid 74a: > f_ctl 800000 seq 1 > 18631538 May 21 11:01:53 poc2 kernel: [ 1528.333893] host7: xid 74a: > f_ctl 880008 seq 2 > 18631539 May 21 11:01:53 poc2 kernel: [ 1528.334816] host7: xid c20: > f_ctl 800000 seq 1 > 18631540 May 21 11:01:53 poc2 kernel: [ 1528.334834] host7: xid c20: > f_ctl 880008 seq 2 > 18631541 May 21 11:01:53 poc2 kernel: [ 1528.334847] host7: xid b81: > f_ctl 800000 seq 1 > 18631542 May 21 11:01:53 poc2 kernel: [ 1528.334858] host7: xid 983: > f_ctl 800000 seq 1 > 18631543 May 21 11:01:53 poc2 kernel: [ 1528.334864] host7: xid b81: > f_ctl 880008 seq 2 > 18631544 May 21 11:01:53 poc2 kernel: [ 1528.334881] host7: xid 983: > f_ctl 880008 seq 2 > 18631545 May 21 11:01:53 poc2 kernel: [ 1528.334972] host7: xid 686: > f_ctl 800000 seq 1 > 18631546 May 21 11:01:53 poc2 kernel: [ 1528.334985] host7: xid 686: > f_ctl 880008 seq 2 > 18631547 May 21 11:01:53 poc2 kernel: [ 1528.335036] host7: xid 704: > f_ctl 800000 seq 1 > 18631548 May 21 11:01:53 poc2 kernel: [ 1528.335052] host7: xid 704: > f_ctl 880008 seq 2 > 18631549 May 21 11:01:53 poc2 kernel: [ 1528.335078] host7: xid 627: > f_ctl 800000 seq 1 > 18631550 May 21 11:01:53 poc2 kernel: [ 1528.335088] host7: xid 627: > f_ctl 880008 seq 2 > 18631551 May 21 11:01:53 poc2 kernel: [ 1528.335202] host7: xid b48: > f_ctl 800000 seq 1 > 18631552 May 21 11:01:53 poc2 kernel: [ 1528.335214] host7: xid b48: > f_ctl 880008 seq 2 > 18631553 May 21 11:01:53 poc2 kernel: [ 1528.335381] host7: xid 74a: > Exchange timer armed : 0 msecs > 18631554 May 21 11:01:53 poc2 kernel: [ 1528.335386] host7: xid 74a: > f_ctl 800000 seq 1 > 18631555 May 21 11:01:53 poc2 kernel: [ 1528.335388] host7: xid 74a: > Exchange timed out > 18631556 May 21 11:01:53 poc2 kernel: [ 1528.335672] host7: xid 869: > f_ctl 800000 seq 1 > 18631557 May 21 11:01:53 poc2 kernel: [ 1528.335688] host7: xid 869: > f_ctl 880008 seq 2 > 18631558 May 21 11:01:53 poc2 kernel: [ 1528.336213] host7: xid 965: > f_ctl 800000 seq 1 > 18631559 May 21 11:01:53 poc2 kernel: [ 1528.336232] host7: xid 965: > f_ctl 880008 seq 2 > 18631560 May 21 11:01:53 poc2 kernel: [ 1528.336477] host7: xid 74a: > f_ctl 800000 seq 2 > 18631561 May 21 11:01:53 poc2 kernel: [ 1528.336482] ft_queue_data_in: > Failed to send frame ffff8802e25c1e00, xid <0x74a>, remaining 196608, > lso_max <0x10000> > 18631562 May 21 11:01:53 poc2 kernel: [ 1528.336486] ft_queue_data_in: > Failed to send frame ffff8802e25c1e00, xid <0x74a>, remaining 131072, > lso_max <0x10000> > 18631563 May 21 11:01:53 poc2 kernel: [ 1528.336489] host7: xid 74a: > f_ctl 800000 seq 3 > 18631564 May 21 11:01:53 poc2 kernel: [ 1528.337645] host7: xid 86d: > f_ctl 800000 seq 1 > 18631565 May 21 11:01:53 poc2 kernel: [ 1528.337759] host7: xid 86d: > f_ctl 880008 seq 2 > 18631566 May 21 11:01:53 poc2 kernel: [ 1528.337827] host7: xid 44e: > f_ctl 800000 seq 1 > 18631567 May 21 11:01:53 poc2 kernel: [ 1528.337846] host7: xid 44e: > f_ctl 880008 seq 2 > 18631568 May 21 11:01:53 poc2 kernel: [ 1528.340521] host7: xid 8c2: > Exchange timer armed : 0 msecs > 18631569 May 21 11:01:53 poc2 kernel: [ 1528.340526] host7: xid 8c2: > f_ctl 800000 seq 1 > 18631570 May 21 11:01:53 poc2 kernel: [ 1528.340667] host7: xid 8c2: > Exchange timed out > 18631571 May 21 11:01:53 poc2 kernel: [ 1528.341064] host7: xid 983: > f_ctl 800000 seq 1 > 18631572 May 21 11:01:53 poc2 kernel: [ 1528.341087] host7: xid 983: > f_ctl 880008 seq 2 > 18631573 May 21 11:01:53 poc2 kernel: [ 1528.341286] host7: xid b48: > f_ctl 800000 seq 1 > 18631574 May 21 11:01:53 poc2 kernel: [ 1528.341306] host7: xid b48: > f_ctl 880008 seq 2 > 18631575 May 21 11:01:53 poc2 kernel: [ 1528.341522] host7: xid 869: > Exchange timer armed : 0 msecs > 18631576 May 21 11:01:53 poc2 kernel: [ 1528.341528] host7: xid 869: > f_ctl 800000 seq 1 > 18631577 May 21 11:01:53 poc2 kernel: [ 1528.341539] host7: xid 869: > Exchange timed out > 18631578 May 21 11:01:53 poc2 kernel: [ 1528.341966] host7: xid 965: > f_ctl 800000 seq 1 > 18631579 May 21 11:01:53 poc2 kernel: [ 1528.341979] host7: xid 965: > f_ctl 880008 seq 2 > 18631580 May 21 11:01:53 poc2 kernel: [ 1528.342450] host7: xid 627: > f_ctl 800000 seq 1 > 18631581 May 21 11:01:53 poc2 kernel: [ 1528.342467] host7: xid 627: > f_ctl 880008 seq 2 > 18631582 May 21 11:01:53 poc2 kernel: [ 1528.342945] host7: xid 70a: > f_ctl 800000 seq 1 > 18631583 May 21 11:01:53 poc2 kernel: [ 1528.342959] host7: xid 70a: > f_ctl 880008 seq 2 > > After stripping out the repeated "host7: xid xxx: f_ctl 800000 seq > x" messages, we have: > > 18630507:May 21 11:01:53 poc2 kernel: [ 1528.148314] host7: xid d6b: > Exchange timer armed : 0 msecs > 18630509:May 21 11:01:53 poc2 kernel: [ 1528.148357] host7: xid d6b: > Exchange timed out > 18630642:May 21 11:01:53 poc2 kernel: [ 1528.169143] host7: xid 88c: > Exchange timer armed : 0 msecs > 18630644:May 21 11:01:53 poc2 kernel: [ 1528.169197] host7: xid 88c: > Exchange timed out > 18630751:May 21 11:01:53 poc2 kernel: [ 1528.191740] host7: xid 6e4: > Exchange timer armed : 0 msecs > 18630754:May 21 11:01:53 poc2 kernel: [ 1528.191763] host7: xid 6e4: > Exchange timed out > 18630755:May 21 11:01:53 poc2 kernel: [ 1528.191777] ft_queue_data_in: > Failed to send frame ffff8805bf245e00, xid <0x6e4>, remaining 458752, > lso_max <0x10000> > 18630756:May 21 11:01:53 poc2 kernel: [ 1528.191782] ft_queue_data_in: > Failed to send frame ffff8805bf245e00, xid <0x6e4>, remaining 393216, > lso_max <0x10000> > 18630757:May 21 11:01:53 poc2 kernel: [ 1528.191786] ft_queue_data_in: > Failed to send frame ffff8805bf245e00, xid <0x6e4>, remaining 327680, > lso_max <0x10000> > 18630758:May 21 11:01:53 poc2 kernel: [ 1528.191790] ft_queue_data_in: > Failed to send frame ffff8805bf245e00, xid <0x6e4>, remaining 262144, > lso_max <0x10000> > 18630759:May 21 11:01:53 poc2 kernel: [ 1528.191794] ft_queue_data_in: > Failed to send frame ffff8805bf245e00, xid <0x6e4>, remaining 196608, > lso_max <0x10000> > 18630760:May 21 11:01:53 poc2 kernel: [ 1528.191798] ft_queue_data_in: > Failed to send frame ffff8805bf245e00, xid <0x6e4>, remaining 131072, > lso_max <0x10000> > 18630761:May 21 11:01:53 poc2 kernel: [ 1528.191801] ft_queue_data_in: > Failed to send frame ffff8805bf245e00, xid <0x6e4>, remaining 65536, > lso_max <0x10000> > 18630762:May 21 11:01:53 poc2 kernel: [ 1528.191805] ft_queue_data_in: > Failed to send frame ffff8805bf245e00, xid <0x6e4>, remaining 0, > lso_max <0x10000> > 18630777:May 21 11:01:53 poc2 kernel: [ 1528.195937] host7: xid 666: > Exchange timer armed : 0 msecs > 18630779:May 21 11:01:53 poc2 kernel: [ 1528.195983] host7: xid 666: > Exchange timed out > 18630780:May 21 11:01:53 poc2 kernel: [ 1528.196336] host7: xid 862: > Exchange timer armed : 0 msecs > 18630782:May 21 11:01:53 poc2 kernel: [ 1528.196348] host7: xid 862: > Exchange timed out > 18630879:May 21 11:01:53 poc2 kernel: [ 1528.211080] host7: xid b61: > Exchange timer armed : 0 msecs > 18630881:May 21 11:01:53 poc2 kernel: [ 1528.211137] host7: xid b61: > Exchange timed out > 18630898:May 21 11:01:53 poc2 kernel: [ 1528.214253] host7: xid 38e: > Exchange timer armed : 0 msecs > 18630900:May 21 11:01:53 poc2 kernel: [ 1528.214284] host7: xid 38e: > Exchange timed out > 18631007:May 21 11:01:53 poc2 kernel: [ 1528.236137] host7: xid 88d: > Exchange timer armed : 0 msecs > 18631009:May 21 11:01:53 poc2 kernel: [ 1528.236172] host7: xid 88d: > Exchange timed out > 18631176:May 21 11:01:53 poc2 kernel: [ 1528.264223] host7: xid 607: > Exchange timer armed : 0 msecs > 18631178:May 21 11:01:53 poc2 kernel: [ 1528.264239] host7: xid 607: > Exchange timed out > 18631185:May 21 11:01:53 poc2 kernel: [ 1528.266310] host7: xid 82d: > Exchange timer armed : 0 msecs > 18631187:May 21 11:01:53 poc2 kernel: [ 1528.266351] host7: xid 82d: > Exchange timed out > 18631226:May 21 11:01:53 poc2 kernel: [ 1528.275452] host7: xid 8a2: > Exchange timer armed : 0 msecs > 18631228:May 21 11:01:53 poc2 kernel: [ 1528.275463] host7: xid 8a2: > Exchange timed out > 18631553:May 21 11:01:53 poc2 kernel: [ 1528.335381] host7: xid 74a: > Exchange timer armed : 0 msecs > 18631555:May 21 11:01:53 poc2 kernel: [ 1528.335388] host7: xid 74a: > Exchange timed out > 18631561:May 21 11:01:53 poc2 kernel: [ 1528.336482] ft_queue_data_in: > Failed to send frame ffff8802e25c1e00, xid <0x74a>, remaining 196608, > lso_max <0x10000> > 18631562:May 21 11:01:53 poc2 kernel: [ 1528.336486] ft_queue_data_in: > Failed to send frame ffff8802e25c1e00, xid <0x74a>, remaining 131072, > lso_max <0x10000> > 18631568:May 21 11:01:53 poc2 kernel: [ 1528.340521] host7: xid 8c2: > Exchange timer armed : 0 msecs > 18631570:May 21 11:01:53 poc2 kernel: [ 1528.340667] host7: xid 8c2: > Exchange timed out > 18631575:May 21 11:01:53 poc2 kernel: [ 1528.341522] host7: xid 869: > Exchange timer armed : 0 msecs > 18631577:May 21 11:01:53 poc2 kernel: [ 1528.341539] host7: xid 869: > Exchange timed out > 18631660:May 21 11:01:53 poc2 kernel: [ 1528.356897] host7: xid e4f: > Exchange timer armed : 0 msecs > 18631662:May 21 11:01:53 poc2 kernel: [ 1528.356975] host7: xid e4f: > Exchange timed out > 18631825:May 21 11:01:53 poc2 kernel: [ 1528.398431] host7: xid 965: > Exchange timer armed : 0 msecs > 18631827:May 21 11:01:53 poc2 kernel: [ 1528.398509] host7: xid 965: > Exchange timed out > 18631844:May 21 11:01:53 poc2 kernel: [ 1528.401772] host7: xid 44e: > Exchange timer armed : 0 msecs > 18631846:May 21 11:01:53 poc2 kernel: [ 1528.401850] host7: xid 44e: > Exchange timed out > 18631859:May 21 11:01:53 poc2 kernel: [ 1528.404996] host7: xid 704: > Exchange timer armed : 0 msecs > 18631861:May 21 11:01:53 poc2 kernel: [ 1528.405043] host7: xid 704: > Exchange timed out > 18631876:May 21 11:01:53 poc2 kernel: [ 1528.412608] host7: xid 86d: > Exchange timer armed : 0 msecs > 18631878:May 21 11:01:53 poc2 kernel: [ 1528.412619] host7: xid 86d: > Exchange timed out > 18631919:May 21 11:01:53 poc2 kernel: [ 1528.431939] host7: xid 686: > Exchange timer armed : 0 msecs > 18631921:May 21 11:01:53 poc2 kernel: [ 1528.432000] host7: xid 686: > Exchange timed out > 18631950:May 21 11:01:53 poc2 kernel: [ 1528.441542] host7: xid def: > Exchange timer armed : 0 msecs > 18631952:May 21 11:01:53 poc2 kernel: [ 1528.441573] host7: xid def: > Exchange timed out > 18631967:May 21 11:01:53 poc2 kernel: [ 1528.447848] host7: xid 3ce: > Exchange timer armed : 0 msecs > 18631969:May 21 11:01:53 poc2 kernel: [ 1528.447863] host7: xid 3ce: > Exchange timed out > 18632008:May 21 11:01:53 poc2 kernel: [ 1528.453868] host7: xid 627: > Exchange timer armed : 0 msecs > 18632010:May 21 11:01:53 poc2 kernel: [ 1528.453879] host7: xid 627: > Exchange timed out > 18655343:May 21 11:01:56 poc2 kernel: [ 1531.747863] host7: xid 48e: > Exchange timer armed : 0 msecs > 18655345:May 21 11:01:56 poc2 kernel: [ 1531.747879] host7: xid 48e: > Exchange timed out > 18655414:May 21 11:01:56 poc2 kernel: [ 1531.758175] host7: xid 667: > Exchange timer armed : 0 msecs > 18655416:May 21 11:01:56 poc2 kernel: [ 1531.758204] host7: xid 667: > Exchange timed out > 18655533:May 21 11:01:56 poc2 kernel: [ 1531.776753] host7: xid 6c6: > Exchange timer armed : 0 msecs > 18655535:May 21 11:01:56 poc2 kernel: [ 1531.776765] host7: xid 6c6: > Exchange timed out > 18655832:May 21 11:01:56 poc2 kernel: [ 1531.818184] host7: xid 647: > Exchange timer armed : 0 msecs > 18655834:May 21 11:01:56 poc2 kernel: [ 1531.818227] host7: xid 647: > Exchange timed out > 18655893:May 21 11:01:56 poc2 kernel: [ 1531.825287] host7: xid 46e: > Exchange timer armed : 0 msecs > 18655895:May 21 11:01:56 poc2 kernel: [ 1531.825344] host7: xid 46e: > Exchange timed out > 18656116:May 21 11:01:56 poc2 kernel: [ 1531.863739] host7: xid dab: > Exchange timer armed : 0 msecs > 18656118:May 21 11:01:56 poc2 kernel: [ 1531.863765] host7: xid dab: > Exchange timed out > 18656124:May 21 11:01:56 poc2 kernel: [ 1531.865740] host7: xid 8ac: > Exchange timer armed : 0 msecs > 18656129:May 21 11:01:56 poc2 kernel: [ 1531.865800] host7: xid 8ac: > Exchange timed out > 18656130:May 21 11:01:56 poc2 kernel: [ 1531.865813] host7: xid 5e7: > Exchange timer armed : 2000 msecs > 18656267:May 21 11:01:56 poc2 kernel: [ 1531.891151] host7: xid e6f: > Exchange timer armed : 0 msecs > 18656269:May 21 11:01:56 poc2 kernel: [ 1531.891177] host7: xid e6f: > Exchange timed out > 18656320:May 21 11:01:56 poc2 kernel: [ 1531.899587] host7: xid 82c: > Exchange timer armed : 0 msecs > 18656322:May 21 11:01:56 poc2 kernel: [ 1531.899606] host7: xid 82c: > Exchange timed out > 18656355:May 21 11:01:56 poc2 kernel: [ 1531.906904] host7: xid 646: > Exchange timer armed : 0 msecs > 18656357:May 21 11:01:56 poc2 kernel: [ 1531.906952] host7: xid 646: > Exchange timed out > 18656608:May 21 11:01:56 poc2 kernel: [ 1531.960276] host7: xid 945: > Exchange timer armed : 0 msecs > 18656610:May 21 11:01:56 poc2 kernel: [ 1531.960293] host7: xid 945: > Exchange timed out > 18656613:May 21 11:01:56 poc2 kernel: [ 1531.960985] host7: xid 724: > Exchange timer armed : 0 msecs > 18656615:May 21 11:01:56 poc2 kernel: [ 1531.961002] host7: xid 724: > Exchange timed out > 18656634:May 21 11:01:56 poc2 kernel: [ 1531.968643] host7: xid 8ad: > Exchange timer armed : 0 msecs > 18656636:May 21 11:01:56 poc2 kernel: [ 1531.968660] host7: xid 8ad: > Exchange timed out > 18656641:May 21 11:01:56 poc2 kernel: [ 1531.974843] host7: xid 5e7: > Exchange timer armed : 10000 msecs > 18656642:May 21 11:01:56 poc2 kernel: [ 1531.974848] host7: xid 5e7: > f_ctl 90000 seq 1 > 18656655:May 21 11:01:56 poc2 kernel: [ 1531.983896] host7: xid 687: > Exchange timer armed : 0 msecs > 18656657:May 21 11:01:56 poc2 kernel: [ 1531.983912] host7: xid 687: > Exchange timed out > 18656746:May 21 11:01:56 poc2 kernel: [ 1532.011935] host7: xid 5e7: > Exchange timer armed : 10000 msecs > 18656747:May 21 11:01:56 poc2 kernel: [ 1532.011940] host7: xid 5e7: > f_ctl 90000 seq 2 > 18669030:May 21 11:01:58 poc2 kernel: [ 1533.870105] host7: xid 5e7: > Exchange timed out > 18841592:May 21 11:02:27 poc2 kernel: [ 1562.646991] host7: xid 90d: > Exchange timer armed : 0 msecs > 18841595:May 21 11:02:27 poc2 kernel: [ 1562.647066] host7: xid 90d: > Exchange timed out > 18842498:May 21 11:02:27 poc2 kernel: [ 1562.795804] host7: xid 8ac: > Exchange timer armed : 0 msecs > 18842500:May 21 11:02:27 poc2 kernel: [ 1562.795820] host7: xid 8ac: > Exchange timed out > 18842547:May 21 11:02:27 poc2 kernel: [ 1562.805335] host7: xid 94c: > Exchange timer armed : 0 msecs > 18842549:May 21 11:02:27 poc2 kernel: [ 1562.805351] host7: xid 94c: > Exchange timed out > 18842584:May 21 11:02:27 poc2 kernel: [ 1562.810550] host7: xid f2f: > Exchange timer armed : 0 msecs > 18842586:May 21 11:02:27 poc2 kernel: [ 1562.810605] host7: xid f2f: > Exchange timed out > 18842593:May 21 11:02:27 poc2 kernel: [ 1562.811735] host7: xid 7e6: > Exchange timer armed : 0 msecs > 18842595:May 21 11:02:27 poc2 kernel: [ 1562.811795] host7: xid 7e6: > Exchange timed out > 18842740:May 21 11:02:27 poc2 kernel: [ 1562.837943] host7: xid e8b: > Exchange timer armed : 0 msecs > 18842742:May 21 11:02:27 poc2 kernel: [ 1562.837966] host7: xid 8cd: > Exchange timer armed : 0 msecs > 18842743:May 21 11:02:27 poc2 kernel: [ 1562.837968] host7: xid e8b: > Exchange timed out > 18842745:May 21 11:02:27 poc2 kernel: [ 1562.838038] host7: xid 8cd: > Exchange timed out > 18842872:May 21 11:02:27 poc2 kernel: [ 1562.863661] host7: xid 824: > Exchange timer armed : 0 msecs > 18842874:May 21 11:02:27 poc2 kernel: [ 1562.863674] host7: xid 824: > Exchange timed out > 18842951:May 21 11:02:27 poc2 kernel: [ 1562.876390] host7: xid 766: > Exchange timer armed : 0 msecs > 18842953:May 21 11:02:27 poc2 kernel: [ 1562.876405] host7: xid 766: > Exchange timed out > 18842960:May 21 11:02:27 poc2 kernel: [ 1562.878444] host7: xid ca8: > Exchange timer armed : 0 msecs > 18842962:May 21 11:02:27 poc2 kernel: [ 1562.878470] host7: xid ca8: > Exchange timed out > 18843025:May 21 11:02:27 poc2 kernel: [ 1562.892335] host7: xid a25: > Exchange timer armed : 0 msecs > 18843027:May 21 11:02:27 poc2 kernel: [ 1562.892400] host7: xid a25: > Exchange timed out > 18843092:May 21 11:02:27 poc2 kernel: [ 1562.905946] host7: xid be1: > Exchange timer armed : 0 msecs > 18843094:May 21 11:02:27 poc2 kernel: [ 1562.905971] host7: xid be1: > Exchange timed out > 18843239:May 21 11:02:27 poc2 kernel: [ 1562.934586] host7: xid 52e: > Exchange timer armed : 0 msecs > 18843241:May 21 11:02:27 poc2 kernel: [ 1562.934601] host7: xid 52e: > Exchange timed out > 18843264:May 21 11:02:27 poc2 kernel: [ 1562.938141] host7: xid bc8: > Exchange timer armed : 0 msecs > 18843266:May 21 11:02:27 poc2 kernel: [ 1562.938164] host7: xid bc8: > Exchange timed out > 18843353:May 21 11:02:27 poc2 kernel: [ 1562.959150] host7: xid c21: > Exchange timer armed : 0 msecs > 18843356:May 21 11:02:27 poc2 kernel: [ 1562.959194] ft_queue_data_in: > 10 callbacks suppressed > 18843357:May 21 11:02:27 poc2 kernel: [ 1562.959196] ft_queue_data_in: > Failed to send frame ffff8800ba6e6400, xid <0xc21>, remaining 458752, > lso_max <0x10000> > 18843358:May 21 11:02:27 poc2 kernel: [ 1562.959199] ft_queue_data_in: > Failed to send frame ffff8800ba6e6400, xid <0xc21>, remaining 393216, > lso_max <0x10000> > 18843359:May 21 11:02:27 poc2 kernel: [ 1562.959202] ft_queue_data_in: > Failed to send frame ffff8800ba6e6400, xid <0xc21>, remaining 327680, > lso_max <0x10000> > 18843360:May 21 11:02:27 poc2 kernel: [ 1562.959204] ft_queue_data_in: > Failed to send frame ffff8800ba6e6400, xid <0xc21>, remaining 262144, > lso_max <0x10000> > 18843361:May 21 11:02:27 poc2 kernel: [ 1562.959208] host7: xid c21: > Exchange timed out > 18843362:May 21 11:02:27 poc2 kernel: [ 1562.959209] ft_queue_data_in: > Failed to send frame ffff8800ba6e6400, xid <0xc21>, remaining 196608, > lso_max <0x10000> > 18843363:May 21 11:02:27 poc2 kernel: [ 1562.959212] ft_queue_data_in: > Failed to send frame ffff8800ba6e6400, xid <0xc21>, remaining 131072, > lso_max <0x10000> > 18843364:May 21 11:02:27 poc2 kernel: [ 1562.959214] ft_queue_data_in: > Failed to send frame ffff8800ba6e6400, xid <0xc21>, remaining 65536, > lso_max <0x10000> > 18843365:May 21 11:02:27 poc2 kernel: [ 1562.959217] ft_queue_data_in: > Failed to send frame ffff8800ba6e6400, xid <0xc21>, remaining 0, > lso_max <0x10000> > 18843416:May 21 11:02:27 poc2 kernel: [ 1562.968554] host7: xid f4f: > Exchange timer armed : 0 msecs > 18843426:May 21 11:02:27 poc2 kernel: [ 1562.968656] host7: xid f4f: > Exchange timed out > 18843581:May 21 11:02:27 poc2 kernel: [ 1562.991761] host7: xid 7e4: > Exchange timer armed : 0 msecs > 18843583:May 21 11:02:27 poc2 kernel: [ 1562.991853] host7: xid 7e4: > Exchange timed out > 18843690:May 21 11:02:27 poc2 kernel: [ 1563.011282] host7: xid a05: > Exchange timer armed : 0 msecs > > Thanks > > > On Wed, May 21, 2014 at 9:21 AM, Vasu Dev <vasu.dev@xxxxxxxxxxxxxxx> wrote: >> On Tue, 2014-05-20 at 22:29 -0700, Jun Wu wrote: >>> MTU were 1500 for both initiator and target. >>> I used "ethtool -K p4p1 tso off" to turn off tcp segmentation offload >>> on all machines. Register setting after the command is shown below. >>> >>> [root@poc3 jkong]# ethtool -k p4p1 >>> Features for p4p1: >>> rx-checksumming: on >>> tx-checksumming: on >>> tx-checksum-ipv4: on >>> tx-checksum-ip-generic: off [fixed] >>> tx-checksum-ipv6: on >>> tx-checksum-fcoe-crc: on [fixed] >>> tx-checksum-sctp: on >>> scatter-gather: on >>> tx-scatter-gather: on >>> tx-scatter-gather-fraglist: off [fixed] >>> tcp-segmentation-offload: off >>> tx-tcp-segmentation: off >>> tx-tcp-ecn-segmentation: off [fixed] >>> tx-tcp6-segmentation: off >>> udp-fragmentation-offload: off [fixed] >>> generic-segmentation-offload: on >>> generic-receive-offload: on >>> large-receive-offload: off >>> rx-vlan-offload: on >>> tx-vlan-offload: on >>> ntuple-filters: off >>> receive-hashing: on >>> highdma: on [fixed] >>> rx-vlan-filter: on >>> vlan-challenged: off [fixed] >>> tx-lockless: off [fixed] >>> netns-local: off [fixed] >>> tx-gso-robust: off [fixed] >>> tx-fcoe-segmentation: on [fixed] >>> tx-gre-segmentation: off [fixed] >>> tx-ipip-segmentation: off [fixed] >>> tx-sit-segmentation: off [fixed] >>> tx-udp_tnl-segmentation: off [fixed] >>> tx-mpls-segmentation: off [fixed] >>> fcoe-mtu: on [fixed] >>> tx-nocache-copy: on >>> loopback: off [fixed] >>> rx-fcs: off [fixed] >>> rx-all: off >>> tx-vlan-stag-hw-insert: off [fixed] >>> rx-vlan-stag-hw-parse: off [fixed] >>> rx-vlan-stag-filter: off [fixed] >>> l2-fwd-offload: off >>> >>> Info on NIC drivers >>> >>> [root@poc3 jkong]# ethtool -i p4p1 >>> driver: ixgbe >>> version: 3.15.1-k >>> firmware-version: 0x80000208 >>> bus-info: 0000:08:00.0 >>> supports-statistics: yes >>> supports-test: yes >>> supports-eeprom-access: yes >>> supports-register-dump: yes >>> supports-priv-flags: no >>> >>> After the change, I repeated the same test and got similar failure on >>> target side: >>> >>> [12253.032595] ft_queue_data_in: Failed to send frame >>> ffff88062a638600, xid <0xa0c>, remaining 458752, lso_max <0x10000> >> >> It is send frame failure and to find out what caused send failure more >> debug info in low level fcoe Tx path functions will be helpful, it can >> be done by:- >> >> # echo 0xFF > /sys/module/libfc/parameters/debug_logging >> # echo 0x1 > /sys/module/fcoe/parameters/debug_logging >> >> Disabling Tx offload may not help here and instead would slow down Tx, >> so have them restored. >> >> Also, are you using switch between hosts and target ? In any case you >> would need DCB PFC or PAUSE enabled to avoid excessive Tx retries though >> that should not cause send failure. >> >> >> //Vasu >> >> >>> [12253.032605] ft_queue_data_in: Failed to send frame >>> ffff88062a638600, xid <0xa0c>, remaining 393216, lso_max <0x10000> >>> [12253.032609] ft_queue_data_in: Failed to send frame >>> ffff88062a638600, xid <0xa0c>, remaining 327680, lso_max <0x10000> >>> [12253.032613] ft_queue_data_in: Failed to send frame >>> ffff88062a638600, xid <0xa0c>, remaining 262144, lso_max <0x10000> >>> [12284.299877] ft_queue_data_in: Failed to send frame >>> ffff8803202ec600, xid <0x3a2>, remaining 196608, lso_max <0x10000> >>> [12284.299885] ft_queue_data_in: Failed to send frame >>> ffff8803202ec600, xid <0x3a2>, remaining 131072, lso_max <0x10000> >>> [12284.299889] ft_queue_data_in: Failed to send frame >>> ffff8803202ec600, xid <0x3a2>, remaining 65536, lso_max <0x10000> >>> [12284.299892] ft_queue_data_in: Failed to send frame >>> ffff8803202ec600, xid <0x3a2>, remaining 0, lso_max <0x10000> >>> [12284.451810] ft_queue_data_in: Failed to send frame >>> ffff88061deb1400, xid <0xecf>, remaining 458752, lso_max <0x10000> >>> [12284.451818] ft_queue_data_in: Failed to send frame >>> ffff88061deb1400, xid <0xecf>, remaining 393216, lso_max <0x10000> >>> [12284.451824] ft_queue_data_in: Failed to send frame >>> ffff88061deb1400, xid <0xecf>, remaining 327680, lso_max <0x10000> >>> [12284.451827] ft_queue_data_in: Failed to send frame >>> ffff88061deb1400, xid <0xecf>, remaining 262144, lso_max <0x10000> >>> [12284.451831] ft_queue_data_in: Failed to send frame >>> ffff88061deb1400, xid <0xecf>, remaining 196608, lso_max <0x10000> >>> [12284.451834] ft_queue_data_in: Failed to send frame >>> ffff88061deb1400, xid <0xecf>, remaining 131072, lso_max <0x10000> >>> [12347.503478] ft_queue_data_in: 2 callbacks suppressed >>> [12347.503486] ft_queue_data_in: Failed to send frame >>> ffff8806142bc800, xid <0xb4f>, remaining 458752, lso_max <0x10000> >>> [12347.503492] ft_queue_data_in: Failed to send frame >>> ffff8806142bc800, xid <0xb4f>, remaining 393216, lso_max <0x10000> >>> [12347.503496] ft_queue_data_in: Failed to send frame >>> ffff8806142bc800, xid <0xb4f>, remaining 327680, lso_max <0x10000> >>> [12347.503517] ft_queue_data_in: Failed to send frame >>> ffff8806142bc800, xid <0xb4f>, remaining 262144, lso_max <0x10000> >>> [12378.402412] ft_queue_data_in: Failed to send frame >>> ffff88062ddeac00, xid <0x6a5>, remaining 458752, lso_max <0x10000> >>> [12378.402420] ft_queue_data_in: Failed to send frame >>> ffff88062ddeac00, xid <0x6a5>, remaining 393216, lso_max <0x10000> >>> [12378.402425] ft_queue_data_in: Failed to send frame >>> ffff88062ddeac00, xid <0x6a5>, remaining 327680, lso_max <0x10000> >>> [12378.402428] ft_queue_data_in: Failed to send frame >>> ffff88062ddeac00, xid <0x6a5>, remaining 262144, lso_max <0x10000> >>> [12378.402432] ft_queue_data_in: Failed to send frame >>> ffff88062ddeac00, xid <0x6a5>, remaining 196608, lso_max <0x10000> >>> [12378.402436] ft_queue_data_in: Failed to send frame >>> ffff88062ddeac00, xid <0x6a5>, remaining 131072, lso_max <0x10000> >>> [12378.402440] ft_queue_data_in: Failed to send frame >>> ffff88062ddeac00, xid <0x6a5>, remaining 65536, lso_max <0x10000> >>> [12378.402444] ft_queue_data_in: Failed to send frame >>> ffff88062ddeac00, xid <0x6a5>, remaining 0, lso_max <0x10000> >>> [13049.224513] ft_queue_data_in: Failed to send frame >>> ffff880614588c00, xid <0xd2f>, remaining 196608, lso_max <0x10000> >>> [13049.224524] ft_queue_data_in: Failed to send frame >>> ffff880614588c00, xid <0xd2f>, remaining 131072, lso_max <0x10000> >>> [13049.224528] ft_queue_data_in: Failed to send frame >>> ffff880614588c00, xid <0xd2f>, remaining 65536, lso_max <0x10000> >>> [13049.224532] ft_queue_data_in: Failed to send frame >>> ffff880614588c00, xid <0xd2f>, remaining 0, lso_max <0x10000> >>> [13052.511306] ft_queue_data_in: Failed to send frame >>> ffff88062d49f000, xid <0x8ae>, remaining 196608, lso_max <0x10000> >>> [13052.511313] ft_queue_data_in: Failed to send frame >>> ffff88062d49f000, xid <0x8ae>, remaining 131072, lso_max <0x10000> >>> [13052.511317] ft_queue_data_in: Failed to send frame >>> ffff88062d49f000, xid <0x8ae>, remaining 65536, lso_max <0x10000> >>> [13052.511321] ft_queue_data_in: Failed to send frame >>> ffff88062d49f000, xid <0x8ae>, remaining 0, lso_max <0x10000> >>> [13087.976748] ft_queue_data_in: Failed to send frame >>> ffff88031afc9c00, xid <0x96b>, remaining 458752, lso_max <0x10000> >>> [13087.998453] ft_queue_data_in: Failed to send frame >>> ffff88032c881200, xid <0xb23>, remaining 458752, lso_max <0x10000> >>> [13087.998459] ft_queue_data_in: Failed to send frame >>> ffff88032c881200, xid <0xb23>, remaining 393216, lso_max <0x10000> >>> [13087.998463] ft_queue_data_in: Failed to send frame >>> ffff88032c881200, xid <0xb23>, remaining 327680, lso_max <0x10000> >>> [13087.998467] ft_queue_data_in: Failed to send frame >>> ffff88032c881200, xid <0xb23>, remaining 262144, lso_max <0x10000> >>> [13087.998470] ft_queue_data_in: Failed to send frame >>> ffff88032c881200, xid <0xb23>, remaining 196608, lso_max <0x10000> >>> [13087.998474] ft_queue_data_in: Failed to send frame >>> ffff88032c881200, xid <0xb23>, remaining 131072, lso_max <0x10000> >>> [13087.998478] ft_queue_data_in: Failed to send frame >>> ffff88032c881200, xid <0xb23>, remaining 65536, lso_max <0x10000> >>> [13087.998482] ft_queue_data_in: Failed to send frame >>> ffff88032c881200, xid <0xb23>, remaining 0, lso_max <0x10000> >>> [13119.177286] ft_queue_data_in: Failed to send frame >>> ffff88062dff7400, xid <0xfcf>, remaining 458752, lso_max <0x10000> >>> [13119.177297] ft_queue_data_in: Failed to send frame >>> ffff88062dff7400, xid <0xfcf>, remaining 393216, lso_max <0x10000> >>> [13119.177302] ft_queue_data_in: Failed to send frame >>> ffff88062dff7400, xid <0xfcf>, remaining 327680, lso_max <0x10000> >>> [13119.177307] ft_queue_data_in: Failed to send frame >>> ffff88062dff7400, xid <0xfcf>, remaining 262144, lso_max <0x10000> >>> [13119.177311] ft_queue_data_in: Failed to send frame >>> ffff88062dff7400, xid <0xfcf>, remaining 196608, lso_max <0x10000> >>> [13119.177316] ft_queue_data_in: Failed to send frame >>> ffff88062dff7400, xid <0xfcf>, remaining 131072, lso_max <0x10000> >>> [13119.177321] ft_queue_data_in: Failed to send frame >>> ffff88062dff7400, xid <0xfcf>, remaining 65536, lso_max <0x10000> >>> [13119.177325] ft_queue_data_in: Failed to send frame >>> ffff88062dff7400, xid <0xfcf>, remaining 0, lso_max <0x10000> >>> [13122.335322] ------------[ cut here ]------------ >>> [13122.335336] WARNING: CPU: 6 PID: 2165 at >>> include/scsi/fc_frame.h:173 fcoe_percpu_receive_thread+0x507/0x53c >>> [fcoe]() >>> [13122.335338] Modules linked in: async_memcpy async_xor xor async_tx >>> fcoe libfcoe tcm_fc libfc scsi_transport_fc scsi_tgt target_core_pscsi >>> target_core_file target_core_iblock iscsi_target_mod target_core_mod >>> 8021q garp mrp bridge stp llc iTCO_wdt gpio_ich iTCO_vendor_support >>> coretemp kvm_intel kvm crc32c_intel microcode serio_raw i2c_i801 >>> lpc_ich mfd_core ses enclosure i7core_edac ioatdma edac_core shpchp >>> acpi_cpufreq nfsd auth_rpcgss nfs_acl lockd sunrpc radeon >>> drm_kms_helper ttm drm ixgbe igb ata_generic mdio pata_acpi ptp >>> pata_jmicron pps_core i2c_algo_bit aacraid dca i2c_core [last >>> unloaded: vd] >>> [13122.335390] CPU: 6 PID: 2165 Comm: fcoethread/6 Tainted: GF >>> O 3.13.10-200.zbfcoepatch.fc20.x86_64 #1 >>> [13122.335392] Hardware name: Supermicro X8DTN/X8DTN, BIOS 2.1c 10/28/2011 >>> [13122.335394] 0000000000000009 ffff88062b04bdd0 ffffffff81687eac >>> 0000000000000000 >>> [13122.335400] ffff88062b04be08 ffffffff8106d4dd ffffe8ffffc41748 >>> ffff88062a444700 >>> [13122.335404] ffff8800b7e926e8 0000000000000002 ffff88062b04be88 >>> ffff88062b04be18 >>> [13122.335408] Call Trace: >>> [13122.335419] [<ffffffff81687eac>] dump_stack+0x45/0x56 >>> [13122.335426] [<ffffffff8106d4dd>] warn_slowpath_common+0x7d/0xa0 >>> [13122.335430] [<ffffffff8106d5ba>] warn_slowpath_null+0x1a/0x20 >>> [13122.335435] [<ffffffffa0651517>] >>> fcoe_percpu_receive_thread+0x507/0x53c [fcoe] >>> [13122.335440] [<ffffffffa0651010>] ? fcoe_set_port_id+0x50/0x50 [fcoe] >>> [13122.335446] [<ffffffff8108f2f2>] kthread+0xd2/0xf0 >>> [13122.335450] [<ffffffff8108f220>] ? insert_kthread_work+0x40/0x40 >>> [13122.335458] [<ffffffff81696dbc>] ret_from_fork+0x7c/0xb0 >>> [13122.335461] [<ffffffff8108f220>] ? insert_kthread_work+0x40/0x40 >>> [13122.335464] ---[ end trace e4509e1053f499ac ]--- >>> >>> Thanks, >>> >>> Jun >>> >>> On Tue, May 20, 2014 at 11:03 AM, Nicholas A. Bellinger >>> <nab@xxxxxxxxxxxxxxx> wrote: >>> > On Mon, 2014-05-19 at 17:29 -0700, Jun Wu wrote: >>> >> Hi Nicholas, >>> >> >>> >> We downloaded the source of our running kernel (3.13.10-200) and >>> >> applied your percpu-ida pre-allocation regression fix, then compiled >>> >> and installed the kernel. I repeated the same test three times, >>> >> running 10 fio sessions to 10 drives on the target through fcoe vn2vn. >>> >> In the first two tests, the target machine hung with the following >>> >> messages: >>> >> >>> >> 15231 May 19 11:49:27 poc1 kernel: [ 1073.783229] ft_queue_data_in: >>> >> Failed to send frame ffff880c0b188200, xid <0x2a5>, remaining 196608, >>> >> lso_max <0x10000> >>> >> 15232 May 19 11:49:27 poc1 kernel: [ 1073.783238] ft_queue_data_in: >>> >> Failed to send frame ffff880c0b188200, xid <0x2a5>, remaining 131072, >>> >> lso_max <0x10000> >>> >> 15233 May 19 11:49:27 poc1 kernel: [ 1073.783242] ft_queue_data_in: >>> >> Failed to send frame ffff880c0b188200, xid <0x2a5>, remaining 65536, >>> >> lso_max <0x10000> >>> >> 15234 May 19 11:49:27 poc1 kernel: [ 1073.783245] ft_queue_data_in: >>> >> Failed to send frame ffff880c0b188200, xid <0x2a5>, remaining 0, >>> >> lso_max <0x10000> >>> >> 15235 May 19 11:49:30 poc1 kernel: [ 1076.907061] ft_queue_data_in: >>> >> Failed to send frame ffff880c1d1df000, xid <0x305>, remaining 196608, >>> >> lso_max <0x10000> >>> >> 15236 May 19 11:49:30 poc1 kernel: [ 1076.907068] ft_queue_data_in: >>> >> Failed to send frame ffff880c1d1df000, xid <0x305>, remaining 131072, >>> >> lso_max <0x10000> >>> >> 15237 May 19 11:49:30 poc1 kernel: [ 1076.907073] ft_queue_data_in: >>> >> Failed to send frame ffff880c1d1df000, xid <0x305>, remaining 65536, >>> >> lso_max <0x10000> >>> >> 15238 May 19 11:49:30 poc1 kernel: [ 1076.907077] ft_queue_data_in: >>> >> Failed to send frame ffff880c1d1df000, xid <0x305>, remaining 0, >>> >> lso_max <0x10000> >>> >> 15239 May 19 11:50:01 poc1 kernel: [ 1107.918910] ft_queue_data_in: >>> >> Failed to send frame ffff88060cd40800, xid <0x3cb>, remaining 458752, >>> >> lso_max <0x10000> >>> >> 15240 May 19 11:50:01 poc1 kernel: [ 1107.918918] ft_queue_data_in: >>> >> Failed to send frame ffff88060cd40800, xid <0x3cb>, remaining 393216, >>> >> lso_max <0x10000> >>> >> 15241 May 19 11:50:01 poc1 kernel: [ 1107.918922] ft_queue_data_in: >>> >> Failed to send frame ffff88060cd40800, xid <0x3cb>, remaining 327680, >>> >> lso_max <0x10000> >>> >> 15242 May 19 11:50:01 poc1 kernel: [ 1107.918925] ft_queue_data_in: >>> >> Failed to send frame ffff88060cd40800, xid <0x3cb>, remaining 262144, >>> >> lso_max <0x10000> >>> >> 15243 May 19 11:50:01 poc1 kernel: [ 1107.918929] ft_queue_data_in: >>> >> Failed to send frame ffff88060cd40800, xid <0x3cb>, remaining 196608, >>> >> lso_max <0x10000> >>> >> 15244 May 19 11:50:01 poc1 kernel: [ 1107.918932] ft_queue_data_in: >>> >> Failed to send frame ffff88060cd40800, xid <0x3cb>, remaining 131072, >>> >> lso_max <0x10000> >>> >> 15245 May 19 11:50:01 poc1 kernel: [ 1107.918936] ft_queue_data_in: >>> >> Failed to send frame ffff88060cd40800, xid <0x3cb>, remaining 65536, >>> >> lso_max <0x10000> >>> >> 15246 May 19 11:50:01 poc1 kernel: [ 1107.918939] ft_queue_data_in: >>> >> Failed to send frame ffff88060cd40800, xid <0x3cb>, remaining 0, >>> >> lso_max <0x10000> >>> >> 15247 May 19 11:50:05 poc1 kernel: [ 1111.450900] ft_queue_data_in: >>> >> Failed to send frame ffff880c0b24ca00, xid <0xea6>, remaining 196608, >>> >> lso_max <0x10000> >>> >> 15248 May 19 11:50:05 poc1 kernel: [ 1111.450908] ft_queue_data_in: >>> >> Failed to send frame ffff880c0b24ca00, xid <0xea6>, remaining 131072, >>> >> lso_max <0x10000> >>> >> 15249 May 19 11:51:12 poc1 kernel: [ 1178.698434] ft_queue_data_in: 6 >>> >> callbacks suppressed >>> >> 15250 May 19 11:51:12 poc1 kernel: [ 1178.698440] ft_queue_data_in: >>> >> Failed to send frame ffff88060ba97400, xid <0xb8a>, remaining 458752, >>> >> lso_max <0x10000> >>> >> 15251 May 19 11:51:12 poc1 kernel: [ 1178.698446] ft_queue_data_in: >>> >> Failed to send frame ffff88060ba97400, xid <0xb8a>, remaining 393216, >>> >> lso_max <0x10000> >>> >> 15252 May 19 11:51:12 poc1 kernel: [ 1178.698449] ft_queue_data_in: >>> >> Failed to send frame ffff88060ba97400, xid <0xb8a>, remaining 327680, >>> >> lso_max <0x10000> >>> >> 15253 May 19 11:51:12 poc1 kernel: [ 1178.698453] ft_queue_data_in: >>> >> Failed to send frame ffff88060ba97400, xid <0xb8a>, remaining 262144, >>> >> lso_max <0x10000> >>> >> 15254 May 19 11:51:12 poc1 kernel: [ 1178.698456] ft_queue_data_in: >>> >> Failed to send frame ffff88060ba97400, xid <0xb8a>, remaining 196608, >>> >> lso_max <0x10000> >>> >> 15255 May 19 11:51:12 poc1 kernel: [ 1178.698460] ft_queue_data_in: >>> >> Failed to send frame ffff88060ba97400, xid <0xb8a>, remaining 131072, >>> >> lso_max <0x10000> >>> >> 15256 May 19 11:51:12 poc1 kernel: [ 1178.698463] ft_queue_data_in: >>> >> Failed to send frame ffff88060ba97400, xid <0xb8a>, remaining 65536, >>> >> lso_max <0x10000> >>> >> 15257 May 19 11:51:12 poc1 kernel: [ 1178.698467] ft_queue_data_in: >>> >> Failed to send frame ffff88060ba97400, xid <0xb8a>, remaining 0, >>> >> lso_max <0x10000> >>> >> >>> > >>> > The call into lport->tt.seq_send() libfc code is failing to send >>> > outgoing solicited data-in. From the output, note the LSO (large >>> > segment offload aka TCP segment offload) feature has been enabled by the >>> > underlying NIC hardware. >>> > >>> > So in order to isolate possible issues, I'd recommend: >>> > >>> > - Disabling hardware offloads on both initiator and target sides (LRO + >>> > LSO) using ethtool -K >>> > - Disabling any jumbo frames settings on either side >>> > >>> > Is there any other non standard network and/or switch settings that are >>> > in place..? Also, please confirm what your NIC + switch setup looks >>> > like. >>> > >>> > Rob & Open-FCoE folks, is there anything else to take into consideration >>> > here..? >>> > >>> >> >>> >> I didn't see the previous message "unable to handle kernel NULL >>> >> pointer dereference at 0000000000000048". So it must have been fixed >>> >> by your change. >>> >> >>> > >>> > Thanks for confirming that bit. >>> > >>> > --nab >>> > >>> _______________________________________________ >>> fcoe-devel mailing list >>> fcoe-devel@xxxxxxxxxxxxx >>> http://lists.open-fcoe.org/mailman/listinfo/fcoe-devel >> >> -- To unsubscribe from this list: send the line "unsubscribe target-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
