I enabled the Tx offload and set the debug_loggings as suggested. A few minutes run generated a 1.5GB messages file. The file has repeated patterns of the following: 18630713 May 21 11:01:53 poc2 kernel: [ 1528.182334] host7: xid b48: f_ctl 800000 seq 1 18630714 May 21 11:01:53 poc2 kernel: [ 1528.182345] host7: xid b48: f_ctl 880008 seq 2 18630715 May 21 11:01:53 poc2 kernel: [ 1528.182601] host7: xid 38e: f_ctl 800000 seq 1 18630716 May 21 11:01:53 poc2 kernel: [ 1528.182621] host7: xid 38e: f_ctl 880008 seq 2 18630717 May 21 11:01:53 poc2 kernel: [ 1528.182771] host7: xid 74a: f_ctl 800000 seq 1 18630718 May 21 11:01:53 poc2 kernel: [ 1528.182785] host7: xid 74a: f_ctl 880008 seq 2 18630719 May 21 11:01:53 poc2 kernel: [ 1528.183161] host7: xid e4f: f_ctl 800000 seq 1 18630720 May 21 11:01:53 poc2 kernel: [ 1528.183181] host7: xid e4f: f_ctl 880008 seq 2 18630721 May 21 11:01:53 poc2 kernel: [ 1528.184285] host7: xid 666: f_ctl 800000 seq 1 18630722 May 21 11:01:53 poc2 kernel: [ 1528.184301] host7: xid 666: f_ctl 880008 seq 2 18630723 May 21 11:01:53 poc2 kernel: [ 1528.184550] host7: xid c20: f_ctl 800000 seq 1 18630724 May 21 11:01:53 poc2 kernel: [ 1528.184589] host7: xid c20: f_ctl 880008 seq 2 18630725 May 21 11:01:53 poc2 kernel: [ 1528.185198] host7: xid 607: f_ctl 800000 seq 1 18630726 May 21 11:01:53 poc2 kernel: [ 1528.185213] host7: xid 607: f_ctl 880008 seq 2 18630727 May 21 11:01:53 poc2 kernel: [ 1528.185659] host7: xid 925: f_ctl 800000 seq 1 18630728 May 21 11:01:53 poc2 kernel: [ 1528.185662] host7: xid b48: f_ctl 800000 seq 1 18630729 May 21 11:01:53 poc2 kernel: [ 1528.185672] host7: xid b48: f_ctl 880008 seq 2 18630730 May 21 11:01:53 poc2 kernel: [ 1528.185680] host7: xid 925: f_ctl 880008 seq 2 18630731 May 21 11:01:53 poc2 kernel: [ 1528.185751] host7: xid b61: f_ctl 800000 seq 1 18630732 May 21 11:01:53 poc2 kernel: [ 1528.185765] host7: xid b61: f_ctl 880008 seq 2 18630733 May 21 11:01:53 poc2 kernel: [ 1528.186413] host7: xid 829: f_ctl 800000 seq 1 18630734 May 21 11:01:53 poc2 kernel: [ 1528.186425] host7: xid 829: f_ctl 880008 seq 2 18630735 May 21 11:01:53 poc2 kernel: [ 1528.186785] host7: xid 6e4: f_ctl 800000 seq 1 18630736 May 21 11:01:53 poc2 kernel: [ 1528.186817] host7: xid 6e4: f_ctl 880008 seq 2 18630737 May 21 11:01:53 poc2 kernel: [ 1528.186932] host7: xid dab: f_ctl 800000 seq 1 18630738 May 21 11:01:53 poc2 kernel: [ 1528.186946] host7: xid dab: f_ctl 880008 seq 2 18630739 May 21 11:01:53 poc2 kernel: [ 1528.187907] host7: xid 8ac: f_ctl 800000 seq 1 18630740 May 21 11:01:53 poc2 kernel: [ 1528.187920] host7: xid 8ac: f_ctl 880008 seq 2 18630741 May 21 11:01:53 poc2 kernel: [ 1528.188656] host7: xid 38e: f_ctl 800000 seq 1 18630742 May 21 11:01:53 poc2 kernel: [ 1528.188675] host7: xid 38e: f_ctl 880008 seq 2 18630743 May 21 11:01:53 poc2 kernel: [ 1528.188889] host7: xid b61: f_ctl 800000 seq 1 18630744 May 21 11:01:53 poc2 kernel: [ 1528.188899] host7: xid b61: f_ctl 880008 seq 2 18630745 May 21 11:01:53 poc2 kernel: [ 1528.189281] host7: xid 88d: f_ctl 800000 seq 1 18630746 May 21 11:01:53 poc2 kernel: [ 1528.189301] host7: xid 88d: f_ctl 880008 seq 2 18630747 May 21 11:01:53 poc2 kernel: [ 1528.189378] host7: xid c20: f_ctl 800000 seq 1 18630748 May 21 11:01:53 poc2 kernel: [ 1528.189392] host7: xid c20: f_ctl 880008 seq 2 18630749 May 21 11:01:53 poc2 kernel: [ 1528.189836] host7: xid 862: f_ctl 800000 seq 1 18630750 May 21 11:01:53 poc2 kernel: [ 1528.189850] host7: xid 862: f_ctl 880008 seq 2 18630751 May 21 11:01:53 poc2 kernel: [ 1528.191740] host7: xid 6e4: Exchange timer armed : 0 msecs 18630752 May 21 11:01:53 poc2 kernel: [ 1528.191747] host7: xid 6e4: f_ctl 800000 seq 1 18630753 May 21 11:01:53 poc2 kernel: [ 1528.191756] host7: xid 6e4: f_ctl 800000 seq 2 18630754 May 21 11:01:53 poc2 kernel: [ 1528.191763] host7: xid 6e4: Exchange timed out 18630755 May 21 11:01:53 poc2 kernel: [ 1528.191777] ft_queue_data_in: Failed to send frame ffff8805bf245e00, xid <0x6e4>, remaining 458752, lso_max <0x10000> 18630756 May 21 11:01:53 poc2 kernel: [ 1528.191782] ft_queue_data_in: Failed to send frame ffff8805bf245e00, xid <0x6e4>, remaining 393216, lso_max <0x10000> 18630757 May 21 11:01:53 poc2 kernel: [ 1528.191786] ft_queue_data_in: Failed to send frame ffff8805bf245e00, xid <0x6e4>, remaining 327680, lso_max <0x10000> 18630758 May 21 11:01:53 poc2 kernel: [ 1528.191790] ft_queue_data_in: Failed to send frame ffff8805bf245e00, xid <0x6e4>, remaining 262144, lso_max <0x10000> 18630759 May 21 11:01:53 poc2 kernel: [ 1528.191794] ft_queue_data_in: Failed to send frame ffff8805bf245e00, xid <0x6e4>, remaining 196608, lso_max <0x10000> 18630760 May 21 11:01:53 poc2 kernel: [ 1528.191798] ft_queue_data_in: Failed to send frame ffff8805bf245e00, xid <0x6e4>, remaining 131072, lso_max <0x10000> 18630761 May 21 11:01:53 poc2 kernel: [ 1528.191801] ft_queue_data_in: Failed to send frame ffff8805bf245e00, xid <0x6e4>, remaining 65536, lso_max <0x10000> 18630762 May 21 11:01:53 poc2 kernel: [ 1528.191805] ft_queue_data_in: Failed to send frame ffff8805bf245e00, xid <0x6e4>, remaining 0, lso_max <0x10000> 18630763 May 21 11:01:53 poc2 kernel: [ 1528.192163] host7: xid b48: f_ctl 800000 seq 1 18630764 May 21 11:01:53 poc2 kernel: [ 1528.192166] host7: xid 607: f_ctl 800000 seq 1 18630765 May 21 11:01:53 poc2 kernel: [ 1528.192176] host7: xid 607: f_ctl 880008 seq 2 18630766 May 21 11:01:53 poc2 kernel: [ 1528.192180] host7: xid b48: f_ctl 880008 seq 2 18630767 May 21 11:01:53 poc2 kernel: [ 1528.192266] host7: xid 666: f_ctl 800000 seq 1 Above is the first time ft_queue_data_in message shows up in the file. Here is another instance: 18631537 May 21 11:01:53 poc2 kernel: [ 1528.333876] host7: xid 74a: f_ctl 800000 seq 1 18631538 May 21 11:01:53 poc2 kernel: [ 1528.333893] host7: xid 74a: f_ctl 880008 seq 2 18631539 May 21 11:01:53 poc2 kernel: [ 1528.334816] host7: xid c20: f_ctl 800000 seq 1 18631540 May 21 11:01:53 poc2 kernel: [ 1528.334834] host7: xid c20: f_ctl 880008 seq 2 18631541 May 21 11:01:53 poc2 kernel: [ 1528.334847] host7: xid b81: f_ctl 800000 seq 1 18631542 May 21 11:01:53 poc2 kernel: [ 1528.334858] host7: xid 983: f_ctl 800000 seq 1 18631543 May 21 11:01:53 poc2 kernel: [ 1528.334864] host7: xid b81: f_ctl 880008 seq 2 18631544 May 21 11:01:53 poc2 kernel: [ 1528.334881] host7: xid 983: f_ctl 880008 seq 2 18631545 May 21 11:01:53 poc2 kernel: [ 1528.334972] host7: xid 686: f_ctl 800000 seq 1 18631546 May 21 11:01:53 poc2 kernel: [ 1528.334985] host7: xid 686: f_ctl 880008 seq 2 18631547 May 21 11:01:53 poc2 kernel: [ 1528.335036] host7: xid 704: f_ctl 800000 seq 1 18631548 May 21 11:01:53 poc2 kernel: [ 1528.335052] host7: xid 704: f_ctl 880008 seq 2 18631549 May 21 11:01:53 poc2 kernel: [ 1528.335078] host7: xid 627: f_ctl 800000 seq 1 18631550 May 21 11:01:53 poc2 kernel: [ 1528.335088] host7: xid 627: f_ctl 880008 seq 2 18631551 May 21 11:01:53 poc2 kernel: [ 1528.335202] host7: xid b48: f_ctl 800000 seq 1 18631552 May 21 11:01:53 poc2 kernel: [ 1528.335214] host7: xid b48: f_ctl 880008 seq 2 18631553 May 21 11:01:53 poc2 kernel: [ 1528.335381] host7: xid 74a: Exchange timer armed : 0 msecs 18631554 May 21 11:01:53 poc2 kernel: [ 1528.335386] host7: xid 74a: f_ctl 800000 seq 1 18631555 May 21 11:01:53 poc2 kernel: [ 1528.335388] host7: xid 74a: Exchange timed out 18631556 May 21 11:01:53 poc2 kernel: [ 1528.335672] host7: xid 869: f_ctl 800000 seq 1 18631557 May 21 11:01:53 poc2 kernel: [ 1528.335688] host7: xid 869: f_ctl 880008 seq 2 18631558 May 21 11:01:53 poc2 kernel: [ 1528.336213] host7: xid 965: f_ctl 800000 seq 1 18631559 May 21 11:01:53 poc2 kernel: [ 1528.336232] host7: xid 965: f_ctl 880008 seq 2 18631560 May 21 11:01:53 poc2 kernel: [ 1528.336477] host7: xid 74a: f_ctl 800000 seq 2 18631561 May 21 11:01:53 poc2 kernel: [ 1528.336482] ft_queue_data_in: Failed to send frame ffff8802e25c1e00, xid <0x74a>, remaining 196608, lso_max <0x10000> 18631562 May 21 11:01:53 poc2 kernel: [ 1528.336486] ft_queue_data_in: Failed to send frame ffff8802e25c1e00, xid <0x74a>, remaining 131072, lso_max <0x10000> 18631563 May 21 11:01:53 poc2 kernel: [ 1528.336489] host7: xid 74a: f_ctl 800000 seq 3 18631564 May 21 11:01:53 poc2 kernel: [ 1528.337645] host7: xid 86d: f_ctl 800000 seq 1 18631565 May 21 11:01:53 poc2 kernel: [ 1528.337759] host7: xid 86d: f_ctl 880008 seq 2 18631566 May 21 11:01:53 poc2 kernel: [ 1528.337827] host7: xid 44e: f_ctl 800000 seq 1 18631567 May 21 11:01:53 poc2 kernel: [ 1528.337846] host7: xid 44e: f_ctl 880008 seq 2 18631568 May 21 11:01:53 poc2 kernel: [ 1528.340521] host7: xid 8c2: Exchange timer armed : 0 msecs 18631569 May 21 11:01:53 poc2 kernel: [ 1528.340526] host7: xid 8c2: f_ctl 800000 seq 1 18631570 May 21 11:01:53 poc2 kernel: [ 1528.340667] host7: xid 8c2: Exchange timed out 18631571 May 21 11:01:53 poc2 kernel: [ 1528.341064] host7: xid 983: f_ctl 800000 seq 1 18631572 May 21 11:01:53 poc2 kernel: [ 1528.341087] host7: xid 983: f_ctl 880008 seq 2 18631573 May 21 11:01:53 poc2 kernel: [ 1528.341286] host7: xid b48: f_ctl 800000 seq 1 18631574 May 21 11:01:53 poc2 kernel: [ 1528.341306] host7: xid b48: f_ctl 880008 seq 2 18631575 May 21 11:01:53 poc2 kernel: [ 1528.341522] host7: xid 869: Exchange timer armed : 0 msecs 18631576 May 21 11:01:53 poc2 kernel: [ 1528.341528] host7: xid 869: f_ctl 800000 seq 1 18631577 May 21 11:01:53 poc2 kernel: [ 1528.341539] host7: xid 869: Exchange timed out 18631578 May 21 11:01:53 poc2 kernel: [ 1528.341966] host7: xid 965: f_ctl 800000 seq 1 18631579 May 21 11:01:53 poc2 kernel: [ 1528.341979] host7: xid 965: f_ctl 880008 seq 2 18631580 May 21 11:01:53 poc2 kernel: [ 1528.342450] host7: xid 627: f_ctl 800000 seq 1 18631581 May 21 11:01:53 poc2 kernel: [ 1528.342467] host7: xid 627: f_ctl 880008 seq 2 18631582 May 21 11:01:53 poc2 kernel: [ 1528.342945] host7: xid 70a: f_ctl 800000 seq 1 18631583 May 21 11:01:53 poc2 kernel: [ 1528.342959] host7: xid 70a: f_ctl 880008 seq 2 After stripping out the repeated "host7: xid xxx: f_ctl 800000 seq x" messages, we have: 18630507:May 21 11:01:53 poc2 kernel: [ 1528.148314] host7: xid d6b: Exchange timer armed : 0 msecs 18630509:May 21 11:01:53 poc2 kernel: [ 1528.148357] host7: xid d6b: Exchange timed out 18630642:May 21 11:01:53 poc2 kernel: [ 1528.169143] host7: xid 88c: Exchange timer armed : 0 msecs 18630644:May 21 11:01:53 poc2 kernel: [ 1528.169197] host7: xid 88c: Exchange timed out 18630751:May 21 11:01:53 poc2 kernel: [ 1528.191740] host7: xid 6e4: Exchange timer armed : 0 msecs 18630754:May 21 11:01:53 poc2 kernel: [ 1528.191763] host7: xid 6e4: Exchange timed out 18630755:May 21 11:01:53 poc2 kernel: [ 1528.191777] ft_queue_data_in: Failed to send frame ffff8805bf245e00, xid <0x6e4>, remaining 458752, lso_max <0x10000> 18630756:May 21 11:01:53 poc2 kernel: [ 1528.191782] ft_queue_data_in: Failed to send frame ffff8805bf245e00, xid <0x6e4>, remaining 393216, lso_max <0x10000> 18630757:May 21 11:01:53 poc2 kernel: [ 1528.191786] ft_queue_data_in: Failed to send frame ffff8805bf245e00, xid <0x6e4>, remaining 327680, lso_max <0x10000> 18630758:May 21 11:01:53 poc2 kernel: [ 1528.191790] ft_queue_data_in: Failed to send frame ffff8805bf245e00, xid <0x6e4>, remaining 262144, lso_max <0x10000> 18630759:May 21 11:01:53 poc2 kernel: [ 1528.191794] ft_queue_data_in: Failed to send frame ffff8805bf245e00, xid <0x6e4>, remaining 196608, lso_max <0x10000> 18630760:May 21 11:01:53 poc2 kernel: [ 1528.191798] ft_queue_data_in: Failed to send frame ffff8805bf245e00, xid <0x6e4>, remaining 131072, lso_max <0x10000> 18630761:May 21 11:01:53 poc2 kernel: [ 1528.191801] ft_queue_data_in: Failed to send frame ffff8805bf245e00, xid <0x6e4>, remaining 65536, lso_max <0x10000> 18630762:May 21 11:01:53 poc2 kernel: [ 1528.191805] ft_queue_data_in: Failed to send frame ffff8805bf245e00, xid <0x6e4>, remaining 0, lso_max <0x10000> 18630777:May 21 11:01:53 poc2 kernel: [ 1528.195937] host7: xid 666: Exchange timer armed : 0 msecs 18630779:May 21 11:01:53 poc2 kernel: [ 1528.195983] host7: xid 666: Exchange timed out 18630780:May 21 11:01:53 poc2 kernel: [ 1528.196336] host7: xid 862: Exchange timer armed : 0 msecs 18630782:May 21 11:01:53 poc2 kernel: [ 1528.196348] host7: xid 862: Exchange timed out 18630879:May 21 11:01:53 poc2 kernel: [ 1528.211080] host7: xid b61: Exchange timer armed : 0 msecs 18630881:May 21 11:01:53 poc2 kernel: [ 1528.211137] host7: xid b61: Exchange timed out 18630898:May 21 11:01:53 poc2 kernel: [ 1528.214253] host7: xid 38e: Exchange timer armed : 0 msecs 18630900:May 21 11:01:53 poc2 kernel: [ 1528.214284] host7: xid 38e: Exchange timed out 18631007:May 21 11:01:53 poc2 kernel: [ 1528.236137] host7: xid 88d: Exchange timer armed : 0 msecs 18631009:May 21 11:01:53 poc2 kernel: [ 1528.236172] host7: xid 88d: Exchange timed out 18631176:May 21 11:01:53 poc2 kernel: [ 1528.264223] host7: xid 607: Exchange timer armed : 0 msecs 18631178:May 21 11:01:53 poc2 kernel: [ 1528.264239] host7: xid 607: Exchange timed out 18631185:May 21 11:01:53 poc2 kernel: [ 1528.266310] host7: xid 82d: Exchange timer armed : 0 msecs 18631187:May 21 11:01:53 poc2 kernel: [ 1528.266351] host7: xid 82d: Exchange timed out 18631226:May 21 11:01:53 poc2 kernel: [ 1528.275452] host7: xid 8a2: Exchange timer armed : 0 msecs 18631228:May 21 11:01:53 poc2 kernel: [ 1528.275463] host7: xid 8a2: Exchange timed out 18631553:May 21 11:01:53 poc2 kernel: [ 1528.335381] host7: xid 74a: Exchange timer armed : 0 msecs 18631555:May 21 11:01:53 poc2 kernel: [ 1528.335388] host7: xid 74a: Exchange timed out 18631561:May 21 11:01:53 poc2 kernel: [ 1528.336482] ft_queue_data_in: Failed to send frame ffff8802e25c1e00, xid <0x74a>, remaining 196608, lso_max <0x10000> 18631562:May 21 11:01:53 poc2 kernel: [ 1528.336486] ft_queue_data_in: Failed to send frame ffff8802e25c1e00, xid <0x74a>, remaining 131072, lso_max <0x10000> 18631568:May 21 11:01:53 poc2 kernel: [ 1528.340521] host7: xid 8c2: Exchange timer armed : 0 msecs 18631570:May 21 11:01:53 poc2 kernel: [ 1528.340667] host7: xid 8c2: Exchange timed out 18631575:May 21 11:01:53 poc2 kernel: [ 1528.341522] host7: xid 869: Exchange timer armed : 0 msecs 18631577:May 21 11:01:53 poc2 kernel: [ 1528.341539] host7: xid 869: Exchange timed out 18631660:May 21 11:01:53 poc2 kernel: [ 1528.356897] host7: xid e4f: Exchange timer armed : 0 msecs 18631662:May 21 11:01:53 poc2 kernel: [ 1528.356975] host7: xid e4f: Exchange timed out 18631825:May 21 11:01:53 poc2 kernel: [ 1528.398431] host7: xid 965: Exchange timer armed : 0 msecs 18631827:May 21 11:01:53 poc2 kernel: [ 1528.398509] host7: xid 965: Exchange timed out 18631844:May 21 11:01:53 poc2 kernel: [ 1528.401772] host7: xid 44e: Exchange timer armed : 0 msecs 18631846:May 21 11:01:53 poc2 kernel: [ 1528.401850] host7: xid 44e: Exchange timed out 18631859:May 21 11:01:53 poc2 kernel: [ 1528.404996] host7: xid 704: Exchange timer armed : 0 msecs 18631861:May 21 11:01:53 poc2 kernel: [ 1528.405043] host7: xid 704: Exchange timed out 18631876:May 21 11:01:53 poc2 kernel: [ 1528.412608] host7: xid 86d: Exchange timer armed : 0 msecs 18631878:May 21 11:01:53 poc2 kernel: [ 1528.412619] host7: xid 86d: Exchange timed out 18631919:May 21 11:01:53 poc2 kernel: [ 1528.431939] host7: xid 686: Exchange timer armed : 0 msecs 18631921:May 21 11:01:53 poc2 kernel: [ 1528.432000] host7: xid 686: Exchange timed out 18631950:May 21 11:01:53 poc2 kernel: [ 1528.441542] host7: xid def: Exchange timer armed : 0 msecs 18631952:May 21 11:01:53 poc2 kernel: [ 1528.441573] host7: xid def: Exchange timed out 18631967:May 21 11:01:53 poc2 kernel: [ 1528.447848] host7: xid 3ce: Exchange timer armed : 0 msecs 18631969:May 21 11:01:53 poc2 kernel: [ 1528.447863] host7: xid 3ce: Exchange timed out 18632008:May 21 11:01:53 poc2 kernel: [ 1528.453868] host7: xid 627: Exchange timer armed : 0 msecs 18632010:May 21 11:01:53 poc2 kernel: [ 1528.453879] host7: xid 627: Exchange timed out 18655343:May 21 11:01:56 poc2 kernel: [ 1531.747863] host7: xid 48e: Exchange timer armed : 0 msecs 18655345:May 21 11:01:56 poc2 kernel: [ 1531.747879] host7: xid 48e: Exchange timed out 18655414:May 21 11:01:56 poc2 kernel: [ 1531.758175] host7: xid 667: Exchange timer armed : 0 msecs 18655416:May 21 11:01:56 poc2 kernel: [ 1531.758204] host7: xid 667: Exchange timed out 18655533:May 21 11:01:56 poc2 kernel: [ 1531.776753] host7: xid 6c6: Exchange timer armed : 0 msecs 18655535:May 21 11:01:56 poc2 kernel: [ 1531.776765] host7: xid 6c6: Exchange timed out 18655832:May 21 11:01:56 poc2 kernel: [ 1531.818184] host7: xid 647: Exchange timer armed : 0 msecs 18655834:May 21 11:01:56 poc2 kernel: [ 1531.818227] host7: xid 647: Exchange timed out 18655893:May 21 11:01:56 poc2 kernel: [ 1531.825287] host7: xid 46e: Exchange timer armed : 0 msecs 18655895:May 21 11:01:56 poc2 kernel: [ 1531.825344] host7: xid 46e: Exchange timed out 18656116:May 21 11:01:56 poc2 kernel: [ 1531.863739] host7: xid dab: Exchange timer armed : 0 msecs 18656118:May 21 11:01:56 poc2 kernel: [ 1531.863765] host7: xid dab: Exchange timed out 18656124:May 21 11:01:56 poc2 kernel: [ 1531.865740] host7: xid 8ac: Exchange timer armed : 0 msecs 18656129:May 21 11:01:56 poc2 kernel: [ 1531.865800] host7: xid 8ac: Exchange timed out 18656130:May 21 11:01:56 poc2 kernel: [ 1531.865813] host7: xid 5e7: Exchange timer armed : 2000 msecs 18656267:May 21 11:01:56 poc2 kernel: [ 1531.891151] host7: xid e6f: Exchange timer armed : 0 msecs 18656269:May 21 11:01:56 poc2 kernel: [ 1531.891177] host7: xid e6f: Exchange timed out 18656320:May 21 11:01:56 poc2 kernel: [ 1531.899587] host7: xid 82c: Exchange timer armed : 0 msecs 18656322:May 21 11:01:56 poc2 kernel: [ 1531.899606] host7: xid 82c: Exchange timed out 18656355:May 21 11:01:56 poc2 kernel: [ 1531.906904] host7: xid 646: Exchange timer armed : 0 msecs 18656357:May 21 11:01:56 poc2 kernel: [ 1531.906952] host7: xid 646: Exchange timed out 18656608:May 21 11:01:56 poc2 kernel: [ 1531.960276] host7: xid 945: Exchange timer armed : 0 msecs 18656610:May 21 11:01:56 poc2 kernel: [ 1531.960293] host7: xid 945: Exchange timed out 18656613:May 21 11:01:56 poc2 kernel: [ 1531.960985] host7: xid 724: Exchange timer armed : 0 msecs 18656615:May 21 11:01:56 poc2 kernel: [ 1531.961002] host7: xid 724: Exchange timed out 18656634:May 21 11:01:56 poc2 kernel: [ 1531.968643] host7: xid 8ad: Exchange timer armed : 0 msecs 18656636:May 21 11:01:56 poc2 kernel: [ 1531.968660] host7: xid 8ad: Exchange timed out 18656641:May 21 11:01:56 poc2 kernel: [ 1531.974843] host7: xid 5e7: Exchange timer armed : 10000 msecs 18656642:May 21 11:01:56 poc2 kernel: [ 1531.974848] host7: xid 5e7: f_ctl 90000 seq 1 18656655:May 21 11:01:56 poc2 kernel: [ 1531.983896] host7: xid 687: Exchange timer armed : 0 msecs 18656657:May 21 11:01:56 poc2 kernel: [ 1531.983912] host7: xid 687: Exchange timed out 18656746:May 21 11:01:56 poc2 kernel: [ 1532.011935] host7: xid 5e7: Exchange timer armed : 10000 msecs 18656747:May 21 11:01:56 poc2 kernel: [ 1532.011940] host7: xid 5e7: f_ctl 90000 seq 2 18669030:May 21 11:01:58 poc2 kernel: [ 1533.870105] host7: xid 5e7: Exchange timed out 18841592:May 21 11:02:27 poc2 kernel: [ 1562.646991] host7: xid 90d: Exchange timer armed : 0 msecs 18841595:May 21 11:02:27 poc2 kernel: [ 1562.647066] host7: xid 90d: Exchange timed out 18842498:May 21 11:02:27 poc2 kernel: [ 1562.795804] host7: xid 8ac: Exchange timer armed : 0 msecs 18842500:May 21 11:02:27 poc2 kernel: [ 1562.795820] host7: xid 8ac: Exchange timed out 18842547:May 21 11:02:27 poc2 kernel: [ 1562.805335] host7: xid 94c: Exchange timer armed : 0 msecs 18842549:May 21 11:02:27 poc2 kernel: [ 1562.805351] host7: xid 94c: Exchange timed out 18842584:May 21 11:02:27 poc2 kernel: [ 1562.810550] host7: xid f2f: Exchange timer armed : 0 msecs 18842586:May 21 11:02:27 poc2 kernel: [ 1562.810605] host7: xid f2f: Exchange timed out 18842593:May 21 11:02:27 poc2 kernel: [ 1562.811735] host7: xid 7e6: Exchange timer armed : 0 msecs 18842595:May 21 11:02:27 poc2 kernel: [ 1562.811795] host7: xid 7e6: Exchange timed out 18842740:May 21 11:02:27 poc2 kernel: [ 1562.837943] host7: xid e8b: Exchange timer armed : 0 msecs 18842742:May 21 11:02:27 poc2 kernel: [ 1562.837966] host7: xid 8cd: Exchange timer armed : 0 msecs 18842743:May 21 11:02:27 poc2 kernel: [ 1562.837968] host7: xid e8b: Exchange timed out 18842745:May 21 11:02:27 poc2 kernel: [ 1562.838038] host7: xid 8cd: Exchange timed out 18842872:May 21 11:02:27 poc2 kernel: [ 1562.863661] host7: xid 824: Exchange timer armed : 0 msecs 18842874:May 21 11:02:27 poc2 kernel: [ 1562.863674] host7: xid 824: Exchange timed out 18842951:May 21 11:02:27 poc2 kernel: [ 1562.876390] host7: xid 766: Exchange timer armed : 0 msecs 18842953:May 21 11:02:27 poc2 kernel: [ 1562.876405] host7: xid 766: Exchange timed out 18842960:May 21 11:02:27 poc2 kernel: [ 1562.878444] host7: xid ca8: Exchange timer armed : 0 msecs 18842962:May 21 11:02:27 poc2 kernel: [ 1562.878470] host7: xid ca8: Exchange timed out 18843025:May 21 11:02:27 poc2 kernel: [ 1562.892335] host7: xid a25: Exchange timer armed : 0 msecs 18843027:May 21 11:02:27 poc2 kernel: [ 1562.892400] host7: xid a25: Exchange timed out 18843092:May 21 11:02:27 poc2 kernel: [ 1562.905946] host7: xid be1: Exchange timer armed : 0 msecs 18843094:May 21 11:02:27 poc2 kernel: [ 1562.905971] host7: xid be1: Exchange timed out 18843239:May 21 11:02:27 poc2 kernel: [ 1562.934586] host7: xid 52e: Exchange timer armed : 0 msecs 18843241:May 21 11:02:27 poc2 kernel: [ 1562.934601] host7: xid 52e: Exchange timed out 18843264:May 21 11:02:27 poc2 kernel: [ 1562.938141] host7: xid bc8: Exchange timer armed : 0 msecs 18843266:May 21 11:02:27 poc2 kernel: [ 1562.938164] host7: xid bc8: Exchange timed out 18843353:May 21 11:02:27 poc2 kernel: [ 1562.959150] host7: xid c21: Exchange timer armed : 0 msecs 18843356:May 21 11:02:27 poc2 kernel: [ 1562.959194] ft_queue_data_in: 10 callbacks suppressed 18843357:May 21 11:02:27 poc2 kernel: [ 1562.959196] ft_queue_data_in: Failed to send frame ffff8800ba6e6400, xid <0xc21>, remaining 458752, lso_max <0x10000> 18843358:May 21 11:02:27 poc2 kernel: [ 1562.959199] ft_queue_data_in: Failed to send frame ffff8800ba6e6400, xid <0xc21>, remaining 393216, lso_max <0x10000> 18843359:May 21 11:02:27 poc2 kernel: [ 1562.959202] ft_queue_data_in: Failed to send frame ffff8800ba6e6400, xid <0xc21>, remaining 327680, lso_max <0x10000> 18843360:May 21 11:02:27 poc2 kernel: [ 1562.959204] ft_queue_data_in: Failed to send frame ffff8800ba6e6400, xid <0xc21>, remaining 262144, lso_max <0x10000> 18843361:May 21 11:02:27 poc2 kernel: [ 1562.959208] host7: xid c21: Exchange timed out 18843362:May 21 11:02:27 poc2 kernel: [ 1562.959209] ft_queue_data_in: Failed to send frame ffff8800ba6e6400, xid <0xc21>, remaining 196608, lso_max <0x10000> 18843363:May 21 11:02:27 poc2 kernel: [ 1562.959212] ft_queue_data_in: Failed to send frame ffff8800ba6e6400, xid <0xc21>, remaining 131072, lso_max <0x10000> 18843364:May 21 11:02:27 poc2 kernel: [ 1562.959214] ft_queue_data_in: Failed to send frame ffff8800ba6e6400, xid <0xc21>, remaining 65536, lso_max <0x10000> 18843365:May 21 11:02:27 poc2 kernel: [ 1562.959217] ft_queue_data_in: Failed to send frame ffff8800ba6e6400, xid <0xc21>, remaining 0, lso_max <0x10000> 18843416:May 21 11:02:27 poc2 kernel: [ 1562.968554] host7: xid f4f: Exchange timer armed : 0 msecs 18843426:May 21 11:02:27 poc2 kernel: [ 1562.968656] host7: xid f4f: Exchange timed out 18843581:May 21 11:02:27 poc2 kernel: [ 1562.991761] host7: xid 7e4: Exchange timer armed : 0 msecs 18843583:May 21 11:02:27 poc2 kernel: [ 1562.991853] host7: xid 7e4: Exchange timed out 18843690:May 21 11:02:27 poc2 kernel: [ 1563.011282] host7: xid a05: Exchange timer armed : 0 msecs Thanks On Wed, May 21, 2014 at 9:21 AM, Vasu Dev <vasu.dev@xxxxxxxxxxxxxxx> wrote: > On Tue, 2014-05-20 at 22:29 -0700, Jun Wu wrote: >> MTU were 1500 for both initiator and target. >> I used "ethtool -K p4p1 tso off" to turn off tcp segmentation offload >> on all machines. Register setting after the command is shown below. >> >> [root@poc3 jkong]# ethtool -k p4p1 >> Features for p4p1: >> rx-checksumming: on >> tx-checksumming: on >> tx-checksum-ipv4: on >> tx-checksum-ip-generic: off [fixed] >> tx-checksum-ipv6: on >> tx-checksum-fcoe-crc: on [fixed] >> tx-checksum-sctp: on >> scatter-gather: on >> tx-scatter-gather: on >> tx-scatter-gather-fraglist: off [fixed] >> tcp-segmentation-offload: off >> tx-tcp-segmentation: off >> tx-tcp-ecn-segmentation: off [fixed] >> tx-tcp6-segmentation: off >> udp-fragmentation-offload: off [fixed] >> generic-segmentation-offload: on >> generic-receive-offload: on >> large-receive-offload: off >> rx-vlan-offload: on >> tx-vlan-offload: on >> ntuple-filters: off >> receive-hashing: on >> highdma: on [fixed] >> rx-vlan-filter: on >> vlan-challenged: off [fixed] >> tx-lockless: off [fixed] >> netns-local: off [fixed] >> tx-gso-robust: off [fixed] >> tx-fcoe-segmentation: on [fixed] >> tx-gre-segmentation: off [fixed] >> tx-ipip-segmentation: off [fixed] >> tx-sit-segmentation: off [fixed] >> tx-udp_tnl-segmentation: off [fixed] >> tx-mpls-segmentation: off [fixed] >> fcoe-mtu: on [fixed] >> tx-nocache-copy: on >> loopback: off [fixed] >> rx-fcs: off [fixed] >> rx-all: off >> tx-vlan-stag-hw-insert: off [fixed] >> rx-vlan-stag-hw-parse: off [fixed] >> rx-vlan-stag-filter: off [fixed] >> l2-fwd-offload: off >> >> Info on NIC drivers >> >> [root@poc3 jkong]# ethtool -i p4p1 >> driver: ixgbe >> version: 3.15.1-k >> firmware-version: 0x80000208 >> bus-info: 0000:08:00.0 >> supports-statistics: yes >> supports-test: yes >> supports-eeprom-access: yes >> supports-register-dump: yes >> supports-priv-flags: no >> >> After the change, I repeated the same test and got similar failure on >> target side: >> >> [12253.032595] ft_queue_data_in: Failed to send frame >> ffff88062a638600, xid <0xa0c>, remaining 458752, lso_max <0x10000> > > It is send frame failure and to find out what caused send failure more > debug info in low level fcoe Tx path functions will be helpful, it can > be done by:- > > # echo 0xFF > /sys/module/libfc/parameters/debug_logging > # echo 0x1 > /sys/module/fcoe/parameters/debug_logging > > Disabling Tx offload may not help here and instead would slow down Tx, > so have them restored. > > Also, are you using switch between hosts and target ? In any case you > would need DCB PFC or PAUSE enabled to avoid excessive Tx retries though > that should not cause send failure. > > > //Vasu > > >> [12253.032605] ft_queue_data_in: Failed to send frame >> ffff88062a638600, xid <0xa0c>, remaining 393216, lso_max <0x10000> >> [12253.032609] ft_queue_data_in: Failed to send frame >> ffff88062a638600, xid <0xa0c>, remaining 327680, lso_max <0x10000> >> [12253.032613] ft_queue_data_in: Failed to send frame >> ffff88062a638600, xid <0xa0c>, remaining 262144, lso_max <0x10000> >> [12284.299877] ft_queue_data_in: Failed to send frame >> ffff8803202ec600, xid <0x3a2>, remaining 196608, lso_max <0x10000> >> [12284.299885] ft_queue_data_in: Failed to send frame >> ffff8803202ec600, xid <0x3a2>, remaining 131072, lso_max <0x10000> >> [12284.299889] ft_queue_data_in: Failed to send frame >> ffff8803202ec600, xid <0x3a2>, remaining 65536, lso_max <0x10000> >> [12284.299892] ft_queue_data_in: Failed to send frame >> ffff8803202ec600, xid <0x3a2>, remaining 0, lso_max <0x10000> >> [12284.451810] ft_queue_data_in: Failed to send frame >> ffff88061deb1400, xid <0xecf>, remaining 458752, lso_max <0x10000> >> [12284.451818] ft_queue_data_in: Failed to send frame >> ffff88061deb1400, xid <0xecf>, remaining 393216, lso_max <0x10000> >> [12284.451824] ft_queue_data_in: Failed to send frame >> ffff88061deb1400, xid <0xecf>, remaining 327680, lso_max <0x10000> >> [12284.451827] ft_queue_data_in: Failed to send frame >> ffff88061deb1400, xid <0xecf>, remaining 262144, lso_max <0x10000> >> [12284.451831] ft_queue_data_in: Failed to send frame >> ffff88061deb1400, xid <0xecf>, remaining 196608, lso_max <0x10000> >> [12284.451834] ft_queue_data_in: Failed to send frame >> ffff88061deb1400, xid <0xecf>, remaining 131072, lso_max <0x10000> >> [12347.503478] ft_queue_data_in: 2 callbacks suppressed >> [12347.503486] ft_queue_data_in: Failed to send frame >> ffff8806142bc800, xid <0xb4f>, remaining 458752, lso_max <0x10000> >> [12347.503492] ft_queue_data_in: Failed to send frame >> ffff8806142bc800, xid <0xb4f>, remaining 393216, lso_max <0x10000> >> [12347.503496] ft_queue_data_in: Failed to send frame >> ffff8806142bc800, xid <0xb4f>, remaining 327680, lso_max <0x10000> >> [12347.503517] ft_queue_data_in: Failed to send frame >> ffff8806142bc800, xid <0xb4f>, remaining 262144, lso_max <0x10000> >> [12378.402412] ft_queue_data_in: Failed to send frame >> ffff88062ddeac00, xid <0x6a5>, remaining 458752, lso_max <0x10000> >> [12378.402420] ft_queue_data_in: Failed to send frame >> ffff88062ddeac00, xid <0x6a5>, remaining 393216, lso_max <0x10000> >> [12378.402425] ft_queue_data_in: Failed to send frame >> ffff88062ddeac00, xid <0x6a5>, remaining 327680, lso_max <0x10000> >> [12378.402428] ft_queue_data_in: Failed to send frame >> ffff88062ddeac00, xid <0x6a5>, remaining 262144, lso_max <0x10000> >> [12378.402432] ft_queue_data_in: Failed to send frame >> ffff88062ddeac00, xid <0x6a5>, remaining 196608, lso_max <0x10000> >> [12378.402436] ft_queue_data_in: Failed to send frame >> ffff88062ddeac00, xid <0x6a5>, remaining 131072, lso_max <0x10000> >> [12378.402440] ft_queue_data_in: Failed to send frame >> ffff88062ddeac00, xid <0x6a5>, remaining 65536, lso_max <0x10000> >> [12378.402444] ft_queue_data_in: Failed to send frame >> ffff88062ddeac00, xid <0x6a5>, remaining 0, lso_max <0x10000> >> [13049.224513] ft_queue_data_in: Failed to send frame >> ffff880614588c00, xid <0xd2f>, remaining 196608, lso_max <0x10000> >> [13049.224524] ft_queue_data_in: Failed to send frame >> ffff880614588c00, xid <0xd2f>, remaining 131072, lso_max <0x10000> >> [13049.224528] ft_queue_data_in: Failed to send frame >> ffff880614588c00, xid <0xd2f>, remaining 65536, lso_max <0x10000> >> [13049.224532] ft_queue_data_in: Failed to send frame >> ffff880614588c00, xid <0xd2f>, remaining 0, lso_max <0x10000> >> [13052.511306] ft_queue_data_in: Failed to send frame >> ffff88062d49f000, xid <0x8ae>, remaining 196608, lso_max <0x10000> >> [13052.511313] ft_queue_data_in: Failed to send frame >> ffff88062d49f000, xid <0x8ae>, remaining 131072, lso_max <0x10000> >> [13052.511317] ft_queue_data_in: Failed to send frame >> ffff88062d49f000, xid <0x8ae>, remaining 65536, lso_max <0x10000> >> [13052.511321] ft_queue_data_in: Failed to send frame >> ffff88062d49f000, xid <0x8ae>, remaining 0, lso_max <0x10000> >> [13087.976748] ft_queue_data_in: Failed to send frame >> ffff88031afc9c00, xid <0x96b>, remaining 458752, lso_max <0x10000> >> [13087.998453] ft_queue_data_in: Failed to send frame >> ffff88032c881200, xid <0xb23>, remaining 458752, lso_max <0x10000> >> [13087.998459] ft_queue_data_in: Failed to send frame >> ffff88032c881200, xid <0xb23>, remaining 393216, lso_max <0x10000> >> [13087.998463] ft_queue_data_in: Failed to send frame >> ffff88032c881200, xid <0xb23>, remaining 327680, lso_max <0x10000> >> [13087.998467] ft_queue_data_in: Failed to send frame >> ffff88032c881200, xid <0xb23>, remaining 262144, lso_max <0x10000> >> [13087.998470] ft_queue_data_in: Failed to send frame >> ffff88032c881200, xid <0xb23>, remaining 196608, lso_max <0x10000> >> [13087.998474] ft_queue_data_in: Failed to send frame >> ffff88032c881200, xid <0xb23>, remaining 131072, lso_max <0x10000> >> [13087.998478] ft_queue_data_in: Failed to send frame >> ffff88032c881200, xid <0xb23>, remaining 65536, lso_max <0x10000> >> [13087.998482] ft_queue_data_in: Failed to send frame >> ffff88032c881200, xid <0xb23>, remaining 0, lso_max <0x10000> >> [13119.177286] ft_queue_data_in: Failed to send frame >> ffff88062dff7400, xid <0xfcf>, remaining 458752, lso_max <0x10000> >> [13119.177297] ft_queue_data_in: Failed to send frame >> ffff88062dff7400, xid <0xfcf>, remaining 393216, lso_max <0x10000> >> [13119.177302] ft_queue_data_in: Failed to send frame >> ffff88062dff7400, xid <0xfcf>, remaining 327680, lso_max <0x10000> >> [13119.177307] ft_queue_data_in: Failed to send frame >> ffff88062dff7400, xid <0xfcf>, remaining 262144, lso_max <0x10000> >> [13119.177311] ft_queue_data_in: Failed to send frame >> ffff88062dff7400, xid <0xfcf>, remaining 196608, lso_max <0x10000> >> [13119.177316] ft_queue_data_in: Failed to send frame >> ffff88062dff7400, xid <0xfcf>, remaining 131072, lso_max <0x10000> >> [13119.177321] ft_queue_data_in: Failed to send frame >> ffff88062dff7400, xid <0xfcf>, remaining 65536, lso_max <0x10000> >> [13119.177325] ft_queue_data_in: Failed to send frame >> ffff88062dff7400, xid <0xfcf>, remaining 0, lso_max <0x10000> >> [13122.335322] ------------[ cut here ]------------ >> [13122.335336] WARNING: CPU: 6 PID: 2165 at >> include/scsi/fc_frame.h:173 fcoe_percpu_receive_thread+0x507/0x53c >> [fcoe]() >> [13122.335338] Modules linked in: async_memcpy async_xor xor async_tx >> fcoe libfcoe tcm_fc libfc scsi_transport_fc scsi_tgt target_core_pscsi >> target_core_file target_core_iblock iscsi_target_mod target_core_mod >> 8021q garp mrp bridge stp llc iTCO_wdt gpio_ich iTCO_vendor_support >> coretemp kvm_intel kvm crc32c_intel microcode serio_raw i2c_i801 >> lpc_ich mfd_core ses enclosure i7core_edac ioatdma edac_core shpchp >> acpi_cpufreq nfsd auth_rpcgss nfs_acl lockd sunrpc radeon >> drm_kms_helper ttm drm ixgbe igb ata_generic mdio pata_acpi ptp >> pata_jmicron pps_core i2c_algo_bit aacraid dca i2c_core [last >> unloaded: vd] >> [13122.335390] CPU: 6 PID: 2165 Comm: fcoethread/6 Tainted: GF >> O 3.13.10-200.zbfcoepatch.fc20.x86_64 #1 >> [13122.335392] Hardware name: Supermicro X8DTN/X8DTN, BIOS 2.1c 10/28/2011 >> [13122.335394] 0000000000000009 ffff88062b04bdd0 ffffffff81687eac >> 0000000000000000 >> [13122.335400] ffff88062b04be08 ffffffff8106d4dd ffffe8ffffc41748 >> ffff88062a444700 >> [13122.335404] ffff8800b7e926e8 0000000000000002 ffff88062b04be88 >> ffff88062b04be18 >> [13122.335408] Call Trace: >> [13122.335419] [<ffffffff81687eac>] dump_stack+0x45/0x56 >> [13122.335426] [<ffffffff8106d4dd>] warn_slowpath_common+0x7d/0xa0 >> [13122.335430] [<ffffffff8106d5ba>] warn_slowpath_null+0x1a/0x20 >> [13122.335435] [<ffffffffa0651517>] >> fcoe_percpu_receive_thread+0x507/0x53c [fcoe] >> [13122.335440] [<ffffffffa0651010>] ? fcoe_set_port_id+0x50/0x50 [fcoe] >> [13122.335446] [<ffffffff8108f2f2>] kthread+0xd2/0xf0 >> [13122.335450] [<ffffffff8108f220>] ? insert_kthread_work+0x40/0x40 >> [13122.335458] [<ffffffff81696dbc>] ret_from_fork+0x7c/0xb0 >> [13122.335461] [<ffffffff8108f220>] ? insert_kthread_work+0x40/0x40 >> [13122.335464] ---[ end trace e4509e1053f499ac ]--- >> >> Thanks, >> >> Jun >> >> On Tue, May 20, 2014 at 11:03 AM, Nicholas A. Bellinger >> <nab@xxxxxxxxxxxxxxx> wrote: >> > On Mon, 2014-05-19 at 17:29 -0700, Jun Wu wrote: >> >> Hi Nicholas, >> >> >> >> We downloaded the source of our running kernel (3.13.10-200) and >> >> applied your percpu-ida pre-allocation regression fix, then compiled >> >> and installed the kernel. I repeated the same test three times, >> >> running 10 fio sessions to 10 drives on the target through fcoe vn2vn. >> >> In the first two tests, the target machine hung with the following >> >> messages: >> >> >> >> 15231 May 19 11:49:27 poc1 kernel: [ 1073.783229] ft_queue_data_in: >> >> Failed to send frame ffff880c0b188200, xid <0x2a5>, remaining 196608, >> >> lso_max <0x10000> >> >> 15232 May 19 11:49:27 poc1 kernel: [ 1073.783238] ft_queue_data_in: >> >> Failed to send frame ffff880c0b188200, xid <0x2a5>, remaining 131072, >> >> lso_max <0x10000> >> >> 15233 May 19 11:49:27 poc1 kernel: [ 1073.783242] ft_queue_data_in: >> >> Failed to send frame ffff880c0b188200, xid <0x2a5>, remaining 65536, >> >> lso_max <0x10000> >> >> 15234 May 19 11:49:27 poc1 kernel: [ 1073.783245] ft_queue_data_in: >> >> Failed to send frame ffff880c0b188200, xid <0x2a5>, remaining 0, >> >> lso_max <0x10000> >> >> 15235 May 19 11:49:30 poc1 kernel: [ 1076.907061] ft_queue_data_in: >> >> Failed to send frame ffff880c1d1df000, xid <0x305>, remaining 196608, >> >> lso_max <0x10000> >> >> 15236 May 19 11:49:30 poc1 kernel: [ 1076.907068] ft_queue_data_in: >> >> Failed to send frame ffff880c1d1df000, xid <0x305>, remaining 131072, >> >> lso_max <0x10000> >> >> 15237 May 19 11:49:30 poc1 kernel: [ 1076.907073] ft_queue_data_in: >> >> Failed to send frame ffff880c1d1df000, xid <0x305>, remaining 65536, >> >> lso_max <0x10000> >> >> 15238 May 19 11:49:30 poc1 kernel: [ 1076.907077] ft_queue_data_in: >> >> Failed to send frame ffff880c1d1df000, xid <0x305>, remaining 0, >> >> lso_max <0x10000> >> >> 15239 May 19 11:50:01 poc1 kernel: [ 1107.918910] ft_queue_data_in: >> >> Failed to send frame ffff88060cd40800, xid <0x3cb>, remaining 458752, >> >> lso_max <0x10000> >> >> 15240 May 19 11:50:01 poc1 kernel: [ 1107.918918] ft_queue_data_in: >> >> Failed to send frame ffff88060cd40800, xid <0x3cb>, remaining 393216, >> >> lso_max <0x10000> >> >> 15241 May 19 11:50:01 poc1 kernel: [ 1107.918922] ft_queue_data_in: >> >> Failed to send frame ffff88060cd40800, xid <0x3cb>, remaining 327680, >> >> lso_max <0x10000> >> >> 15242 May 19 11:50:01 poc1 kernel: [ 1107.918925] ft_queue_data_in: >> >> Failed to send frame ffff88060cd40800, xid <0x3cb>, remaining 262144, >> >> lso_max <0x10000> >> >> 15243 May 19 11:50:01 poc1 kernel: [ 1107.918929] ft_queue_data_in: >> >> Failed to send frame ffff88060cd40800, xid <0x3cb>, remaining 196608, >> >> lso_max <0x10000> >> >> 15244 May 19 11:50:01 poc1 kernel: [ 1107.918932] ft_queue_data_in: >> >> Failed to send frame ffff88060cd40800, xid <0x3cb>, remaining 131072, >> >> lso_max <0x10000> >> >> 15245 May 19 11:50:01 poc1 kernel: [ 1107.918936] ft_queue_data_in: >> >> Failed to send frame ffff88060cd40800, xid <0x3cb>, remaining 65536, >> >> lso_max <0x10000> >> >> 15246 May 19 11:50:01 poc1 kernel: [ 1107.918939] ft_queue_data_in: >> >> Failed to send frame ffff88060cd40800, xid <0x3cb>, remaining 0, >> >> lso_max <0x10000> >> >> 15247 May 19 11:50:05 poc1 kernel: [ 1111.450900] ft_queue_data_in: >> >> Failed to send frame ffff880c0b24ca00, xid <0xea6>, remaining 196608, >> >> lso_max <0x10000> >> >> 15248 May 19 11:50:05 poc1 kernel: [ 1111.450908] ft_queue_data_in: >> >> Failed to send frame ffff880c0b24ca00, xid <0xea6>, remaining 131072, >> >> lso_max <0x10000> >> >> 15249 May 19 11:51:12 poc1 kernel: [ 1178.698434] ft_queue_data_in: 6 >> >> callbacks suppressed >> >> 15250 May 19 11:51:12 poc1 kernel: [ 1178.698440] ft_queue_data_in: >> >> Failed to send frame ffff88060ba97400, xid <0xb8a>, remaining 458752, >> >> lso_max <0x10000> >> >> 15251 May 19 11:51:12 poc1 kernel: [ 1178.698446] ft_queue_data_in: >> >> Failed to send frame ffff88060ba97400, xid <0xb8a>, remaining 393216, >> >> lso_max <0x10000> >> >> 15252 May 19 11:51:12 poc1 kernel: [ 1178.698449] ft_queue_data_in: >> >> Failed to send frame ffff88060ba97400, xid <0xb8a>, remaining 327680, >> >> lso_max <0x10000> >> >> 15253 May 19 11:51:12 poc1 kernel: [ 1178.698453] ft_queue_data_in: >> >> Failed to send frame ffff88060ba97400, xid <0xb8a>, remaining 262144, >> >> lso_max <0x10000> >> >> 15254 May 19 11:51:12 poc1 kernel: [ 1178.698456] ft_queue_data_in: >> >> Failed to send frame ffff88060ba97400, xid <0xb8a>, remaining 196608, >> >> lso_max <0x10000> >> >> 15255 May 19 11:51:12 poc1 kernel: [ 1178.698460] ft_queue_data_in: >> >> Failed to send frame ffff88060ba97400, xid <0xb8a>, remaining 131072, >> >> lso_max <0x10000> >> >> 15256 May 19 11:51:12 poc1 kernel: [ 1178.698463] ft_queue_data_in: >> >> Failed to send frame ffff88060ba97400, xid <0xb8a>, remaining 65536, >> >> lso_max <0x10000> >> >> 15257 May 19 11:51:12 poc1 kernel: [ 1178.698467] ft_queue_data_in: >> >> Failed to send frame ffff88060ba97400, xid <0xb8a>, remaining 0, >> >> lso_max <0x10000> >> >> >> > >> > The call into lport->tt.seq_send() libfc code is failing to send >> > outgoing solicited data-in. From the output, note the LSO (large >> > segment offload aka TCP segment offload) feature has been enabled by the >> > underlying NIC hardware. >> > >> > So in order to isolate possible issues, I'd recommend: >> > >> > - Disabling hardware offloads on both initiator and target sides (LRO + >> > LSO) using ethtool -K >> > - Disabling any jumbo frames settings on either side >> > >> > Is there any other non standard network and/or switch settings that are >> > in place..? Also, please confirm what your NIC + switch setup looks >> > like. >> > >> > Rob & Open-FCoE folks, is there anything else to take into consideration >> > here..? >> > >> >> >> >> I didn't see the previous message "unable to handle kernel NULL >> >> pointer dereference at 0000000000000048". So it must have been fixed >> >> by your change. >> >> >> > >> > Thanks for confirming that bit. >> > >> > --nab >> > >> _______________________________________________ >> fcoe-devel mailing list >> fcoe-devel@xxxxxxxxxxxxx >> http://lists.open-fcoe.org/mailman/listinfo/fcoe-devel > > -- To unsubscribe from this list: send the line "unsubscribe target-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
