Hi, I'm running a redundant storage server pair with DRBD and LIO. On the secondary node I had some RAID card resets, which triggered DRBD to disconnect the node - but before that, requests were held for several seconds in the DRBD layer, which seems to be a problem for LIO (or maybe the subsequent initiator reconnects - I'm not sure). My efforts to reproduce the bug in a test environment failed so far, but in production it works reliably (the RAID card in the secondary node stalled 3 times and the primary node crashed every time). I've tried upgrading the kernel between crashes 1 and 2, but that didn't help. Regards, Matyas 1st crash: emerg kernel: [6995187.657152] general protection fault: 0000 [#3] SMP warning kernel: [6995187.657292] CPU 3 warning kernel: [6995187.657333] Modules linked in: tcm_loop tcm_fc iscsi_target_mod target_core_pscsi target_core_file target_core_iblock target_core_mod nfnetlink_log nfnetlink dm_snapshot usb_storage parport_pc ppdev lp parport drbd lru_cache libfc scsi_transport_fc scsi_tgt configfs bnep rfcomm bluetooth rfkill uinput nfsd nfs nfs_acl auth_rpcgss fscache lockd sunrpc nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ip6_tables nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack xt_tcpudp iptable_filter ip_tables x_tables loop snd_pcm snd_page_alloc snd_timer acpi_cpufreq iTCO_wdt iTCO_vendor_support joydev mperf i7core_edac processor snd soundcore psmouse i2c_i801 coretemp serio_raw ioatdma edac_core i2c_core pcspkr evdev thermal_sys button crc32c_intel ext4 crc16 jbd2 mbcache dm_mod raid456 async_raid6_recov async_memcpy async_pq async_xor xor async_tx raid6_pq raid1 md_mod ses sg enclosure sd_mod crc_t10dif usbhidhid uhci_hcd megaraid_sas ahci libahci libata ehci_hcd scsi_mod usbcore usb_ info kernel: common ixgbe mdio igb dca [last unloaded: target_core_mod] warning kernel: [6995187.661785] warning kernel: [6995187.661843] Pid: 19644, comm: iscsi_trx Tainted: G B D 3.2.0-4-amd64 #1 Debian 3.2.51-1 Supermicro X8DT3/X8DT3 warning kernel: [6995187.662081] RIP: 0010:[<ffffffffa04b69f1>] [<ffffffffa04b69f1>] iscsit_find_cmd_from_ttt+0x32/0x8a [iscsi_target_mod] warning kernel: [6995187.662221] RSP: 0018:ffff880142d65de0 EFLAGS: 00010293 warning kernel: [6995187.662287] RAX: ffff88030b89f368 RBX: ffff88030b89f000 RCX: 00000000207c4cea warning kernel: [6995187.662373] RDX: dead000000100100 RSI: 0000000000068e85 RDI: ffff88030b89f2e8 warning kernel: [6995187.662459] RBP: 0000000000068e85 R08: 00000001686fd3a0 R09: ffff88032ff66380 warning kernel: [6995187.662546] R10: ffff88032ff66380 R11: 0000000000013780 R12: dead0000000ffee0 warning kernel: [6995187.662632] R13: ffff88030b89f2e8 R14: 0000000000000000 R15: 0000000000000000 warning kernel: [6995187.662718] FS: 0000000000000000(0000) GS:ffff88033fc60000(0000) knlGS:0000000000000000 warning kernel: [6995187.662807] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b warning kernel: [6995187.662873] CR2: 00007fddc3187130 CR3: 0000000001605000 CR4: 00000000000006e0 warning kernel: [6995187.662960] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 warning kernel: [6995187.663045] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 warning kernel: [6995187.663131] Process iscsi_trx (pid: 19644, threadinfo ffff880142d64000, task ffff88032e1d0970) emerg kernel: [6995187.663220] Stack: warning kernel: [6995187.663278] ffff88030b89f000 ffff88030b89f000 0000000000000000 ffff880142d65e88 warning kernel: [6995187.663509] 0000000000000000 ffffffffa04bb1cf 0000000000000000 ffff88033fcb3780 warning kernel: [6995187.663740] ffff88032e1d0d30 0000000200000000 00000000814052a0 ffff880310e6b280 emerg kernel: [6995187.663971] Call Trace: warning kernel: [6995187.664037] [<ffffffffa04bb1cf>] ? iscsi_target_rx_thread+0x118d/0x1943 [iscsi_target_mod] warning kernel: [6995187.664131] [<ffffffffa04ba042>] ? iscsit_thread_get_cpumask+0x88/0x88 [iscsi_target_mod] warning kernel: [6995187.664224] [<ffffffff8105f631>] ? kthread+0x76/0x7e warning kernel: [6995187.664292] [<ffffffff81356374>] ? kernel_thread_helper+0x4/0x10 warning kernel: [6995187.664361] [<ffffffff8105f5bb>] ? kthread_worker_fn+0x139/0x139 warning kernel: [6995187.664430] [<ffffffff81356370>] ? gs_change+0x13/0x13 emerg kernel: [6995187.664495] Code: 00 00 41 54 55 89 f5 53 53 48 89 fb 4c 89 ef e8 93 89 e9 e0 4c 8b a3 68 03 00 00 48 8d 83 68 03 00 00 49 81 ec 20 02 00 00 eb 20 <41> 39 6c 24 24 75 0a 4c 89 ef e8 29 89 e9 e0 eb 3b4d 8b a4 24 alert kernel: [6995187.666958] RIP [<ffffffffa04b69f1>] iscsit_find_cmd_from_ttt+0x32/0x8a [iscsi_target_mod] warning kernel: [6995187.667088] RSP <ffff880142d65de0> warning kernel: [6995187.667217] ---[ end trace b9be73a1398857b9 ]--- emerg kernel: [6995187.667282] Kernel panic - not syncing: Fatal exception in interrupt 2nd crash: warning kernel: [568490.274380] block drbd0: Remote failed to finish a request within ko-count * timeout info kernel: [568490.274390] block drbd0: peer( Secondary -> Unknown ) conn( Connected -> Timeout ) pdsk( UpToDate -> DUnknown ) info kernel: [568490.274539] block drbd0: new current UUID BC772B18F6FE7FB7:9E9A652A9DC0D857:7116C7324E030315:7115C7324E030315 info kernel: [568490.274842] block drbd0: asender terminated info kernel: [568490.274849] block drbd0: Terminating drbd0_asender info kernel: [568490.275598] block drbd0: Connection closed info kernel: [568490.275605] block drbd0: conn( Timeout -> Unconnected ) info kernel: [568490.275610] block drbd0: receiver terminated info kernel: [568490.275613] block drbd0: Restarting drbd0_receiver info kernel: [568490.275616] block drbd0: receiver (re)started info kernel: [568490.275621] block drbd0: conn( Unconnected -> WFConnection ) emerg kernel: [568492.473906] general protection fault: 0000 [#1] SMP warning kernel: [568492.474040] CPU 0 warning kernel: [568492.474085] Modules linked in: tcm_loop tcm_fc iscsi_target_mod target_core_pscsi target_core_file target_core_iblock target_core_mod netconsole drbd lru_cache libfc scsi_transport_fc scsi_tgt configfs bnep rfcomm parport_pc bluetooth ppdev rfkill lp parport uinput nfsd nfs nfs_acl auth_rpcgss fscache lockd sunrpc nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ip6_tables nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack xt_tcpudp iptable_filter ip_tables x_tables loop snd_pcm snd_page_alloc snd_timer ioatdma iTCO_wdt psmouse i7core_edac edac_core snd soundcore pcspkr iTCO_vendor_support coretemp joydev evdev serio_raw acpi_cpufreq mperf processor i2c_i801 thermal_sys crc32c_intel button ext4 crc16 jbd2 mbcache dm_mod raid1 md_mod sg ses sd_mod enclosure crc_t10dif usbhid hid usb_storage uhci_hcd ahci libahci libata megaraid_sas ehci_hcd usbcore scsi_mod usb_common ixgbe igb mdio i2c_algo_bit i2c_core dca [last unloaded: target_core_mod] warning kernel: [568492.478364] warning kernel: [568492.478426] Pid: 5056, comm: iscsi_trx Not tainted 3.2.0-4-amd64 #1 Debian 3.2.57-3+deb7u2 Supermicro X8DT3/X8DT3 warning kernel: [568492.478710] RIP: 0010:[<ffffffffa054ce4d>] [<ffffffffa054ce4d>] iscsit_close_connection+0x5b/0x4bb [iscsi_target_mod] warning kernel: [568492.478858] RSP: 0018:ffff8803149ebdb0 EFLAGS: 00010202 warning kernel: [568492.478927] RAX: dead000000100100 RBX: ffff880313c4f000 RCX: 0002050a77f690fd warning kernel: [568492.479017] RDX: 0000000000000001 RSI: ffff880314a3da98 RDI: ffffffffa054ce5f warning kernel: [568492.479107] RBP: ffff88032f14a000 R08: ffff8803149ea000 R09: ffffffff81600000 warning kernel: [568492.479197] R10: ffffffff81600000 R11: ffffffff81600000 R12: ffff880313c4f2e8 warning kernel: [568492.479287] R13: dead0000000ffee0 R14: ffff880313c4f208 R15: 0000000000000000 warning kernel: [568492.479377] FS: 0000000000000000(0000) GS:ffff88033fc00000(0000) knlGS:0000000000000000 warning kernel: [568492.479469] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b warning kernel: [568492.479539] CR2: 00007f375d7d03e0 CR3: 0000000001605000 CR4: 00000000000006f0 warning kernel: [568492.479629] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 warning kernel: [568492.479718] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 warning kernel: [568492.479808] Process iscsi_trx (pid: 5056, threadinfo ffff8803149ea000, task ffff88031247d040) emerg kernel: [568492.479901] Stack: warning kernel: [568492.479961] 0000000000000030 0000000000000001 ffff880313c4f368 ffffffff8134fd9b warning kernel: [568492.480204] 0000000000000246 ffff880313c4f000 ffff88032f592580 ffff8803149ebe88 warning kernel: [568492.480447] 0000000000000000 ffff880313c4f2e8 0000000000000000 ffffffffa054c948 emerg kernel: [568492.480689] Call Trace: warning kernel: [568492.480757] [<ffffffff8134fd9b>] ? _raw_spin_lock_bh+0xe/0x1c warning kernel: [568492.480836] [<ffffffffa054c948>] ? iscsi_target_rx_thread+0x18e4/0x1930 [iscsi_target_mod] warning kernel: [568492.480936] [<ffffffffa054b064>] ? iscsit_thread_get_cpumask+0x88/0x88 [iscsi_target_mod] warning kernel: [568492.481033] [<ffffffff8105f6d9>] ? kthread+0x76/0x7e warning kernel: [568492.481105] [<ffffffff81356d74>] ? kernel_thread_helper+0x4/0x10 warning kernel: [568492.481179] [<ffffffff8105f663>] ? kthread_worker_fn+0x139/0x139 warning kernel: [568492.481251] [<ffffffff81356d70>] ? gs_change+0x13/0x13 emerg kernel: [568492.481319] Code: 89 df e8 4b 21 ff ff 4c 89 e7 e8 5c 2f e0 e0 4c 8b ab 68 03 00 00 48 8d 83 68 03 00 00 48 89 44 24 10 49 81 ed 20 02 00 00 eb 20 <41> 83 bd a0 00 00 00 01 75 08 4c 89 ef e8 1f 59 ff ff 4d 8b ad alert kernel: [568492.484018] RIP [<ffffffffa054ce4d>] iscsit_close_connection+0x5b/0x4bb [iscsi_target_mod] warning kernel: [568492.484156] RSP <ffff8803149ebdb0> warning kernel: [568492.484262] ---[ end trace 46ecbbe37bed4a2e ]--- emerg kernel: [568492.484331] Kernel panic - not syncing: Fatal exception in interrupt warning kernel: [568492.484407] Pid: 5056, comm: iscsi_trx Tainted: G D 3.2.0-4-amd64 #1 Debian 3.2.57-3+deb7u2 warning kernel: [568492.484504] Call Trace: warning kernel: [568492.484570] [<ffffffff813492d8>] ? panic+0x95/0x1a2 3rd crash: warning kernel: [1022062.839595] block drbd0: Remote failed to finish a request within ko-count * timeout info kernel: [1022062.839604] block drbd0: peer( Secondary -> Unknown ) conn( Connected -> Timeout ) pdsk( UpToDate -> DUnknown ) info kernel: [1022062.839697] block drbd0: new current UUID 9E9A652A9DC0D857:7115C7324E030315:8355E6AB18AD03EE:8354E6AB18AD03EF info kernel: [1022062.839957] block drbd0: asender terminated info kernel: [1022062.839962] block drbd0: Terminating drbd0_asender info kernel: [1022062.840629] block drbd0: Connection closed info kernel: [1022062.840636] block drbd0: conn( Timeout -> Unconnected ) info kernel: [1022062.840641] block drbd0: receiver terminated info kernel: [1022062.840644] block drbd0: Restarting drbd0_receiver info kernel: [1022062.840647] block drbd0: receiver (re)started info kernel: [1022062.840651] block drbd0: conn( Unconnected -> WFConnection ) emerg kernel: [1022062.840840] general protection fault: 0000 [#1] SMP warning kernel: [1022062.840986] CPU 5 warning kernel: [1022062.841028] Modules linked in: tcm_loop tcm_fc iscsi_target_mod target_core_pscsi target_core_file target_core_iblock target_core_mod netconsole parport_pc ppdev bnep rfcomm bluetooth rfkill lp parport drbd lru_cache libfc scsi_transport_fc scsi_tgt configfs uinput nfsd nfs nfs_acl auth_rpcgss fscache lockd sunrpc nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ip6_tables nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack xt_tcpudp iptable_filter ip_tables x_tables loop snd_pcm snd_page_alloc snd_timer ioatdma acpi_cpufreq psmouse i7core_edac iTCO_wdt serio_raw mperf edac_core iTCO_vendor_support coretemp joydev evdev snd soundcore pcspkr processor i2c_i801 thermal_sys crc32c_intel button ext4 crc16 jbd2 mbcache dm_mod raid1 md_mod sg sd_mod ses enclosure crc_t10dif usbhid hid usb_storage uhci_hcd megaraid_sas ahci libahci libata scsi_mod ehci_hcd usbcore usb_common ixgbe mdio igb i2c_algo_bit i2c_core dca [last unloaded: target_core_mod] warning kernel: [1022062.845110] warning kernel: [1022062.845169] Pid: 5165, comm: iscsi_trx Not tainted 3.2.0-4-amd64 #1 Debian 3.2.57-3+deb7u2 Supermicro X8DT3/X8DT3 warning kernel: [1022062.845373] RIP: 0010:[<ffffffffa0442696>] [<ffffffffa0442696>] core_tmr_release_req+0x2c/0x66 [target_core_mod] warning kernel: [1022062.845514] RSP: 0018:ffff880314badd80 EFLAGS: 00010046 warning kernel: [1022062.845581] RAX: 0000000000000286 RBX: ffff8802ce58d7c0 RCX: dead000000100100 warning kernel: [1022062.845668] RDX: dead000000200200 RSI: 0000000000000286 RDI: ffff880329dc45d4 warning kernel: [1022062.845754] RBP: ffff880329dc45d4 R08: ffff880314badd60 R09: ffff8803155fc040 warning kernel: [1022062.845840] R10: ffff8803155fc040 R11: ffff8803155fc040 R12: ffff88032f365ae8 warning kernel: [1022062.845927] R13: ffff8803121d0ac0 R14: ffff88032f365a08 R15: ffff8803121d0ac0 warning kernel: [1022062.846014] FS: 0000000000000000(0000) GS:ffff88033fca0000(0000) knlGS:0000000000000000 warning kernel: [1022062.846103] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b warning kernel: [1022062.846170] CR2: 00007f73283f89de CR3: 0000000001605000 CR4: 00000000000006e0 warning kernel: [1022062.846257] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 warning kernel: [1022062.846344] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 warning kernel: [1022062.846430] Process iscsi_trx (pid: 5165, threadinfo ffff880314bac000, task ffff88032e951780) emerg kernel: [1022062.846519] Stack: warning kernel: [1022062.846577] 0000000000000282 ffff8803156517c0 ffff8803151ea800 ffffffffa0445b6e warning kernel: [1022062.846813] ffff88032f365800 ffffffffa053af12 0000000000000030 0000000000000001 warning kernel: [1022062.847046] ffff88032f365b68 ffff8803151ea800 0000000000000246 ffff88032f365800 emerg kernel: [1022062.847280] Call Trace: warning kernel: [1022062.847349] [<ffffffffa0445b6e>] ? transport_release_cmd+0x21/0x71 [target_core_mod] warning kernel: [1022062.847442] [<ffffffffa053af12>] ? iscsit_close_connection+0x120/0x4bb [iscsi_target_mod] warning kernel: [1022062.847536] [<ffffffffa053a948>] ? iscsi_target_rx_thread+0x18e4/0x1930 [iscsi_target_mod] warning kernel: [1022062.847630] [<ffffffffa0539064>] ? iscsit_thread_get_cpumask+0x88/0x88 [iscsi_target_mod] warning kernel: [1022062.847723] [<ffffffff8105f6d9>] ? kthread+0x76/0x7e warning kernel: [1022062.847792] [<ffffffff81356d74>] ? kernel_thread_helper+0x4/0x10 warning kernel: [1022062.847862] [<ffffffff8105f663>] ? kthread_worker_fn+0x139/0x139 warning kernel: [1022062.847931] [<ffffffff81356d70>] ? gs_change+0x13/0x13 emerg kernel: [1022062.847997] Code: 53 48 89 fb 56 48 8b 6f 30 48 85 ed 74 45 48 81 c5 d4 01 00 00 48 89 ef e8 98 d5 f0 e0 48 8b 4b 40 48 8b 53 48 48 89 c6 48 89 ef <48> 89 51 08 48 89 0a 48 ba 00 01 10 00 00 00 ad de 48 b9 00 02 alert kernel: [1022062.850596] RIP [<ffffffffa0442696>] core_tmr_release_req+0x2c/0x66 [target_core_mod] warning kernel: [1022062.856021] RSP <ffff880314badd80> warning kernel: [1022062.856083] ---[ end trace 953259b8f570f942 ]--- -- To unsubscribe from this list: send the line "unsubscribe target-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
