Hi Nab, On Thursday 19 June 2014 01:32 AM, Nicholas A. Bellinger wrote:
Hi Santosh, On Wed, 2014-06-18 at 15:34 +0530, Santosh Kulkarni wrote:Hi Nab, Facing Kernel OOP when i am issuing Writes across Multiple connections in a single session. The crash happens immediately when the DataOut PDU is sent on the second connection.To confirm, this occurs when the initiator sends a ISCSI_OP_SCSI_CMD with ITT=X setting it's allegiance to one connection, and then (illegally) sends a ISCSI_OP_SCSI_DATA_OUT with ITT=X on another connection, right..?
There was a slight mistake in the way i formed my sentence earlier.The WRITE10 request and DataOut PDUs are being issued on the "same connection" within a session. The Crash happens only when this is issued on multiple connections.ITT remains same across all connections
If i carry out WRITE only on single connections the target is behaving fine.And the behavior is not persistent.It happens intermittently.Here is a new crash dump
Jun 19 12:58:30 lio-dev kernel: [ 92.103007] Command ITT: 0xa3a3a3a3 received DataOUT for a NON-WRITE command. Jun 19 12:58:30 lio-dev kernel: [ 92.103014] ISCSI_FLAG_CMD_WRITE & ISCSI_FLAG_CMD_FINAL not set. Bad iSCSI Initiator. Jun 19 12:58:30 lio-dev kernel: [ 92.103015] ISCSI_FLAG_CMD_WRITE & ISCSI_FLAG_CMD_FINAL not set. Bad iSCSI Initiator. Jun 19 12:58:30 lio-dev kernel: [ 92.103016] ISCSI_FLAG_CMD_WRITE & ISCSI_FLAG_CMD_FINAL not set. Bad iSCSI Initiator. Jun 19 12:58:30 lio-dev kernel: [ 92.103018] ISCSI_FLAG_CMD_WRITE & ISCSI_FLAG_CMD_FINAL not set. Bad iSCSI Initiator. Jun 19 12:58:30 lio-dev kernel: [ 92.103019] ISCSI_FLAG_CMD_WRITE & ISCSI_FLAG_CMD_FINAL not set. Bad iSCSI Initiator. Jun 19 12:58:30 lio-dev kernel: [ 92.103020] ISCSI_FLAG_CMD_WRITE & ISCSI_FLAG_CMD_FINAL not set. Bad iSCSI Initiator. Jun 19 12:58:30 lio-dev kernel: [ 92.103021] ISCSI_FLAG_CMD_WRITE & ISCSI_FLAG_CMD_FINAL not set. Bad iSCSI Initiator. Jun 19 12:58:30 lio-dev kernel: [ 92.103022] ISCSI_FLAG_CMD_WRITE & ISCSI_FLAG_CMD_FINAL not set. Bad iSCSI Initiator. Jun 19 12:58:30 lio-dev kernel: [ 92.103023] ISCSI_FLAG_CMD_WRITE & ISCSI_FLAG_CMD_FINAL not set. Bad iSCSI Initiator. Jun 19 12:58:30 lio-dev kernel: [ 92.103025] ISCSI_FLAG_CMD_WRITE & ISCSI_FLAG_CMD_FINAL not set. Bad iSCSI Initiator. Jun 19 12:58:30 lio-dev kernel: [ 92.105691] ISCSI_FLAG_CMD_WRITE & ISCSI_FLAG_CMD_FINAL not set. Bad iSCSI Initiator. Jun 19 12:58:31 lio-dev kernel: [ 93.263372] ITT: 0x00000000 response_queue_count: -1 Jun 19 12:58:31 lio-dev kernel: [ 93.272815] Command ITT: 0xa3a3a3a3 received DataOUT for a NON-WRITE command. Jun 19 12:58:31 lio-dev kernel: [ 93.272823] ISCSI_FLAG_CMD_WRITE & ISCSI_FLAG_CMD_FINAL not set. Bad iSCSI Initiator. Jun 19 12:58:31 lio-dev kernel: [ 93.272825] ISCSI_FLAG_CMD_WRITE & ISCSI_FLAG_CMD_FINAL not set. Bad iSCSI Initiator. Jun 19 12:58:31 lio-dev kernel: [ 93.272827] ISCSI_FLAG_CMD_WRITE & ISCSI_FLAG_CMD_FINAL not set. Bad iSCSI Initiator. Jun 19 12:58:31 lio-dev kernel: [ 93.272828] ISCSI_FLAG_CMD_WRITE & ISCSI_FLAG_CMD_FINAL not set. Bad iSCSI Initiator. Jun 19 12:58:31 lio-dev kernel: [ 93.272830] ISCSI_FLAG_CMD_WRITE & ISCSI_FLAG_CMD_FINAL not set. Bad iSCSI Initiator. Jun 19 12:58:31 lio-dev kernel: [ 93.272831] ISCSI_FLAG_CMD_WRITE & ISCSI_FLAG_CMD_FINAL not set. Bad iSCSI Initiator. Jun 19 12:58:31 lio-dev kernel: [ 93.272832] ISCSI_FLAG_CMD_WRITE & ISCSI_FLAG_CMD_FINAL not set. Bad iSCSI Initiator. Jun 19 12:58:31 lio-dev kernel: [ 93.272834] ISCSI_FLAG_CMD_WRITE & ISCSI_FLAG_CMD_FINAL not set. Bad iSCSI Initiator. Jun 19 12:58:31 lio-dev kernel: [ 93.272835] ISCSI_FLAG_CMD_WRITE & ISCSI_FLAG_CMD_FINAL not set. Bad iSCSI Initiator. Jun 19 12:58:31 lio-dev kernel: [ 93.272837] ISCSI_FLAG_CMD_WRITE & ISCSI_FLAG_CMD_FINAL not set. Bad iSCSI Initiator. Jun 19 12:58:31 lio-dev kernel: [ 93.273657] ISCSI_FLAG_CMD_WRITE & ISCSI_FLAG_CMD_FINAL not set. Bad iSCSI Initiator. Jun 19 12:58:31 lio-dev kernel: [ 93.274336] ITT: 0x00000000 response_queue_count: -1 Jun 19 12:58:32 lio-dev kernel: [ 93.960399] Command ITT: 0xa3a3a3a3 received DataOUT for a NON-WRITE command. Jun 19 12:58:32 lio-dev kernel: [ 93.960407] ISCSI_FLAG_CMD_WRITE & ISCSI_FLAG_CMD_FINAL not set. Bad iSCSI Initiator. Jun 19 12:58:32 lio-dev kernel: [ 93.960409] ISCSI_FLAG_CMD_WRITE & ISCSI_FLAG_CMD_FINAL not set. Bad iSCSI Initiator. Jun 19 12:58:32 lio-dev kernel: [ 93.960410] ISCSI_FLAG_CMD_WRITE & ISCSI_FLAG_CMD_FINAL not set. Bad iSCSI Initiator. Jun 19 12:58:32 lio-dev kernel: [ 93.960411] ISCSI_FLAG_CMD_WRITE & ISCSI_FLAG_CMD_FINAL not set. Bad iSCSI Initiator. Jun 19 12:58:32 lio-dev kernel: [ 93.960412] ISCSI_FLAG_CMD_WRITE & ISCSI_FLAG_CMD_FINAL not set. Bad iSCSI Initiator. Jun 19 12:58:32 lio-dev kernel: [ 93.960414] ISCSI_FLAG_CMD_WRITE & ISCSI_FLAG_CMD_FINAL not set. Bad iSCSI Initiator. Jun 19 12:58:32 lio-dev kernel: [ 93.960415] ISCSI_FLAG_CMD_WRITE & ISCSI_FLAG_CMD_FINAL not set. Bad iSCSI Initiator. Jun 19 12:58:32 lio-dev kernel: [ 93.960416] ISCSI_FLAG_CMD_WRITE & ISCSI_FLAG_CMD_FINAL not set. Bad iSCSI Initiator. Jun 19 12:58:32 lio-dev kernel: [ 93.960417] ISCSI_FLAG_CMD_WRITE & ISCSI_FLAG_CMD_FINAL not set. Bad iSCSI Initiator. Jun 19 12:58:32 lio-dev kernel: [ 93.960418] ISCSI_FLAG_CMD_WRITE & ISCSI_FLAG_CMD_FINAL not set. Bad iSCSI Initiator. Jun 19 12:58:32 lio-dev kernel: [ 93.961346] ISCSI_FLAG_CMD_WRITE & ISCSI_FLAG_CMD_FINAL not set. Bad iSCSI Initiator. Jun 19 12:58:32 lio-dev kernel: [ 93.961934] ITT: 0x00000000 response_queue_count: -1 Jun 19 12:58:32 lio-dev kernel: [ 94.578680] ------------[ cut here ]------------ Jun 19 12:58:32 lio-dev kernel: [ 94.578686] WARNING: CPU: 0 PID: 622 at lib/list_debug.c:33 __list_add+0xbe/0xd0() Jun 19 12:58:32 lio-dev kernel: [ 94.578688] list_add corruption. prev->next should be next (ffff88003cd41100), but was (null). (prev=ffff88003bf271e0). Jun 19 12:58:32 lio-dev kernel: [ 94.578689] Modules linked in: target_core_pscsi target_core_file target_core_iblock iscsi_target_mod target_core_mod ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter bnep bluetooth ip6_tables 6lowpan_iphc rfkill joydev snd_intel8x0 snd_ac97_codec ac97_bus snd_seq snd_seq_device snd_pcm ppdev pcspkr i2c_piix4 i2c_core microcode e1000 snd_timer snd soundcore parport_pc parport uinput Jun 19 12:58:32 lio-dev kernel: [ 94.578706] CPU: 0 PID: 622 Comm: iscsi_trx Not tainted 3.14.0-rc6+ #3 Jun 19 12:58:32 lio-dev kernel: [ 94.578707] Hardware name: innotek GmbH VirtualBox, BIOS VirtualBox 12/01/2006 Jun 19 12:58:32 lio-dev kernel: [ 94.578708] 0000000000000009 ffff88003c0bdbd8 ffffffff81693b55 0000000000000007 Jun 19 12:58:32 lio-dev kernel: [ 94.578710] ffff88003c0bdc28 ffff88003c0bdc18 ffffffff8106d82c ffff88003c2e3f00 Jun 19 12:58:32 lio-dev kernel: [ 94.578711] ffff88003bf284e0 ffff88003cd41100 ffff88003bf271e0 0000000000000246
Jun 19 12:58:32 lio-dev kernel: [ 94.578713] Call Trace:Jun 19 12:58:32 lio-dev kernel: [ 94.578717] [<ffffffff81693b55>] dump_stack+0x46/0x58 Jun 19 12:58:32 lio-dev kernel: [ 94.578719] [<ffffffff8106d82c>] warn_slowpath_common+0x8c/0xc0 Jun 19 12:58:32 lio-dev kernel: [ 94.578721] [<ffffffff8106d916>] warn_slowpath_fmt+0x46/0x50 Jun 19 12:58:32 lio-dev kernel: [ 94.578733] [<ffffffffa01d5dba>] ? target_alloc_sgl+0xca/0x150 [target_core_mod] Jun 19 12:58:32 lio-dev kernel: [ 94.578735] [<ffffffff8133a07e>] __list_add+0xbe/0xd0 Jun 19 12:58:32 lio-dev kernel: [ 94.578740] [<ffffffffa01d5f52>] transport_generic_new_cmd+0x112/0x1b0 [target_core_mod] Jun 19 12:58:32 lio-dev kernel: [ 94.578745] [<ffffffffa01d6034>] transport_handle_cdb_direct+0x44/0xa0 [target_core_mod] Jun 19 12:58:32 lio-dev kernel: [ 94.578752] [<ffffffffa0231404>] iscsit_execute_cmd+0x264/0x290 [iscsi_target_mod] Jun 19 12:58:32 lio-dev kernel: [ 94.578757] [<ffffffffa023a44d>] iscsit_sequence_cmd+0xcd/0x1b0 [iscsi_target_mod] Jun 19 12:58:32 lio-dev kernel: [ 94.578762] [<ffffffffa023d798>] iscsit_process_scsi_cmd+0x98/0x120 [iscsi_target_mod] Jun 19 12:58:32 lio-dev kernel: [ 94.578766] [<ffffffffa0240c79>] iscsi_target_rx_thread+0x569/0xf00 [iscsi_target_mod] Jun 19 12:58:32 lio-dev kernel: [ 94.578769] [<ffffffff810a9274>] ? dequeue_task_fair+0x314/0x4e0 Jun 19 12:58:32 lio-dev kernel: [ 94.578771] [<ffffffff810a452f>] ? __dequeue_entity+0x2f/0x50 Jun 19 12:58:32 lio-dev kernel: [ 94.578773] [<ffffffff8169722e>] ? __schedule+0x2de/0x740 Jun 19 12:58:32 lio-dev kernel: [ 94.578777] [<ffffffffa0240710>] ? iscsi_target_tx_thread+0x210/0x210 [iscsi_target_mod] Jun 19 12:58:32 lio-dev kernel: [ 94.578779] [<ffffffff8108fec9>] kthread+0xc9/0xe0 Jun 19 12:58:32 lio-dev kernel: [ 94.578780] [<ffffffff8108fe00>] ? flush_kthread_worker+0x80/0x80 Jun 19 12:58:32 lio-dev kernel: [ 94.578783] [<ffffffff816a2fac>] ret_from_fork+0x7c/0xb0 Jun 19 12:58:32 lio-dev kernel: [ 94.578785] [<ffffffff8108fe00>] ? flush_kthread_worker+0x80/0x80 Jun 19 12:58:32 lio-dev kernel: [ 94.578786] ---[ end trace 3d9b91cfbeb0cb8d ]--- Jun 19 12:58:32 lio-dev kernel: [ 94.582746] ------------[ cut here ]------------ Jun 19 12:58:32 lio-dev kernel: [ 94.582750] WARNING: CPU: 0 PID: 622 at lib/list_debug.c:33 __list_add+0xbe/0xd0() Jun 19 12:58:32 lio-dev kernel: [ 94.582751] list_add corruption. prev->next should be next (ffff88003cd41100), but was ffff88003bf271e0. (prev=ffff88003bf271e0). Jun 19 12:58:32 lio-dev kernel: [ 94.582751] Modules linked in: target_core_pscsi target_core_file target_core_iblock iscsi_target_mod target_core_mod ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter bnep bluetooth ip6_tables 6lowpan_iphc rfkill joydev snd_intel8x0 snd_ac97_codec ac97_bus snd_seq snd_seq_device snd_pcm ppdev pcspkr i2c_piix4 i2c_core microcode e1000 snd_timer snd soundcore parport_pc parport uinput Jun 19 12:58:32 lio-dev kernel: [ 94.582762] CPU: 0 PID: 622 Comm: iscsi_trx Tainted: G W 3.14.0-rc6+ #3 Jun 19 12:58:32 lio-dev kernel: [ 94.582763] Hardware name: innotek GmbH VirtualBox, BIOS VirtualBox 12/01/2006 Jun 19 12:58:32 lio-dev kernel: [ 94.582763] 0000000000000009 ffff88003c0bdbd8 ffffffff81693b55 0000000000000007 Jun 19 12:58:32 lio-dev kernel: [ 94.582765] ffff88003c0bdc28 ffff88003c0bdc18 ffffffff8106d82c ffff88003c2e3f00 Jun 19 12:58:32 lio-dev kernel: [ 94.582766] ffff88003bf271e0 ffff88003cd41100 ffff88003bf271e0 0000000000000246
Jun 19 12:58:32 lio-dev kernel: [ 94.582768] Call Trace:Jun 19 12:58:32 lio-dev kernel: [ 94.582770] [<ffffffff81693b55>] dump_stack+0x46/0x58 Jun 19 12:58:32 lio-dev kernel: [ 94.582771] [<ffffffff8106d82c>] warn_slowpath_common+0x8c/0xc0 Jun 19 12:58:32 lio-dev kernel: [ 94.582773] [<ffffffff8106d916>] warn_slowpath_fmt+0x46/0x50 Jun 19 12:58:32 lio-dev kernel: [ 94.582779] [<ffffffffa01d5dba>] ? target_alloc_sgl+0xca/0x150 [target_core_mod] Jun 19 12:58:32 lio-dev kernel: [ 94.582781] [<ffffffff8133a07e>] __list_add+0xbe/0xd0 Jun 19 12:58:32 lio-dev kernel: [ 94.582786] [<ffffffffa01d5f52>] transport_generic_new_cmd+0x112/0x1b0 [target_core_mod] Jun 19 12:58:32 lio-dev kernel: [ 94.582791] [<ffffffffa01d6034>] transport_handle_cdb_direct+0x44/0xa0 [target_core_mod] Jun 19 12:58:32 lio-dev kernel: [ 94.582796] [<ffffffffa0231404>] iscsit_execute_cmd+0x264/0x290 [iscsi_target_mod] Jun 19 12:58:32 lio-dev kernel: [ 94.582800] [<ffffffffa023a44d>] iscsit_sequence_cmd+0xcd/0x1b0 [iscsi_target_mod] Jun 19 12:58:32 lio-dev kernel: [ 94.582805] [<ffffffffa023d798>] iscsit_process_scsi_cmd+0x98/0x120 [iscsi_target_mod] Jun 19 12:58:32 lio-dev kernel: [ 94.582809] [<ffffffffa0240c79>] iscsi_target_rx_thread+0x569/0xf00 [iscsi_target_mod] Jun 19 12:58:32 lio-dev kernel: [ 94.582811] [<ffffffff810a9274>] ? dequeue_task_fair+0x314/0x4e0 Jun 19 12:58:32 lio-dev kernel: [ 94.582812] [<ffffffff810a452f>] ? __dequeue_entity+0x2f/0x50 Jun 19 12:58:32 lio-dev kernel: [ 94.582814] [<ffffffff8169722e>] ? __schedule+0x2de/0x740 Jun 19 12:58:32 lio-dev kernel: [ 94.582818] [<ffffffffa0240710>] ? iscsi_target_tx_thread+0x210/0x210 [iscsi_target_mod] Jun 19 12:58:32 lio-dev kernel: [ 94.582820] [<ffffffff8108fec9>] kthread+0xc9/0xe0 Jun 19 12:58:32 lio-dev kernel: [ 94.582822] [<ffffffff8108fe00>] ? flush_kthread_worker+0x80/0x80 Jun 19 12:58:32 lio-dev kernel: [ 94.582824] [<ffffffff816a2fac>] ret_from_fork+0x7c/0xb0 Jun 19 12:58:32 lio-dev kernel: [ 94.582825] [<ffffffff8108fe00>] ? flush_kthread_worker+0x80/0x80 Jun 19 12:58:32 lio-dev kernel: [ 94.582826] ---[ end trace 3d9b91cfbeb0cb8e ]--- Jun 19 12:58:32 lio-dev kernel: [ 94.582827] ------------[ cut here ]------------ Jun 19 12:58:32 lio-dev kernel: [ 94.582828] WARNING: CPU: 0 PID: 622 at lib/list_debug.c:36 __list_add+0x9c/0xd0() Jun 19 12:58:32 lio-dev kernel: [ 94.582829] list_add double add: new=ffff88003bf271e0, prev=ffff88003bf271e0, next=ffff88003cd41100. Jun 19 12:58:32 lio-dev kernel: [ 94.582830] Modules linked in: target_core_pscsi target_core_file target_core_iblock iscsi_target_mod target_core_mod ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter bnep bluetooth ip6_tables 6lowpan_iphc rfkill joydev snd_intel8x0 snd_ac97_codec ac97_bus snd_seq snd_seq_device snd_pcm ppdev pcspkr i2c_piix4 i2c_core microcode e1000 snd_timer snd soundcore parport_pc parport uinput Jun 19 12:58:32 lio-dev kernel: [ 94.582838] CPU: 0 PID: 622 Comm: iscsi_trx Tainted: G W 3.14.0-rc6+ #3 Jun 19 12:58:32 lio-dev kernel: [ 94.582839] Hardware name: innotek GmbH VirtualBox, BIOS VirtualBox 12/01/2006 Jun 19 12:58:32 lio-dev kernel: [ 94.582840] 0000000000000009 ffff88003c0bdbd8 ffffffff81693b55 0000000000000007 Jun 19 12:58:32 lio-dev kernel: [ 94.582841] ffff88003c0bdc28 ffff88003c0bdc18 ffffffff8106d82c ffff88003c2e3f00 Jun 19 12:58:32 lio-dev kernel: [ 94.582842] ffff88003bf271e0 ffff88003cd41100 ffff88003bf271e0 0000000000000246
Jun 19 12:58:32 lio-dev kernel: [ 94.582843] Call Trace:Jun 19 12:58:32 lio-dev kernel: [ 94.582845] [<ffffffff81693b55>] dump_stack+0x46/0x58 Jun 19 12:58:32 lio-dev kernel: [ 94.582846] [<ffffffff8106d82c>] warn_slowpath_common+0x8c/0xc0 Jun 19 12:58:32 lio-dev kernel: [ 94.582848] [<ffffffff8106d916>] warn_slowpath_fmt+0x46/0x50 Jun 19 12:58:32 lio-dev kernel: [ 94.582853] [<ffffffffa01d5dba>] ? target_alloc_sgl+0xca/0x150 [target_core_mod] Jun 19 12:58:32 lio-dev kernel: [ 94.582854] [<ffffffff8133a05c>] __list_add+0x9c/0xd0 Jun 19 12:58:32 lio-dev kernel: [ 94.582859] [<ffffffffa01d5f52>] transport_generic_new_cmd+0x112/0x1b0 [target_core_mod] Jun 19 12:58:32 lio-dev kernel: [ 94.582864] [<ffffffffa01d6034>] transport_handle_cdb_direct+0x44/0xa0 [target_core_mod] Jun 19 12:58:32 lio-dev kernel: [ 94.582868] [<ffffffffa0231404>] iscsit_execute_cmd+0x264/0x290 [iscsi_target_mod] Jun 19 12:58:32 lio-dev kernel: [ 94.582883] [<ffffffffa023a44d>] iscsit_sequence_cmd+0xcd/0x1b0 [iscsi_target_mod] Jun 19 12:58:32 lio-dev kernel: [ 94.582889] [<ffffffffa023d798>] iscsit_process_scsi_cmd+0x98/0x120 [iscsi_target_mod] Jun 19 12:58:32 lio-dev kernel: [ 94.582894] [<ffffffffa0240c79>] iscsi_target_rx_thread+0x569/0xf00 [iscsi_target_mod] Jun 19 12:58:32 lio-dev kernel: [ 94.582896] [<ffffffff810a9274>] ? dequeue_task_fair+0x314/0x4e0 Jun 19 12:58:32 lio-dev kernel: [ 94.582898] [<ffffffff810a452f>] ? __dequeue_entity+0x2f/0x50 Jun 19 12:58:32 lio-dev kernel: [ 94.582900] [<ffffffff8169722e>] ? __schedule+0x2de/0x740 Jun 19 12:58:32 lio-dev kernel: [ 94.582905] [<ffffffffa0240710>] ? iscsi_target_tx_thread+0x210/0x210 [iscsi_target_mod] Jun 19 12:58:32 lio-dev kernel: [ 94.582907] [<ffffffff8108fec9>] kthread+0xc9/0xe0 Jun 19 12:58:32 lio-dev kernel: [ 94.582909] [<ffffffff8108fe00>] ? flush_kthread_worker+0x80/0x80 Jun 19 12:58:32 lio-dev kernel: [ 94.582912] [<ffffffff816a2fac>] ret_from_fork+0x7c/0xb0 Jun 19 12:58:32 lio-dev kernel: [ 94.582913] [<ffffffff8108fe00>] ? flush_kthread_worker+0x80/0x80 Jun 19 12:58:32 lio-dev kernel: [ 94.582915] ---[ end trace 3d9b91cfbeb0cb8f ]--- Jun 19 12:58:32 lio-dev kernel: [ 94.583506] ------------[ cut here ]------------ Jun 19 12:58:32 lio-dev kernel: [ 94.583510] WARNING: CPU: 0 PID: 620 at lib/list_debug.c:33 __list_add+0xbe/0xd0() Jun 19 12:58:32 lio-dev kernel: [ 94.583511] list_add corruption. prev->next should be next (ffff88003cd41100), but was dead000000100100. (prev=ffff88003bf271e0). Jun 19 12:58:32 lio-dev kernel: [ 94.583511] Modules linked in: target_core_pscsi target_core_file target_core_iblock iscsi_target_mod target_core_mod ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter bnep bluetooth ip6_tables 6lowpan_iphc rfkill joydev snd_intel8x0 snd_ac97_codec ac97_bus snd_seq snd_seq_device snd_pcm ppdev pcspkr i2c_piix4 i2c_core microcode e1000 snd_timer snd soundcore parport_pc parport uinput Jun 19 12:58:32 lio-dev kernel: [ 94.583521] CPU: 0 PID: 620 Comm: iscsi_trx Tainted: G W 3.14.0-rc6+ #3 Jun 19 12:58:32 lio-dev kernel: [ 94.583522] Hardware name: innotek GmbH VirtualBox, BIOS VirtualBox 12/01/2006 Jun 19 12:58:32 lio-dev kernel: [ 94.583523] 0000000000000009 ffff88003d33dbd8 ffffffff81693b55 0000000000000007 Jun 19 12:58:32 lio-dev kernel: [ 94.583524] ffff88003d33dc28 ffff88003d33dc18 ffffffff8106d82c ffff88003c2e5100 Jun 19 12:58:32 lio-dev kernel: [ 94.583525] ffff88003bf276a0 ffff88003cd41100 ffff88003bf271e0 0000000000000246
Jun 19 12:58:32 lio-dev kernel: [ 94.583527] Call Trace:Jun 19 12:58:32 lio-dev kernel: [ 94.583528] [<ffffffff81693b55>] dump_stack+0x46/0x58 Jun 19 12:58:32 lio-dev kernel: [ 94.583530] [<ffffffff8106d82c>] warn_slowpath_common+0x8c/0xc0 Jun 19 12:58:32 lio-dev kernel: [ 94.583531] [<ffffffff8106d916>] warn_slowpath_fmt+0x46/0x50 Jun 19 12:58:32 lio-dev kernel: [ 94.583537] [<ffffffffa01d5dba>] ? target_alloc_sgl+0xca/0x150 [target_core_mod] Jun 19 12:58:32 lio-dev kernel: [ 94.583539] [<ffffffff8133a07e>] __list_add+0xbe/0xd0 Jun 19 12:58:32 lio-dev kernel: [ 94.583544] [<ffffffffa01d5f52>] transport_generic_new_cmd+0x112/0x1b0 [target_core_mod] Jun 19 12:58:32 lio-dev kernel: [ 94.583548] [<ffffffffa01d6034>] transport_handle_cdb_direct+0x44/0xa0 [target_core_mod] Jun 19 12:58:32 lio-dev kernel: [ 94.583553] [<ffffffffa0231404>] iscsit_execute_cmd+0x264/0x290 [iscsi_target_mod] Jun 19 12:58:32 lio-dev kernel: [ 94.583557] [<ffffffffa023a44d>] iscsit_sequence_cmd+0xcd/0x1b0 [iscsi_target_mod] Jun 19 12:58:32 lio-dev kernel: [ 94.583562] [<ffffffffa023d798>] iscsit_process_scsi_cmd+0x98/0x120 [iscsi_target_mod] Jun 19 12:58:32 lio-dev kernel: [ 94.583566] [<ffffffffa0240c79>] iscsi_target_rx_thread+0x569/0xf00 [iscsi_target_mod] Jun 19 12:58:32 lio-dev kernel: [ 94.583568] [<ffffffff810a9274>] ? dequeue_task_fair+0x314/0x4e0 Jun 19 12:58:32 lio-dev kernel: [ 94.583570] [<ffffffff8169722e>] ? __schedule+0x2de/0x740 Jun 19 12:58:32 lio-dev kernel: [ 94.583585] [<ffffffffa0240710>] ? iscsi_target_tx_thread+0x210/0x210 [iscsi_target_mod] Jun 19 12:58:32 lio-dev kernel: [ 94.583587] [<ffffffff8108fec9>] kthread+0xc9/0xe0 Jun 19 12:58:32 lio-dev kernel: [ 94.583589] [<ffffffff8108fe00>] ? flush_kthread_worker+0x80/0x80 Jun 19 12:58:32 lio-dev kernel: [ 94.583591] [<ffffffff816a2fac>] ret_from_fork+0x7c/0xb0 Jun 19 12:58:32 lio-dev kernel: [ 94.583593] [<ffffffff8108fe00>] ? flush_kthread_worker+0x80/0x80 Jun 19 12:58:32 lio-dev kernel: [ 94.583594] ---[ end trace 3d9b91cfbeb0cb90 ]--- Jun 19 12:58:32 lio-dev kernel: [ 94.586795] ------------[ cut here ]------------ Jun 19 12:58:32 lio-dev kernel: [ 94.586799] WARNING: CPU: 0 PID: 620 at lib/list_debug.c:33 __list_add+0xbe/0xd0() Jun 19 12:58:32 lio-dev kernel: [ 94.586801] list_add corruption. prev->next should be next (ffff88003cd41100), but was ffff88003bf271e0. (prev=ffff88003bf271e0). Jun 19 12:58:32 lio-dev kernel: [ 94.586801] Modules linked in: target_core_pscsi target_core_file target_core_iblock iscsi_target_mod target_core_mod ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter bnep bluetooth ip6_tables 6lowpan_iphc rfkill joydev snd_intel8x0 snd_ac97_codec ac97_bus snd_seq snd_seq_device snd_pcm ppdev pcspkr i2c_piix4 i2c_core microcode e1000 snd_timer snd soundcore parport_pc parport uinput Jun 19 12:58:32 lio-dev kernel: [ 94.586812] CPU: 0 PID: 620 Comm: iscsi_trx Tainted: G W 3.14.0-rc6+ #3 Jun 19 12:58:32 lio-dev kernel: [ 94.586813] Hardware name: innotek GmbH VirtualBox, BIOS VirtualBox 12/01/2006 Jun 19 12:58:32 lio-dev kernel: [ 94.586814] 0000000000000009 ffff88003d33dbd8 ffffffff81693b55 0000000000000007 Jun 19 12:58:32 lio-dev kernel: [ 94.586816] ffff88003d33dc28 ffff88003d33dc18 ffffffff8106d82c ffff88003c2e5100 Jun 19 12:58:32 lio-dev kernel: [ 94.586817] ffff88003bf271e0 ffff88003cd41100 ffff88003bf271e0 0000000000000246
Jun 19 12:58:32 lio-dev kernel: [ 94.586818] Call Trace:Jun 19 12:58:32 lio-dev kernel: [ 94.586820] [<ffffffff81693b55>] dump_stack+0x46/0x58 Jun 19 12:58:32 lio-dev kernel: [ 94.586822] [<ffffffff8106d82c>] warn_slowpath_common+0x8c/0xc0 Jun 19 12:58:32 lio-dev kernel: [ 94.586824] [<ffffffff8106d916>] warn_slowpath_fmt+0x46/0x50 Jun 19 12:58:32 lio-dev kernel: [ 94.586842] [<ffffffffa01d5dba>] ? target_alloc_sgl+0xca/0x150 [target_core_mod] Jun 19 12:58:32 lio-dev kernel: [ 94.586844] [<ffffffff8133a07e>] __list_add+0xbe/0xd0 Jun 19 12:58:32 lio-dev kernel: [ 94.586851] [<ffffffffa01d5f52>] transport_generic_new_cmd+0x112/0x1b0 [target_core_mod] Jun 19 12:58:32 lio-dev kernel: [ 94.586857] [<ffffffffa01d6034>] transport_handle_cdb_direct+0x44/0xa0 [target_core_mod] Jun 19 12:58:32 lio-dev kernel: [ 94.586863] [<ffffffffa0231404>] iscsit_execute_cmd+0x264/0x290 [iscsi_target_mod] Jun 19 12:58:32 lio-dev kernel: [ 94.586869] [<ffffffffa023a44d>] iscsit_sequence_cmd+0xcd/0x1b0 [iscsi_target_mod] Jun 19 12:58:32 lio-dev kernel: [ 94.586875] [<ffffffffa023d798>] iscsit_process_scsi_cmd+0x98/0x120 [iscsi_target_mod] Jun 19 12:58:32 lio-dev kernel: [ 94.586880] [<ffffffffa0240c79>] iscsi_target_rx_thread+0x569/0xf00 [iscsi_target_mod] Jun 19 12:58:32 lio-dev kernel: [ 94.586883] [<ffffffff810a9274>] ? dequeue_task_fair+0x314/0x4e0 Jun 19 12:58:32 lio-dev kernel: [ 94.586885] [<ffffffff8169722e>] ? __schedule+0x2de/0x740 Jun 19 12:58:32 lio-dev kernel: [ 94.586890] [<ffffffffa0240710>] ? iscsi_target_tx_thread+0x210/0x210 [iscsi_target_mod] Jun 19 12:58:32 lio-dev kernel: [ 94.586893] [<ffffffff8108fec9>] kthread+0xc9/0xe0 Jun 19 12:58:32 lio-dev kernel: [ 94.586895] [<ffffffff8108fe00>] ? flush_kthread_worker+0x80/0x80 Jun 19 12:58:32 lio-dev kernel: [ 94.586897] [<ffffffff816a2fac>] ret_from_fork+0x7c/0xb0 Jun 19 12:58:32 lio-dev kernel: [ 94.586899] [<ffffffff8108fe00>] ? flush_kthread_worker+0x80/0x80 Jun 19 12:58:32 lio-dev kernel: [ 94.586900] ---[ end trace 3d9b91cfbeb0cb91 ]--- Jun 19 12:58:32 lio-dev kernel: [ 94.586901] ------------[ cut here ]------------ Jun 19 12:58:32 lio-dev kernel: [ 94.586903] WARNING: CPU: 0 PID: 620 at lib/list_debug.c:36 __list_add+0x9c/0xd0() Jun 19 12:58:32 lio-dev kernel: [ 94.586904] list_add double add: new=ffff88003bf271e0, prev=ffff88003bf271e0, next=ffff88003cd41100. Jun 19 12:58:32 lio-dev kernel: [ 94.586905] Modules linked in: target_core_pscsi target_core_file target_core_iblock iscsi_target_mod target_core_mod ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter bnep bluetooth ip6_tables 6lowpan_iphc rfkill joydev snd_intel8x0 snd_ac97_codec ac97_bus snd_seq snd_seq_device snd_pcm ppdev pcspkr i2c_piix4 i2c_core microcode e1000 snd_timer snd soundcore parport_pc parport uinput Jun 19 12:58:32 lio-dev kernel: [ 94.586915] CPU: 0 PID: 620 Comm: iscsi_trx Tainted: G W 3.14.0-rc6+ #3 Jun 19 12:58:32 lio-dev kernel: [ 94.586916] Hardware name: innotek GmbH VirtualBox, BIOS VirtualBox 12/01/2006 Jun 19 12:58:32 lio-dev kernel: [ 94.586917] 0000000000000009 ffff88003d33dbd8 ffffffff81693b55 0000000000000007 Jun 19 12:58:32 lio-dev kernel: [ 94.586919] ffff88003d33dc28 ffff88003d33dc18 ffffffff8106d82c ffff88003c2e5100 Jun 19 12:58:32 lio-dev kernel: [ 94.586920] ffff88003bf271e0 ffff88003cd41100 ffff88003bf271e0 0000000000000246
Jun 19 12:58:32 lio-dev kernel: [ 94.586922] Call Trace:Jun 19 12:58:32 lio-dev kernel: [ 94.586924] [<ffffffff81693b55>] dump_stack+0x46/0x58 Jun 19 12:58:32 lio-dev kernel: [ 94.586925] [<ffffffff8106d82c>] warn_slowpath_common+0x8c/0xc0 Jun 19 12:58:32 lio-dev kernel: [ 94.586927] [<ffffffff8106d916>] warn_slowpath_fmt+0x46/0x50 Jun 19 12:58:32 lio-dev kernel: [ 94.586934] [<ffffffffa01d5dba>] ? target_alloc_sgl+0xca/0x150 [target_core_mod] Jun 19 12:58:32 lio-dev kernel: [ 94.586936] [<ffffffff8133a05c>] __list_add+0x9c/0xd0 Jun 19 12:58:32 lio-dev kernel: [ 94.586942] [<ffffffffa01d5f52>] transport_generic_new_cmd+0x112/0x1b0 [target_core_mod] Jun 19 12:58:32 lio-dev kernel: [ 94.586947] [<ffffffffa01d6034>] transport_handle_cdb_direct+0x44/0xa0 [target_core_mod] Jun 19 12:58:32 lio-dev kernel: [ 94.586953] [<ffffffffa0231404>] iscsit_execute_cmd+0x264/0x290 [iscsi_target_mod] Jun 19 12:58:32 lio-dev kernel: [ 94.586959] [<ffffffffa023a44d>] iscsit_sequence_cmd+0xcd/0x1b0 [iscsi_target_mod] Jun 19 12:58:32 lio-dev kernel: [ 94.586964] [<ffffffffa023d798>] iscsit_process_scsi_cmd+0x98/0x120 [iscsi_target_mod] Jun 19 12:58:32 lio-dev kernel: [ 94.586969] [<ffffffffa0240c79>] iscsi_target_rx_thread+0x569/0xf00 [iscsi_target_mod] Jun 19 12:58:32 lio-dev kernel: [ 94.586972] [<ffffffff810a9274>] ? dequeue_task_fair+0x314/0x4e0 Jun 19 12:58:32 lio-dev kernel: [ 94.586985] [<ffffffff8169722e>] ? __schedule+0x2de/0x740 Jun 19 12:58:32 lio-dev kernel: [ 94.586990] [<ffffffffa0240710>] ? iscsi_target_tx_thread+0x210/0x210 [iscsi_target_mod] Jun 19 12:58:32 lio-dev kernel: [ 94.586992] [<ffffffff8108fec9>] kthread+0xc9/0xe0 Jun 19 12:58:32 lio-dev kernel: [ 94.586994] [<ffffffff8108fe00>] ? flush_kthread_worker+0x80/0x80 Jun 19 12:58:32 lio-dev kernel: [ 94.586997] [<ffffffff816a2fac>] ret_from_fork+0x7c/0xb0 Jun 19 12:58:32 lio-dev kernel: [ 94.586998] [<ffffffff8108fe00>] ? flush_kthread_worker+0x80/0x80 Jun 19 12:58:32 lio-dev kernel: [ 94.587000] ---[ end trace 3d9b91cfbeb0cb92 ]--- Jun 19 12:58:32 lio-dev kernel: [ 94.587262] Command ITT: 0xa3a3a3a3 received DataOUT for a NON-WRITE command. Jun 19 12:58:32 lio-dev kernel: [ 94.587267] ISCSI_FLAG_CMD_WRITE & ISCSI_FLAG_CMD_FINAL not set. Bad iSCSI Initiator. Jun 19 12:58:32 lio-dev kernel: [ 94.587269] ISCSI_FLAG_CMD_WRITE & ISCSI_FLAG_CMD_FINAL not set. Bad iSCSI Initiator. Jun 19 12:58:32 lio-dev kernel: [ 94.587270] ISCSI_FLAG_CMD_WRITE & ISCSI_FLAG_CMD_FINAL not set. Bad iSCSI Initiator. Jun 19 12:58:32 lio-dev kernel: [ 94.587271] ISCSI_FLAG_CMD_WRITE & ISCSI_FLAG_CMD_FINAL not set. Bad iSCSI Initiator. Jun 19 12:58:32 lio-dev kernel: [ 94.587272] ISCSI_FLAG_CMD_WRITE & ISCSI_FLAG_CMD_FINAL not set. Bad iSCSI Initiator. Jun 19 12:58:32 lio-dev kernel: [ 94.587273] ISCSI_FLAG_CMD_WRITE & ISCSI_FLAG_CMD_FINAL not set. Bad iSCSI Initiator. Jun 19 12:58:32 lio-dev kernel: [ 94.587274] ISCSI_FLAG_CMD_WRITE & ISCSI_FLAG_CMD_FINAL not set. Bad iSCSI Initiator. Jun 19 12:58:32 lio-dev kernel: [ 94.587275] ISCSI_FLAG_CMD_WRITE & ISCSI_FLAG_CMD_FINAL not set. Bad iSCSI Initiator. Jun 19 12:58:32 lio-dev kernel: [ 94.587276] ISCSI_FLAG_CMD_WRITE & ISCSI_FLAG_CMD_FINAL not set. Bad iSCSI Initiator. Jun 19 12:58:32 lio-dev kernel: [ 94.587277] ISCSI_FLAG_CMD_WRITE & ISCSI_FLAG_CMD_FINAL not set. Bad iSCSI Initiator. Jun 19 12:58:32 lio-dev kernel: [ 94.588171] ISCSI_FLAG_CMD_WRITE & ISCSI_FLAG_CMD_FINAL not set. Bad iSCSI Initiator. Jun 19 12:58:32 lio-dev kernel: [ 94.588673] ITT: 0x00000000 response_queue_count: -1
Is there an issue with target not able to retrieve the correct ITT ? I did not get time to look into it.
Please include a wireshark capture, along with iscsi_target_mod pr_debug output enabled.
Attaching the pcap. I am basically opening 2 connections.First being the leading connection.And I am performing a READ10 and WRITE10 operation on both the connections.
Thanks, --nabHere is the Crash Dump [188185.653521] ------------[ cut here ]------------ [188185.653527] WARNING: CPU: 0 PID: 963 at lib/list_debug.c:33 __list_add+0xbe/0xd0() [188185.653529] list_add corruption. prev->next should be next (ffff88003ccf3100), but was ffff8800269276a0. (prev=ffff8800269276a0). [188185.653529] Modules linked in: target_core_pscsi target_core_file target_core_iblock iscsi_target_mod target_core_mod bnep bluetooth 6lowpan_iphc rfkill ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables snd_intel8x0 snd_ac97_codec ac97_bus joydev snd_seq snd_seq_device snd_pcm ppdev pcspkr e1000 microcode parport_pc parport snd_timer snd soundcore i2c_piix4 i2c_core uinput [188185.653545] CPU: 0 PID: 963 Comm: iscsi_trx Tainted: G W 3.14.0-rc6+ #3 [188185.653546] Hardware name: innotek GmbH VirtualBox, BIOS VirtualBox 12/01/2006 [188185.653547] 0000000000000009 ffff88003bbc9bd8 ffffffff81693b55 0000000000000007 [188185.653549] ffff88003bbc9c28 ffff88003bbc9c18 ffffffff8106d82c ffff88003bf8a400 [188185.653551] ffff8800269276a0 ffff88003ccf3100 ffff8800269276a0 0000000000000246 [188185.653552] Call Trace: [188185.653555] [<ffffffff81693b55>] dump_stack+0x46/0x58 [188185.653557] [<ffffffff8106d82c>] warn_slowpath_common+0x8c/0xc0 [188185.653559] [<ffffffff8106d916>] warn_slowpath_fmt+0x46/0x50 [188185.653569] [<ffffffffa01f6dba>] ? target_alloc_sgl+0xca/0x150 [target_core_mod] [188185.653570] [<ffffffff8133a07e>] __list_add+0xbe/0xd0 [188185.653577] [<ffffffffa01f6f52>] transport_generic_new_cmd+0x112/0x1b0 [target_core_mod] [188185.653582] [<ffffffffa01f7034>] transport_handle_cdb_direct+0x44/0xa0 [target_core_mod] [188185.653588] [<ffffffffa023b404>] iscsit_execute_cmd+0x264/0x290 [iscsi_target_mod] [188185.653593] [<ffffffffa024444d>] iscsit_sequence_cmd+0xcd/0x1b0 [iscsi_target_mod] [188185.653598] [<ffffffffa0247798>] iscsit_process_scsi_cmd+0x98/0x120 [iscsi_target_mod] [188185.653602] [<ffffffffa024ac79>] iscsi_target_rx_thread+0x569/0xf00 [iscsi_target_mod] [188185.653605] [<ffffffff810a9274>] ? dequeue_task_fair+0x314/0x4e0 [188185.653607] [<ffffffff8169722e>] ? __schedule+0x2de/0x740 [188185.653611] [<ffffffffa024a710>] ? iscsi_target_tx_thread+0x210/0x210 [iscsi_target_mod] [188185.653613] [<ffffffff8108fec9>] kthread+0xc9/0xe0 [188185.653615] [<ffffffff8108fe00>] ? flush_kthread_worker+0x80/0x80 [188185.653617] [<ffffffff816a2fac>] ret_from_fork+0x7c/0xb0 [188185.653619] [<ffffffff8108fe00>] ? flush_kthread_worker+0x80/0x80 [188185.653620] ---[ end trace 3dd841d40b2b0701 ]--- [188185.653621] ------------[ cut here ]------------ [188185.653622] WARNING: CPU: 0 PID: 963 at lib/list_debug.c:36 __list_add+0x9c/0xd0() [188185.653623] list_add double add: new=ffff8800269276a0, prev=ffff8800269276a0, next=ffff88003ccf3100. [188185.653624] Modules linked in: target_core_pscsi target_core_file target_core_iblock iscsi_target_mod target_core_mod bnep bluetooth 6lowpan_iphc rfkill ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables snd_intel8x0 snd_ac97_codec ac97_bus joydev snd_seq snd_seq_device snd_pcm ppdev pcspkr e1000 microcode parport_pc parport snd_timer snd soundcore i2c_piix4 i2c_core uinput [188185.653633] CPU: 0 PID: 963 Comm: iscsi_trx Tainted: G W 3.14.0-rc6+ #3 [188185.653634] Hardware name: innotek GmbH VirtualBox, BIOS VirtualBox 12/01/2006 [188185.653634] 0000000000000009 ffff88003bbc9bd8 ffffffff81693b55 0000000000000007 [188185.653636] ffff88003bbc9c28 ffff88003bbc9c18 ffffffff8106d82c ffff88003bf8a400 [188185.653637] ffff8800269276a0 ffff88003ccf3100 ffff8800269276a0 0000000000000246 [188185.653638] Call Trace: [188185.653640] [<ffffffff81693b55>] dump_stack+0x46/0x58 [188185.653641] [<ffffffff8106d82c>] warn_slowpath_common+0x8c/0xc0 [188185.653643] [<ffffffff8106d916>] warn_slowpath_fmt+0x46/0x50 [188185.653648] [<ffffffffa01f6dba>] ? target_alloc_sgl+0xca/0x150 [target_core_mod] [188185.653650] [<ffffffff8133a05c>] __list_add+0x9c/0xd0 [188185.653655] [<ffffffffa01f6f52>] transport_generic_new_cmd+0x112/0x1b0 [target_core_mod] [188185.653660] [<ffffffffa01f7034>] transport_handle_cdb_direct+0x44/0xa0 [target_core_mod] [188185.653664] [<ffffffffa023b404>] iscsit_execute_cmd+0x264/0x290 [iscsi_target_mod] [188185.653669] [<ffffffffa024444d>] iscsit_sequence_cmd+0xcd/0x1b0 [iscsi_target_mod] [188185.653674] [<ffffffffa0247798>] iscsit_process_scsi_cmd+0x98/0x120 [iscsi_target_mod] [188185.653678] [<ffffffffa024ac79>] iscsi_target_rx_thread+0x569/0xf00 [iscsi_target_mod] [188185.653680] [<ffffffff810a9274>] ? dequeue_task_fair+0x314/0x4e0 [188185.653682] [<ffffffff8169722e>] ? __schedule+0x2de/0x740 [188185.653686] [<ffffffffa024a710>] ? iscsi_target_tx_thread+0x210/0x210 [iscsi_target_mod] [188185.653688] [<ffffffff8108fec9>] kthread+0xc9/0xe0 [188185.653689] [<ffffffff8108fe00>] ? flush_kthread_worker+0x80/0x80 [188185.653691] [<ffffffff816a2fac>] ret_from_fork+0x7c/0xb0 [188185.653693] [<ffffffff8108fe00>] ? flush_kthread_worker+0x80/0x80 [188185.653694] ---[ end trace 3dd841d40b2b0702 ]--- [188185.653963] Command ITT: 0xa3a3a3a3 received DataOUT after last DataOUT received, dumping payload Kernel Details [root@lio-dev ~]# uname -a Linux lio-dev 3.14.0-rc6+ #3 SMP Thu Apr 10 13:27:04 IST 2014 x86_64 x86_64 x86_64 GNU/Linux [root@lio-dev ~]# cat /sys/kernel/config/target/version Target Engine Core ConfigFS Infrastructure v4.1.0 on Linux/x86_64 on 3.14.0-rc6+ Regards Santosh -- To unsubscribe from this list: send the line "unsubscribe target-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
Attachment:
kernel_crash_write
Description: Binary data
