Re: [PATCH v2] scsi: qla2xxx: Spinlock recursion in qla_target

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



> On Jun 13, 2018, at 6:05 AM, Mikhail Malygin <m.malygin@xxxxxxxxx> wrote:
> 
> Here is the patch used for verification:
> 
> [PATCH] scsi: qla2xxx: Fixup spinlock recursion in qla_target
> 
> The patch reverts changes done in qlt_schedule_sess_for_deletion()
> To avoid spinlock recursion sess->vha->work_lock should be used
> instead of ha->tgt.sess_lock, that can be locked in
> callers: qlt_reset() or qlt_handle_login()
> 

Thanks for testing out the change. 

> Fixes: 1c6cacf4ea6c04 ("scsi: qla2xxx: Fixup locking for session deletion")
> 
> Signed-off-by: Mikhail Malygin <m.malygin@xxxxxxxxx>

I want to see following tags for correctness

Fixes: 1c6cacf4ea6c04 ("scsi: qla2xxx: Fixup locking for session deletion”)
Cc: <stable@xxxxxxxxxxxxxxx> #4.17
Reported-by: Mikhail Malygin <m.malygin@xxxxxxxxx>
Tested-by: Mikhail Malygin <m.malygin@xxxxxxxxx>
Signed-off-by: Himanshu Madhani <himanshu.madhani@xxxxxxxxxx>

> —
> drivers/scsi/qla2xxx/qla_target.c | 6 +++---
> 1 file changed, 3 insertions(+), 3 deletions(-)
> 
> diff --git a/drivers/scsi/qla2xxx/qla_target.c b/drivers/scsi/qla2xxx/qla_target.c
> index 025dc2d3f3de..032897e22e78 100644
> --- a/drivers/scsi/qla2xxx/qla_target.c
> +++ b/drivers/scsi/qla2xxx/qla_target.c
> @@ -1247,16 +1247,16 @@ void qlt_schedule_sess_for_deletion(struct fc_port *sess)
> 			return;
> 	}
> 
> -	spin_lock_irqsave(&ha->tgt.sess_lock, flags);
> 	if (sess->deleted == QLA_SESS_DELETED)
> 		sess->logout_on_delete = 0;
> 
> +	spin_lock_irqsave(&sess->vha->work_lock, flags);
> 	if (sess->deleted == QLA_SESS_DELETION_IN_PROGRESS) {
> -		spin_unlock_irqrestore(&ha->tgt.sess_lock, flags);
> +		spin_unlock_irqrestore(&sess->vha->work_lock, flags);
> 		return;
> 	}
> 	sess->deleted = QLA_SESS_DELETION_IN_PROGRESS;
> -	spin_unlock_irqrestore(&ha->tgt.sess_lock, flags);
> +	spin_unlock_irqrestore(&sess->vha->work_lock, flags);
> 
> 	sess->disc_state = DSC_DELETE_PEND;
> 
> -- 
> 2.15.1
> 
> 
> From: Mikhail Malygin
> Sent: Wednesday, June 13, 2018 2:35 PM
> To: Madhani, Himanshu
> Cc: linux-scsi; Hannes Reinecke; Ivan Tchoub
> Subject: Re: [PATCH v2] scsi: qla2xxx: Spinlock recursion in qla_target
>     
> Hi Himanshu,
> 
> I checked the same scenarios using work_lock instead of sess_lock, and I could not reproduce
> any of the issues mentioned before.
> 
> Thanks,
> Mikhail
> 
> 
> From: Madhani, Himanshu <Himanshu.Madhani@xxxxxxxxxx>
> Sent: Wednesday, June 13, 2018 12:29 AM
> To: Mikhail Malygin
> Cc: linux-scsi; Hannes Reinecke; Ivan Tchoub
> Subject: Re: [PATCH v2] scsi: qla2xxx: Spinlock recursion in qla_target
>     
> Hi Mikail, 
> 
>> On Jun 12, 2018, at 9:08 AM, m.malygin@xxxxxxxxx wrote:
>> 
>> From: Mikhail Malygin <m.malygin@xxxxxxxxx>
>> 
>> This patch addresses issue causing spinlock recursion in qla_target.c:
>> 1. qlt_handle_login takes vha->hw->tgt.sess_lock, then calls qlt_schedule_sess_for_deletion
>> where it tries to take spinlock again.
> 
> We had posted patch to serialize session deletion with following patches
> 
> https://git.kernel.org/pub/scm/linux/kernel/git/mkp/scsi.git/commit/?id=1ae634eb28533b82f9777a47c1ade44cb8c0182b
> 
> https://git.kernel.org/pub/scm/linux/kernel/git/mkp/scsi.git/commit/?id=d8630bb95f46ea118dede63bd75533faa64f9612
> 
> https://git.kernel.org/pub/scm/linux/kernel/git/mkp/scsi.git/commit/?id=d8630bb95f46ea118dede63bd75533faa64f9612
> 
> However, this patch looks like introduced regression,
> 
> https://git.kernel.org/pub/scm/linux/kernel/git/mkp/scsi.git/commit/?id=1c6cacf4ea6c04a58a0e3057f5ed60c24a4ffeff
> 
> can you use work_lock as it was before the change and see if that helps with both issue 1 and 2.
> 
> something like this 
> 
> diff --git a/drivers/scsi/qla2xxx/qla_target.c b/drivers/scsi/qla2xxx/qla_target.c
> index 7ed47800c660..d649f85d9657 100644
> --- a/drivers/scsi/qla2xxx/qla_target.c
> +++ b/drivers/scsi/qla2xxx/qla_target.c
> @@ -1239,16 +1239,16 @@ void qlt_schedule_sess_for_deletion(struct fc_port *sess)
>                         return;
>         }
> 
> -       spin_lock_irqsave(&ha->tgt.sess_lock, flags);
>         if (sess->deleted == QLA_SESS_DELETED)
>                 sess->logout_on_delete = 0;
> 
> +       spin_lock_irqsave(&sess->vha->work_lock, flags);
>         if (sess->deleted == QLA_SESS_DELETION_IN_PROGRESS) {
> -               spin_unlock_irqrestore(&ha->tgt.sess_lock, flags);
> +               spin_unlock_irqrestore(&sess->vha->work_lock, flags);
>                 return;
>         }
>         sess->deleted = QLA_SESS_DELETION_IN_PROGRESS;
> -       spin_unlock_irqrestore(&ha->tgt.sess_lock, flags);
> +       spin_unlock_irqrestore(&sess->vha->work_lock, flags);
> 
>         sess->disc_state = DSC_DELETE_PEND;
> 
>> 2. qlt_reset takes the same lock, then calls qlt_schedule_sess_for_deleteion via qlt_clear_tgt_db
>> 
>> Stacktrace for qlt_handle_login
>> 
>> BUG: spinlock lockup suspected on CPU#0, swapper/0/0
>> lock: 0xc00000c07aa8bec0, .magic: dead4ead, .owner: swapper/0/0, .owner_cpu: 0
>> CPU: 0 PID: 0 Comm: swapper/0 Tainted: G           OE   NX 4.4.132-ttln.24-debug #1
>> Call Trace:
>> [c00000dfff6d7830] [c0000000008060c0] dump_stack+0xb0/0xf0 (unreliable)
>> [c00000dfff6d7870] [c0000000007ff6ec] spin_dump+0xa8/0xc4
>> [c00000dfff6d78e0] [c000000000128320] do_raw_spin_lock+0x140/0x1d0
>> [c00000dfff6d7920] [c0000000007f7354] _raw_spin_lock_irqsave+0x34/0x50
>> [c00000dfff6d7950] [d00000001edf3220] qlt_schedule_sess_for_deletion+0x90/0x250 [qla2xxx]
>> [c00000dfff6d79c0] [d00000001edf6b08] qlt_find_sess_invalidate_other+0x1d8/0x230 [qla2xxx]
>> [c00000dfff6d7a70] [d00000001edf710c] qlt_handle_login+0x5ac/0x760 [qla2xxx]
>> [c00000dfff6d7b10] [d00000001edf7ccc] qlt_handle_imm_notify+0xa0c/0x10b0 [qla2xxx]
>> [c00000dfff6d7c00] [d00000001edf85f0] qlt_24xx_atio_pkt+0x280/0x400 [qla2xxx]
>> [c00000dfff6d7ca0] [d00000001edfa9d8] qlt_24xx_process_atio_queue+0x368/0x7d0 [qla2xxx]
>> [c00000dfff6d7d80] [d00000001edfb898] qla83xx_msix_atio_q+0x58/0x90 [qla2xxx]
>> [c00000dfff6d7dc0] [c000000000133cd0] __handle_irq_event_percpu+0xa0/0x2f0
>> [c00000dfff6d7e80] [c000000000133f5c] handle_irq_event_percpu+0x3c/0x90
>> [c00000dfff6d7ec0] [c000000000134018] handle_irq_event+0x68/0xb0
>> [c00000dfff6d7f00] [c000000000139278] handle_fasteoi_irq+0xf8/0x260
>> [c00000dfff6d7f40] [c000000000132e80] generic_handle_irq+0x50/0x80
>> [c00000dfff6d7f60] [c000000000014c44] __do_irq+0x84/0x1d0
>> [c00000dfff6d7f90] [c000000000027924] call_do_irq+0x14/0x24
>> [c000000000f13a20] [c000000000014e30] do_IRQ+0xa0/0x120
>> [c000000000f13a70] [c000000000002694] hardware_interrupt_common+0x114/0x180
>> --- interrupt: 501 at snooze_loop+0xc4/0x1a0
>>     LR = snooze_loop+0x16c/0x1a0
>> [c000000000f13d60] [c00000000063b41c] nap_loop+0x5c/0x120 (unreliable)
>> [c000000000f13da0] [c000000000637f9c] cpuidle_enter_state+0xbc/0x3d0
>> [c000000000f13e00] [c00000000011db10] call_cpuidle+0x50/0x80
>> [c000000000f13e20] [c00000000011e138] cpu_startup_entry+0x388/0x490
>> [c000000000f13ee0] [c00000000000c260] rest_init+0xb0/0xd0
>> [c000000000f13f00] [c000000000aa4070] start_kernel+0x55c/0x578
>> [c000000000f13f90] [c000000000008e6c] start_here_common+0x20/0xb4
>> nvme nvme0: I/O 782 QID 9 timeout, completion polled
>> nvme nvme0: I/O 99 QID 12 timeout, completion polled
>> nvme nvme0: I/O 925 QID 4 timeout, completio
>> 
>> 
>> Stacktrace for qlt_reset:
>> 
>> BUG: spinlock recursion on CPU#0, swapper/0/0
>> lock: 0xc00000207d5ffec0, .magic: dead4ead, .owner: swapper/0/0, .owner_cpu: 0
>> CPU: 0 PID: 0 Comm: swapper/0 Tainted: G           OE   NX 4.4.132-ttln.25-debug #1
>> Call Trace:
>> [c000003fff71b8d0] [c0000000008060c0] dump_stack+0xb0/0xf0 (unreliable)
>> [c000003fff71b910] [c0000000007ff6ec] spin_dump+0xa8/0xc4
>> [c000003fff71b980] [c0000000001283a4] do_raw_spin_lock+0x1c4/0x1d0
>> [c000003fff71b9c0] [c0000000007f7354] _raw_spin_lock_irqsave+0x34/0x50
>> [c000003fff71b9f0] [d0000000128933b4] qlt_schedule_sess_for_deletion+0x44/0x80 [qla2xxx]
>> [c000003fff71ba30] [d000000012893454] qlt_clear_tgt_db+0x64/0x90 [qla2xxx]
>> [c000003fff71ba60] [d000000012893604] qlt_reset+0x184/0x1f0 [qla2xxx]
>> [c000003fff71bb10] [d000000012897a2c] qlt_handle_imm_notify+0x74c/0x10b0 [qla2xxx]
>> [c000003fff71bc00] [d000000012898610] qlt_24xx_atio_pkt+0x280/0x400 [qla2xxx]
>> [c000003fff71bca0] [d00000001289a9f8] qlt_24xx_process_atio_queue+0x368/0x7d0 [qla2xxx]
>> [c000003fff71bd80] [d00000001289b8b8] qla83xx_msix_atio_q+0x58/0x90 [qla2xxx]
>> [c000003fff71bdc0] [c000000000133cd0] __handle_irq_event_percpu+0xa0/0x2f0
>> [c000003fff71be80] [c000000000133f5c] handle_irq_event_percpu+0x3c/0x90
>> [c000003fff71bec0] [c000000000134018] handle_irq_event+0x68/0xb0
>> [c000003fff71bf00] [c000000000139278] handle_fasteoi_irq+0xf8/0x260
>> [c000003fff71bf40] [c000000000132e80] generic_handle_irq+0x50/0x80
>> [c000003fff71bf60] [c000000000014c44] __do_irq+0x84/0x1d0
>> [c000003fff71bf90] [c000000000027924] call_do_irq+0x14/0x24
>> [c000000000f13a40] [c000000000014e30] do_IRQ+0xa0/0x120
>> [c000000000f13a90] [c000000000002694] hardware_interrupt_common+0x114/0x180
>> --- interrupt: 501 at arch_local_irq_restore+0x5c/0x90
>>     LR = arch_local_irq_restore+0x40/0x90
>> [c000000000f13d80] [c000000000f10000] init_thread_union+0x0/0x2000 (unreliable)
>> [c000000000f13da0] [c000000000637fec] cpuidle_enter_state+0x10c/0x3d0
>> [c000000000f13e00] [c00000000011db10] call_cpuidle+0x50/0x80
>> [c000000000f13e20] [c00000000011e138] cpu_startup_entry+0x388/0x490
>> [c000000000f13ee0] [c00000000000c260] rest_init+0xb0/0xd0
>> [c000000000f13f00] [c000000000aa4070] start_kernel+0x55c/0x578
>> [c000000000f13f90] [c000000000008e6c] start_here_common+0x20/0xb4
>> 
>> Steps to reproduce:
>> 1. Configure qla card as target and export lun
>> 2. Connect it to FC switch using both ports.
>> 3. Connect initiator to the switch using both ports.
>> 4a. Switch the initiator ports to reproduce the first issue
>> 4b. Switch the target ports on switch to reproduce the second one
>> 
>> Signed-off-by: Mikhail Malygin <m.malygin@xxxxxxxxx>
>> 
>> Fixed qla_reset
>> ---
>> drivers/scsi/qla2xxx/qla_target.c | 27 +++++++++++++++++----------
>> 1 file changed, 17 insertions(+), 10 deletions(-)
>> 
>> diff --git a/drivers/scsi/qla2xxx/qla_target.c b/drivers/scsi/qla2xxx/qla_target.c
>> index 025dc2d3f3de..75f9c648255a 100644
>> --- a/drivers/scsi/qla2xxx/qla_target.c
>> +++ b/drivers/scsi/qla2xxx/qla_target.c
>> @@ -1227,11 +1227,9 @@ static void qla24xx_chk_fcp_state(struct fc_port *sess)
>>         }
>> }
>> 
>> -void qlt_schedule_sess_for_deletion(struct fc_port *sess)
>> +static void qlt_schedule_sess_for_deletion_locked(struct fc_port *sess)
>> {
>>         struct qla_tgt *tgt = sess->tgt;
>> -     struct qla_hw_data *ha = sess->vha->hw;
>> -     unsigned long flags;
>> 
>>         if (sess->disc_state == DSC_DELETE_PEND)
>>                 return;
>> @@ -1247,16 +1245,13 @@ void qlt_schedule_sess_for_deletion(struct fc_port *sess)
>>                         return;
>>         }
>> 
>> -     spin_lock_irqsave(&ha->tgt.sess_lock, flags);
>>         if (sess->deleted == QLA_SESS_DELETED)
>>                 sess->logout_on_delete = 0;
>> 
>>         if (sess->deleted == QLA_SESS_DELETION_IN_PROGRESS) {
>> -             spin_unlock_irqrestore(&ha->tgt.sess_lock, flags);
>>                 return;
>>         }
>>         sess->deleted = QLA_SESS_DELETION_IN_PROGRESS;
>> -     spin_unlock_irqrestore(&ha->tgt.sess_lock, flags);
>> 
>>         sess->disc_state = DSC_DELETE_PEND;
>> 
>> @@ -1269,6 +1264,16 @@ void qlt_schedule_sess_for_deletion(struct fc_port *sess)
>>         WARN_ON(!queue_work(sess->vha->hw->wq, &sess->del_work));
>> }
>> 
>> +void qlt_schedule_sess_for_deletion(struct fc_port *sess)
>> +{
>> +     struct qla_hw_data *ha = sess->vha->hw;
>> +     unsigned long flags;
>> +
>> +     spin_lock_irqsave(&ha->tgt.sess_lock, flags);
>> +     qlt_schedule_sess_for_deletion_locked(sess);
>> +     spin_unlock_irqrestore(&ha->tgt.sess_lock, flags);
>> +}
>> +
>> static void qlt_clear_tgt_db(struct qla_tgt *tgt)
>> {
>>         struct fc_port *sess;
>> @@ -1276,7 +1281,7 @@ static void qlt_clear_tgt_db(struct qla_tgt *tgt)
>> 
>>         list_for_each_entry(sess, &vha->vp_fcports, list) {
>>                 if (sess->se_sess)
>> -                     qlt_schedule_sess_for_deletion(sess);
>> +                     qlt_schedule_sess_for_deletion_locked(sess);
>>         }
>> 
>>         /* At this point tgt could be already dead */
>> @@ -1513,7 +1518,9 @@ int qlt_stop_phase1(struct qla_tgt *tgt)
>>          */
>>         mutex_lock(&vha->vha_tgt.tgt_mutex);
>>         tgt->tgt_stop = 1;
>> +     spin_lock_irqsave(&ha->tgt.sess_lock, flags);
>>         qlt_clear_tgt_db(tgt);
>> +     spin_unlock_irqrestore(&ha->tgt.sess_lock, flags);
>>         mutex_unlock(&vha->vha_tgt.tgt_mutex);
>>         mutex_unlock(&qla_tgt_mutex);
>> 
>> @@ -4527,7 +4534,7 @@ qlt_find_sess_invalidate_other(scsi_qla_host_t *vha, uint64_t wwn,
>>                                  * might have cleared it when requested this session
>>                                  * deletion, so don't touch it
>>                                  */
>> -                             qlt_schedule_sess_for_deletion(other_sess);
>> +                             qlt_schedule_sess_for_deletion_locked(other_sess);
>>                         } else {
>>                                 /*
>>                                  * Another wwn used to have our s_id/loop_id
>> @@ -4540,7 +4547,7 @@ qlt_find_sess_invalidate_other(scsi_qla_host_t *vha, uint64_t wwn,
>>                                 other_sess->keep_nport_handle = 1;
>>                                 if (other_sess->disc_state != DSC_DELETED)
>>                                         *conflict_sess = other_sess;
>> -                             qlt_schedule_sess_for_deletion(other_sess);
>> +                             qlt_schedule_sess_for_deletion_locked(other_sess);
>>                         }
>>                         continue;
>>                 }
>> @@ -4554,7 +4561,7 @@ qlt_find_sess_invalidate_other(scsi_qla_host_t *vha, uint64_t wwn,
>> 
>>                         /* Same loop_id but different s_id
>>                          * Ok to kill and logout */
>> -                     qlt_schedule_sess_for_deletion(other_sess);
>> +                     qlt_schedule_sess_for_deletion_locked(other_sess);
>>                 }
>>         }
>> 
>> -- 
>> 2.15.1 (Apple Git-101)
>> 
> 
> Thanks,
> - Himanshu
> 
>         

Thanks,
- Himanshu





[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[Index of Archives]     [SCSI Target Devel]     [Linux SCSI Target Infrastructure]     [Kernel Newbies]     [IDE]     [Security]     [Git]     [Netfilter]     [Bugtraq]     [Yosemite News]     [MIPS Linux]     [ARM Linux]     [Linux Security]     [Linux RAID]     [Linux ATA RAID]     [Linux IIO]     [Samba]     [Device Mapper]

  Powered by Linux