On Wed, Aug 09, 2017 at 05:10:01PM +0000, Bart Van Assche wrote: > On Wed, 2017-08-09 at 12:43 -0400, Laurence Oberman wrote: > > Your latest patch on stock upstream without Ming's latest patches is > > behaving for me. > > > > As already mentioned, the requeue -11 and clone failure messages are > > gone and I am not actually seeing any soft lockups or hard lockups. > > > > When Ming gets back I will work with him on his patch set and the lockups. > > > > Running 10 parallel writes which easily trips into soft lockups on > > Ming's kernel (even with your patch) has been stable here on 4.13-RC3 > > with your patch. > > > > I will leave it running for a while now but the patch is good. > > > > If it survives 4 hours I will add a Tested-by to your latest patch. > > Hello Laurence, > > I'm working on an additional patch that should reduce unnecessary requeuing > even further. I will let you know when it's ready. > > Additionally, please trim e-mails when replying such that e-mails do not get > too long. soft lockup still can be observed easily with patch d4acf3650c7c( block: Make blk_mq_delay_kick_requeue_list() rerun the queue at a quiet time), but no hard lockup. With the patchset of 'blk-mq-sched: improve SCSI-MQ performance', hard lockup can be observed following some failure log: [ 269.277653] device-mapper: multipath: blk_get_request() returned -11 - requeuing [ 269.321244] device-mapper: multipath: blk_get_request() returned -11 - requeuing ... [ 273.421688] scsi host2: SRP abort called [ 273.444577] scsi host2: Sending SRP abort for tag 0x6007e [ 273.673871] scsi host2: Null scmnd for RSP w/tag 0x0000000006007e received on ch 6 / QP 0x30 ... [ 274.372110] device-mapper: multipath: blk_get_request() returned -11 - requeuing [ 278.658671] scsi host2: SRP abort called [ 278.690630] scsi host2: SRP abort called [ 278.717634] scsi host2: SRP abort called [ 278.745629] scsi host2: SRP abort called [ 279.083227] multipath_clone_and_map: 1092 callbacks suppressed .... [ 296.210503] scsi host2: SRP reset_device called .... [ 303.784287] NMI watchdog: Watchdog detected hard LOCKUP on cpu 10 The trick thing is that both hard lockup and soft lockup share one same stack trace. Another question, I don't understand why request is allocated with GFP_ATOMIC in multipath_clone_and_map(), looks it shouldn't be necessary. -- Ming
