I am running into deadlock during domain validation when the request queue is full. I am using the MPT Fusion spi driver and have run into this problem with 2.6.16 and the latest scsi_misc kernels. The system is running a load test on a u320 pSCSI bus with a drive that will occasionally hang the bus until a host reset clears the condition. This particular drive in known to not handle QAS very well. After the host reset, the MPT Fusion driver attempts domain validation on all drives on the bus. During DV, one or more of the queues lockup while trying to execute various SCSI commands (INQUIRY, WRITE_BUFFER, etc) using the scsi_execute() call. A stack trace shows: [ 2318.524898] events/1 D a0000001007258f0 0 16 2 (L-TLB) [ 2318.532030] [ 2318.532031] Call Trace: [ 2318.532202] [<a000000100724750>] schedule+0x1550/0x1840 [ 2318.532204] sp=e00000010a8dfc60 bsp=e00000010a8d8ff0 [ 2318.546975] [<a0000001007258f0>] io_schedule+0x50/0x80 [ 2318.546977] sp=e00000010a8dfcf0 bsp=e00000010a8d8fd0 [ 2318.554417] [<a0000001003b8820>] get_request_wait+0x200/0x2c0 [ 2318.554419] sp=e00000010a8dfcf0 bsp=e00000010a8d8f78 [ 2318.562540] [<a0000001003b8990>] blk_get_request+0xb0/0x120 [ 2318.562542] sp=e00000010a8dfd40 bsp=e00000010a8d8f40 [ 2318.579166] [<a00000010058b5e0>] scsi_execute+0x40/0x1e0 [ 2318.579168] sp=e00000010a8dfd40 bsp=e00000010a8d8ee8 [ 2318.586863] [<a0000001005980f0>] spi_execute+0x70/0x120 [ 2318.586865] sp=e00000010a8dfd40 bsp=e00000010a8d8e88 [ 2318.594204] [<a000000100599650>] spi_dv_device_echo_buffer+0x2f0/0x520 [ 2318.594206] sp=e00000010a8dfdc0 bsp=e00000010a8d8e30 [ 2318.607333] [<a000000100597a30>] spi_dv_retrain+0x70/0x520 [ 2318.607335] sp=e00000010a8dfde0 bsp=e00000010a8d8dc0 [ 2318.616119] [<a000000100599170>] spi_dv_device+0xdf0/0xf00 [ 2318.616121] sp=e00000010a8dfde0 bsp=e00000010a8d8d40 [ 2318.630538] [<a00000020db7e360>] mptspi_dv_device+0x160/0x2c0 [mptspi] [ 2318.630540] sp=e00000010a8dfdf0 bsp=e00000010a8d8ce0 [ 2318.638341] [<a00000020db7e660>] mptspi_dv_renegotiate_work+0x1a0/0x220 [mptspi] [ 2318.638343] sp=e00000010a8dfdf0 bsp=e00000010a8d8cb0 [ 2318.652773] [<a0000001000b80c0>] run_workqueue+0x1c0/0x320 [ 2318.652775] sp=e00000010a8dfe00 bsp=e00000010a8d8c80 [ 2318.660003] [<a0000001000b8460>] worker_thread+0x240/0x280 [ 2318.660005] sp=e00000010a8dfe00 bsp=e00000010a8d8c50 [ 2318.667536] [<a0000001000c24e0>] kthread+0xa0/0x120 [ 2318.667538] sp=e00000010a8dfe30 bsp=e00000010a8d8c20 [ 2318.681699] [<a0000001000129f0>] kernel_thread_helper+0xd0/0x100 [ 2318.681701] sp=e00000010a8dfe30 bsp=e00000010a8d8bf0 [ 2318.689121] [<a0000001000094c0>] start_kernel_thread+0x20/0x40 [ 2318.689124] sp=e00000010a8dfe30 bsp=e00000010a8d8bf0 Some code examination and tracing show that get_request_wait() calls get_request() to obtain a request. If get_request() returns NULL, it will wait and try again. Here is the code from get_request_wait(): rq = get_request(q, rw_flags, bio, GFP_NOIO); while (!rq) { DEFINE_WAIT(wait); struct request_list *rl = &q->rq; prepare_to_wait_exclusive(&rl->wait[rw], &wait, TASK_UNINTERRUPTIBLE); rq = get_request(q, rw_flags, bio, GFP_NOIO); if (!rq) { struct io_context *ioc; blk_add_trace_generic(q, bio, rw, BLK_TA_SLEEPRQ); __generic_unplug_device(q); spin_unlock_irq(q->queue_lock); io_schedule(); /* * After sleeping, we become a "batching" process and * will be able to allocate at least one request, and * up to a big batch of them for a small period time. * See ioc_batching, ioc_set_batching */ ioc = current_io_context(GFP_NOIO, q->node); ioc_set_batching(q, ioc); spin_lock_irq(q->queue_lock); } finish_wait(&rl->wait[rw], &wait); } Note the io_schedule() here. As far as I can tell, there is not wakeup for this wait queue. The only wakeup's occur when a request is freed. No requests can be processed because the error handling is holding off request processing until the error condition is cleared so we get a deadlock. Looking through get_request() we see: if (rl->count[rw]+1 >= queue_congestion_on_threshold(q)) { if (rl->count[rw]+1 >= q->nr_requests) { ioc = current_io_context(GFP_ATOMIC, q->node); /* * The queue will fill after this allocation, so set * it as full, and mark this process as "batching". * This process will be allowed to complete a batch of * requests, others will be blocked. */ if (!blk_queue_full(q, rw)) { ioc_set_batching(q, ioc); blk_set_queue_full(q, rw); } else { if (may_queue != ELV_MQUEUE_MUST && !ioc_batching(q, ioc)) { /* * The queue is full and the allocating * process is not a "batcher", and not * exempted by the IO scheduler goto out; } } } blk_set_queue_congested(q, rw); } In this heavily loaded system, we get into the "goto out" because count > nr_requests. The "goto out" will lead to returning NULL. This condition would not occur if ioc_batching was set, but this is not done until after the io_schedule() in get_request_wait(). -- Andrew Patterson - To unsubscribe from this list: send the line "unsubscribe linux-scsi" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html