On Tue, 2008-12-02 at 15:18 -0800, Nicholas A. Bellinger wrote: > On Tue, 2008-12-02 at 01:13 -0800, Mike Anderson wrote: > > Nicholas A. Bellinger <nab@xxxxxxxxxxxxxxx> wrote: > > > On Tue, 2008-12-02 at 00:30 -0800, Nicholas A. Bellinger wrote: > > > > On Mon, 2008-12-01 at 22:40 -0800, Mike Anderson wrote: > > > > > Nicholas A. Bellinger <nab@xxxxxxxxxxxxxxx> wrote: > > > > > > On Tue, 2008-12-02 at 13:18 +0900, Tejun Heo wrote: > > > > > > > > > > > > > > >>> The other one is a BUG_ON in blk/blk-timeout.c:177 in blk_add_timeout() > > > > > > > >>> that happens after a few hundred MB of READ_10 traffic, which also > > > > > > > >>> appears to pass through elv_dequeue_request() at some point: > > > > > > > >>> > > > > > > > >>> http://linux-iscsi.org/builds/user/nab/2.6.28-rc6-oops-2.png > > > > > > > >>> http://linux-iscsi.org/builds/user/nab/2.6.28-rc6-oops-4.png > > > > > > > > > > > > > > Hmmm... this means blk_add_timer() is being called after the request > > > > > > > is already completed. > > > > > > > > > > or is it possible since elv_dequeue_request BUG_ON check of queuelist did > > > > > not trigger a request is on the queuelist with a timeout_list not empty. > > > > > > > > > > It would be interesting for a debug run to change the > > > > > "BUG_ON(!list_empty(&req->timeout_list))" in blk_add_timer to print out > > > > > the cmd_flags plus req->atomic_flags and also add a > > > > > "BUG_ON(!list_empty(&rq->timeout_list))" to elv_insert to ensure a request > > > > > is never added to the queuelist with a timeout_list not empty. > > > > > > > > > > > > > Ok, so blk_dump_rq_flags() is now being called in > > > > block/blk-timeout.c:blk_add_timer() for the case > > > > BUG_ON(list_empty(&req->timeout_list)) case: > > > > > > > > http://linux-iscsi.org/builds/user/nab/2.6.28-rc6-oops-6.png > > > > > > > > Hmm, the outputted "sector " range is definately is bogus, as the only > > > > READ_10 that have been sent are at LBA offset 0 for 8 * 512 byte sectors > > > > for the partition table during Open/iSCSI LUN scanning. > > > > > > > > > > Also, the following code from block/blk-core.c:blk_dump_rq_flags(): > > > > > > if (blk_pc_request(rq)) { > > > printk(KERN_INFO " cdb: "); > > > for (bit = 0; bit < BLK_MAX_CDB; bit++) > > > printk("%02x ", rq->cmd[bit]); > > > printk("\n"); > > > } > > > > > > is not printing out the copied CDB in struct request->cmd[], which makes > > > me think the struct request->cmd_flags (that blk_pc_request() is > > > checking) are also bogus when blk_add_timer() is being called.. > > > > > Based on the output from 2.6.28-rc6-oops-6.png the flags value of 82c21 > > looks like > > (__REQ_ALLOCED,__REQ_ELVPRIV,__REQ_DONTPREP,__REQ_STARTED,__REQ_SORTED,__REQ_RW), > > but it is late so someone maybe should check my shift counts. > > > > The type is also REQ_TYPE_FS so the blk_pc cdb: info will not be printed. > > > > I will talk to you more in the morning. > > > > Ok, after double checking the original patch again, I noticed that > struct request was getting released with __blk_put_request() from the > struct request->end_io() caller context in pscsi_req_done(), and again > after the original target mode CDB was acknowledged from the fabric in > pscsi_free_task() with blk_put_request().. Not good. > > Anyways, I removed the extra blk_put_request() in pscsi_free_task() and > am running some badblocks tests now. Things are obviously looking much > better. > Ok, passing badblocks tests now and looking stable on v2.6.28-rc7 using Tejun's patch, here the commit diff for the LIO-Core v3.0 PSCSI changes: http://git.kernel.org/?p=linux/kernel/git/nab/lio-core-2.6.git;a=commitdiff;h=1b14b5e1fc9a7074322b1015bb86eca2a8ef4560 Now the question becomes, how can we get rid of scsi_req_map_sg() usage (and struct bio in between) all together..? --nab -- To unsubscribe from this list: send the line "unsubscribe linux-scsi" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
