On Fri, Jan 27 2006, Mike Christie wrote: > Jens Axboe wrote: > >On Fri, Jan 27 2006, Jens Axboe wrote: > > > >>On Fri, Jan 27 2006, Neil Brown wrote: > >> > >>>On Friday January 27, chase.venters@xxxxxxxxxxxx wrote: > >>> > >>>>Greetings, > >>>> Just a quick recap - there are at least 4 reports of 2.6.15 users > >>>>experiencing severe slab leaks with scsi_cmd_cache. It seems that a few > >>>>of us have a board (Asus P5GDC-V Deluxe) in common. We seem to have > >>>>raid in common. After dealing with this leak for a while, I decided > >>>> to do some dancing around with git bisect. I've landed on a possible > >>>>point of regression: > >>>> > >>>>commit: a9701a30470856408d08657eb1bd7ae29a146190 > >>>>[PATCH] md: support BIO_RW_BARRIER for md/raid1 > >>>> > >>>> I spent about an hour and a half reading through the patch, trying > >>>> to see if I could make sense of what might be wrong. The result (after > >>>>I dug into the code to make a change I foolishly thought made sense) > >>>>was a hung kernel. > >>>> This is important because when I rebooted into the kernel that had > >>>> been giving me trouble, it started an md resync and I'm now watching > >>>>(at least during this resync) the slab usage for scsi_cmd_cache stay > >>>>sane: > >>>> > >>>>turbotaz ~ # cat /proc/slabinfo | grep scsi_cmd_cache > >>>>scsi_cmd_cache 30 30 384 10 1 : tunables 54 27 > >>>>8 : slabdata 3 3 0 > >>>> > >>> > >>>This suggests that the problem happens when a BIO_RW_BARRIER write is > >>>sent to the device. With this patch, md flags all superblock writes > >>>as BIO_RW_BARRIER However md is not so likely to update the superblock > >>>often > >>>during a resync. > >>> > >>>There is a (rough) count of the number of superblock writes in the > >>>"Events" counter which "mdadm -D" will display. > >>>You could try collecting 'Events' counter together with the > >>>'active_objs' count from /proc/slabinfo and graph the pairs - see if > >>>they are linear. > >>> > >>>I believe a BIO_RW_BARRIER is likely to send some sort of 'flush' > >>>command to the device, and the driver for your particular device may > >>>well be losing scsi_cmd_cache allocation when doing that, but I leave > >>>that to someone how knows more about that code. > >> > >>I already checked up on that since I suspected barriers initially. The > >>path there for scsi is sd.c:sd_issue_flush() which looks pretty straight > >>forward. In the end it goes through the block layer and gets back to the > >>SCSI layer as a regular REQ_BLOCK_PC request. > > > > > >Sorry, that was for the ->issue_flush() that md also does but did before > >the barrier addition as well. Most of the barrier handling is done in > >the block layer, but it could show leaks in SCSI of course. FWIW, I > >tested barriers with and without md on SCSI here a few days ago and > >didn't see any leaks at all. > > > > It does not have anything to do with this in scsi_io_completion does it? > > if (blk_complete_barrier_rq(q, req, good_bytes >> 9)) > return; > > For that case the scsi_cmnd does not get freed. Does it come back around > again and get released from a different path? Certainly smells fishy. Unfortunately I cannot take a look at this until monday :/ But adding some tracing there might be really interesting. Since we are not seeing bio and/or req leaks, this does look very promising. -- Jens Axboe - : send the line "unsubscribe linux-scsi" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
