On Wednesday 30 June 2004 12:43 pm, Dave Olien wrote: > The interesting stack trace is, I think this one. My analysis follows. > > kcopyd D C46C3800 0 1688 7 9 (L-TLB) > f6479bb4 00000046 00010000 c46c3800 00000000 00000000 f65a891c c8062ca8 > c8062ce0 c3fdb084 00000010 00000200 740fb02c 00000072 c3caf2d0 > c1000be0 0000054b 740fb02c 00000072 f633b608 f7ffc8d4 00000246 c1000be0 > f7ffc8d4 Call Trace: > [<c04203a8>] io_schedule+0x28/0x40 > [<c0143930>] mempool_alloc+0x1a0/0x200 > [<c016c82c>] bio_alloc+0x1c/0x1a0 > [<c016ca00>] bio_clone+0x10/0x90 > [<c0340f34>] clone_bio+0x24/0x60 > [<c034102c>] __clone_and_map+0xbc/0x300 > [<c0341307>] __split_bio+0x97/0x110 > [<c0341407>] dm_request+0x87/0xb0 > [<c028664b>] generic_make_request+0x14b/0x1c0 > [<c028672b>] submit_bio+0x6b/0x110 > [<c03470e3>] do_region+0x173/0x190 > [<c03471b9>] dispatch_io+0xb9/0xc0 > [<c034732a>] async_io+0x5a/0x70 > [<c03474b3>] dm_io_async+0x53/0x60 > [<c0347b83>] run_io_job+0x43/0x80 > [<c0347d94>] process_jobs+0xc4/0x2a0 > [<c01338a5>] worker_thread+0x1f5/0x370 > [<c0137fc5>] kthread+0xa5/0xb0 > [<c01032e5>] kernel_thread_helper+0x5/0x10 > > There are two problems that need addressing. This first one is that > it looks to me like we're deadlocked allocating bio structures. > > All of the bios from the global pool are tied up with those pending_commit > operations. I'm now aware why device mapper has it's own local > pool of bio's, and hence it's own local copy of fs/bio.c. But notice > the call to bio_clone() on the stack trace is allocating from the global > bio pool. > > SO, patch number 1. is to give dm-io.c it's own copy of the bio_clone() > that allocates from dm-io.c's local pool of bio's and bvec's. The call to bio_clone() in the above trace is not in dm-io.c - it's in dm.c and part of the normal I/O path. I've always been a bit wary of calling the generic bio_clone() at that point, but I've never been able to generate a case that causes memory starvation like you're seeing. In my opinion, what we *really* need to do is move the local bio pool stuff in dm-io.c into fs/bio.c, and provide a kernel-wide mechanism for creating pools of bios. Then we could use that mechanism from both dm.c and dm-io.c. Of course, we'll need to convince a few more people than just the three of us before we can make such a change. :) -- Kevin Corry kevcorry@xxxxxxxxxx http://evms.sourceforge.net/