On Tue, 2011-02-01 at 19:01 -0800, Nicholas A. Bellinger wrote:
> On Tue, 2011-02-01 at 18:55 +0100, Fubo Chen wrote:
> > On Mon, Jan 31, 2011 at 9:55 PM, Nicholas A. Bellinger
> > <nab@xxxxxxxxxxxxxxx> wrote:
> > > [ ... ]
> > >
> > > Hmmm, I don't see how this would make a difference, and FYI the above
> > > test loops for 'rmmod tcm_mvsas' where running with slub_debug=FZ w/o
> > > issue.
> > >
> > > Well, if you are certain things are working fine on .37-FINAL, you can
> > > try using 'git bisect' from a known working LIO .37 commit and build
> > > +test until you locate an offending commit.
> > >
> > > But again, this appears to be working in lio-core-2.6.git/linus-38-rc2,
> > > please verify this is what is being tested..?
> >
> > Thanks for looking at this. This is what I get with v2.6.38-rc2,
> > tcm_mvsas and slub poisoning:
> >
> > # cat /proc/cmdline
> > BOOT_IMAGE=/boot/vmlinuz-2.6.38-rc2
> > root=UUID=c2d91556-8ed3-4a2a-95d9-50d0203bcfcc ro quiet splash
> > slub_debug=FPUZ
> > # modprobe tcm_mvsas
> > # rmmod tcm_mvsas
> > # rmmod target_core_mod
> > Segmentation fault
> >
>
> Thanks for this info.. I am now able to reproduce w/ .38-rc2 using
> slub_debug=FPUZ.. (More below)
>
> > and on the console:
> >
> > <<<<<<<<<<<<<<<<<<<<<< BEGIN FABRIC API >>>>>>>>>>>>>>>>>>>>>>
> > Initialized struct target_fabric_configfs: ffff880025e09090 for mvsas
> > <<<<<<<<<<<<<<<<<<<<<< END FABRIC API >>>>>>>>>>>>>>>>>>>>>>
> > TCM_MVSAS[0] - Set fabric -> tcm_mvsas_fabric_configfs
> > <<<<<<<<<<<<<<<<<<<<<< BEGIN FABRIC API >>>>>>>>>>>>>>>>>>>>>>
> > Target_Core_ConfigFS: DEREGISTER -> Releasing tf: mvsas
> > <<<<<<<<<<<<<<<<<<<<<< END FABRIC API >>>>>>>>>>>>>>>>>>>>>>
> > TCM_MVSAS[0] - Cleared tcm_mvsas_fabric_configfs
> > general protection fault: 0000 [#1] SMP
> > last sysfs file:
> > /sys/devices/pci0000:00/0000:00:11.0/0000:02:03.0/usb1/1-0:1.0/uevent
> > CPU 0
> > Modules linked in: target_core_mod(-) configfs netconsole iscsi_tcp
> > libiscsi_tcp libiscsi scsi_transport_iscsi binfmt_misc psmouse
> > serio_raw i2c_piix4 shpchp mptspi mptscsih e1000 mptbase
> > scsi_transport_spi floppy [last unloaded: tcm_mvsas]
> >
> > Pid: 1432, comm: rmmod Not tainted 2.6.38-rc2 #4 440BX Desktop
> > Reference Platform/VMware Virtual Platform
> > RIP: 0010:[<ffffffff81094684>] [<ffffffff81094684>] __lock_acquire+0x64/0x1510
> > RSP: 0018:ffff880022697b18 EFLAGS: 00010046
> > RAX: 0000000000000046 RBX: 6b6b6b6b6b6b6be3 RCX: 0000000000000000
> > RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
> > RBP: ffff880022697be8 R08: 0000000000000001 R09: 0000000000000000
> > R10: 0000000000000000 R11: 0000000000000001 R12: 0000000000000002
> > R13: 0000000000000000 R14: 0000000000000000 R15: ffff88002a1a2350
> > FS: 00007f844069c700(0000) GS:ffff88003d600000(0000) knlGS:0000000000000000
> > CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> > CR2: 00007f8440189fc0 CR3: 0000000025d67000 CR4: 00000000000006f0
> > DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> > DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> > Process rmmod (pid: 1432, threadinfo ffff880022696000, task ffff88002a1a2350)
> > Stack:
> > 0000000000000004 ffff88002a1a2350 ffffffff82030820 ffffffff81010dfd
> > ffff880022697b68 ffffffff81ed0590 ffff880022697b68 0000000000000000
> > 3161938ca065261c ffff88002a1a2b08 ffff880022697c48 0000000000000002
> > Call Trace:
> > [<ffffffff81010dfd>] ? save_stack_trace+0x2d/0x50
> > [<ffffffff81095bd0>] lock_acquire+0xa0/0x150
> > [<ffffffffa00e5f7f>] ? detach_groups+0x2f/0x120 [configfs]
> > [<ffffffff81540f44>] ? __mutex_lock_common+0x2a4/0x3e0
> > [<ffffffffa00e5ff4>] ? detach_groups+0xa4/0x120 [configfs]
> > [<ffffffff815427f6>] _raw_spin_lock+0x36/0x70
> > [<ffffffffa00e5f7f>] ? detach_groups+0x2f/0x120 [configfs]
> > [<ffffffffa00e5f7f>] detach_groups+0x2f/0x120 [configfs]
> > [<ffffffffa00e5f36>] configfs_detach_group+0x16/0x30 [configfs]
> > [<ffffffffa00e6002>] detach_groups+0xb2/0x120 [configfs]
> > [<ffffffffa00e5f36>] configfs_detach_group+0x16/0x30 [configfs]
> > [<ffffffffa00e6002>] detach_groups+0xb2/0x120 [configfs]
> > [<ffffffffa00e5f36>] configfs_detach_group+0x16/0x30 [configfs]
> > [<ffffffffa00e6002>] detach_groups+0xb2/0x120 [configfs]
> > [<ffffffffa00e5f36>] configfs_detach_group+0x16/0x30 [configfs]
> > [<ffffffffa00e6002>] detach_groups+0xb2/0x120 [configfs]
> > [<ffffffffa00e5f36>] configfs_detach_group+0x16/0x30 [configfs]
> > [<ffffffffa00e6112>] configfs_unregister_subsystem+0xa2/0x130 [configfs]
> > [<ffffffffa00efc84>] target_core_exit_configfs+0x184/0x1c0 [target_core_mod]
> > [<ffffffff810a0a12>] sys_delete_module+0x1a2/0x280
> > [<ffffffff81542559>] ? trace_hardirqs_on_thunk+0x3a/0x3f
> > [<ffffffff81002f82>] system_call_fastpath+0x16/0x1b
> > Code: 8b 05 c1 64 9a 00 4c 89 75 f0 48 89 fb 41 89 d5 4c 8b 55 10 45
> > 85 c0 0f 84 4a 04 00 00 8b 3d 28 86 cd 00 85 ff 0f 84 5c 04 00 00 <48>
> > 81 3b 20 05 dd 81 b8 01 00 00 00 44 0f 44 e0 83 fe 01 0f 86
> > RIP [<ffffffff81094684>] __lock_acquire+0x64/0x1510
> > RSP <ffff880022697b18>
> > ---[ end trace 4abcf014267c1c85 ]---
> > --
>
> So this is coming from target_core_exit_configfs() ->
> configfs_unregister_system() from a simple 'modprobe target_core_mo ;
> rmmod target_core_mod' with slub_debug=FPUZ..
>
> It appears to be related to the TCM top level struct
> configfs_subsystem->su_group->default_groups[], which we setup in
> target_core_init_configfs() and from which are released individually in
> target_core_exit_configfs() before calling configfs_unregister_system().
>
> Note that target_core_exit_configfs() is following the same logic as
> default_groups for non struct configfs_subsystem backed groups, so I am
> thinking this is going to be the root culprit.
>
> After a quick test w/o the above subsys->su_group.default_groups
> allocation/release (and the rest of the top level cg->default_groups[]
> disabled), the GFP no longer appears. They appear to be coming more
> than a single stale struct configfs_dirent->s_children from the top
> level TCM default groups attached fs/configfs/dir.c:detach_groups().
> (jlbec CC'ed)
>
> I am still looking at what is the expected way to handle multiple
> default_groups (including a default_group with children) with struct
> configfs_subsystem deregister() in fs/configfs/dir.c code, and will send
> a followup later this evening.
>
> Thanks again for your report,
>
Ok, after some more research and testing there appears to be two issues
in target_core_exit_configfs() wrt to default groups. First, the call
to configfs_unregister_subsystem() is expected to drain top level struct
configfs_subsystem->su_group.default_groups[] in fs/configfs/dir.c:
configfs_unregister_subsystem() -> unlink_group(), and not directly by
the configfs consumer.
These second issue is core_alua_free_lu_gp(se_global->default_lu_gp)
releasing default_lu_gp->lun_group before lu_gp_cg->default_groups is
drained.
Here the change that is now resolving the issue on my end with .38-rc2
using slub_debug=FPUZ, and I will send out a proper patch for
lio-core-2.6.git/linus-38-rc2 shortly.. Please verify this works for
you.
Thanks!
--nab
diff --git a/drivers/target/target_core_configfs.c b/drivers/target/target_core_configfs.c
index 9ff1942..7d7dfbc 100644
--- a/drivers/target/target_core_configfs.c
+++ b/drivers/target/target_core_configfs.c
@@ -3262,8 +3262,7 @@ static void target_core_exit_configfs(void)
config_item_put(item);
}
kfree(lu_gp_cg->default_groups);
- core_alua_free_lu_gp(se_global->default_lu_gp);
- se_global->default_lu_gp = NULL;
+ lu_gp_cg->default_groups = NULL;
alua_cg = &se_global->alua_group;
for (i = 0; alua_cg->default_groups[i]; i++) {
@@ -3272,6 +3271,7 @@ static void target_core_exit_configfs(void)
config_item_put(item);
}
kfree(alua_cg->default_groups);
+ alua_cg->default_groups = NULL;
hba_cg = &se_global->target_core_hbagroup;
for (i = 0; hba_cg->default_groups[i]; i++) {
@@ -3280,15 +3280,17 @@ static void target_core_exit_configfs(void)
config_item_put(item);
}
kfree(hba_cg->default_groups);
-
- for (i = 0; subsys->su_group.default_groups[i]; i++) {
- item = &subsys->su_group.default_groups[i]->cg_item;
- subsys->su_group.default_groups[i] = NULL;
- config_item_put(item);
- }
+ hba_cg->default_groups = NULL;
+ /*
+ * We expect subsys->su_group.default_groups to be released
+ * by configfs subsystem provider logic..
+ */
+ configfs_unregister_subsystem(subsys);
kfree(subsys->su_group.default_groups);
- configfs_unregister_subsystem(subsys);
+ core_alua_free_lu_gp(se_global->default_lu_gp);
+ se_global->default_lu_gp = NULL;
+
printk(KERN_INFO "TARGET_CORE[0]: Released ConfigFS Fabric"
" Infrastructure\n");
--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
