Re: 3.9.4 Oops running xfstests (WAS Re: 3.9.3: Oops running xfstests)

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 




----- Original Message -----
> From: "Dave Chinner" <david@xxxxxxxxxxxxx>
> To: "CAI Qian" <caiqian@xxxxxxxxxx>
> Cc: xfs@xxxxxxxxxxx, stable@xxxxxxxxxxxxxxx, "LKML" <linux-kernel@xxxxxxxxxxxxxxx>, "linux-mm" <linux-mm@xxxxxxxxx>
> Sent: Monday, June 3, 2013 12:00:38 PM
> Subject: Re: 3.9.4 Oops running xfstests (WAS Re: 3.9.3: Oops running xfstests)
> 
> On Sun, Jun 02, 2013 at 11:04:11PM -0400, CAI Qian wrote:
> > 
> > > There's memory corruption all over the place.  It is most likely
> > > that trinity is causing this - it's purpose is to trigger corruption
> > > issues, but they aren't always immediately seen.  If you can trigger
> > > this xfs trace without trinity having been run and without all the
> > > RCU/idle/scheduler/cgroup issues occuring at the same time, then
> > > it's likely to be caused by XFS. But right now, I'd say XFS is just
> > > an innocent bystander caught in the crossfire. There's nothing I can
> > > do from an XFS persepctive to track this down...
> > OK, this can be reproduced by just running LTP and then xfstests without
> > trinity at all...
> 
> Cai, can you be more precise about what is triggering it?  LTP and
> xfstests do a large amount of stuff, and stack traces do not do not
> help narrow down the cause at all.  Can you provide the follwoing
> information and perform the follwoing steps:
> 
> 	1. What xfstest is tripping over it?
Test #20.
> 	2. Can you reproduce it just by running that one specific test
> 	  on a pristine system (i.e. freshly mkfs'd filesystems,
> 	  immediately after boot)
Yes, it was reproduced without LTP at all.
[   98.534402] XFS (dm-0): Mounting Filesystem
[   98.586673] XFS (dm-0): Ending clean mount
[   99.741704] XFS (dm-2): Mounting Filesystem
[  100.117248] XFS (dm-2): Ending clean mount
[  100.723228] XFS (dm-0): Mounting Filesystem
[  100.775965] XFS (dm-0): Ending clean mount
[  101.980250] BUG: unable to handle kernel NULL pointer dereference at 0000000000000098
[  101.988136] IP: [<ffffffff81098cac>] tg_load_down+0x4c/0x80
[  101.993737] PGD 0 
[  101.995769] Oops: 0002 [#1] SMP 
[  101.999038] Modules linked in: lockd sunrpc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_MASQUERADE ip6table_nat nf_nat_ipv6 ip6table_mangle ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 iptable_nat nf_nat_ipv4 nf_nat iptable_mangle ipt_REJECT nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ip_tables sg snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_intel snd_hda_codec snd_hwdep snd_seq snd_seq_device snd_pcm hp_wmi sparse_keymap rfkill iTCO_wdt e1000e pcspkr snd_page_alloc iTCO_vendor_support mei ptp pps_core lpc_ich i2c_i801 snd_timer mfd_core microcode(+) snd soundcore xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif nouveau video mxm_wmi i2c_algo_bit drm_kms_helper ahci ata_generic ttm libahci pata_acpi drm i2c_core libata wmi dm_mirror dm_region_hash dm_log dm_mod
[  102.075355] CPU 2 
[  102.077197] Pid: 356, comm: kworker/2:2 Not tainted 3.9.4 #1 Hewlett-Packard HP Z210 Workstation/1587h
[  102.086691] RIP: 0010:[<ffffffff81098cac>]  [<ffffffff81098cac>] tg_load_down+0x4c/0x80
[  102.094705] RSP: 0018:ffff880078307c78  EFLAGS: 00010002
[  102.100020] RAX: 0001f2b5a618ed0f RBX: 0000000000000001 RCX: 000000000000068a
[  102.107157] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8800772ceee8
[  102.114293] RBP: ffff880078307c78 R08: 0000000000000008 R09: ffff88007d094400
[  102.121422] R10: 0000000000000344 R11: 0000000000000001 R12: ffffffff81c78560
[  102.128552] R13: ffffffff8108c460 R14: 0000000000000000 R15: ffff8800772ceee8
[  102.135682] FS:  0000000000000000(0000) GS:ffff88007d100000(0000) knlGS:0000000000000000
[  102.143776] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  102.149524] CR2: 0000000000000098 CR3: 00000000018fa000 CR4: 00000000000407e0
[  102.156654] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[  102.163785] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[  102.170915] Process kworker/2:2 (pid: 356, threadinfo ffff880078306000, task ffff88007736b580)
[  102.179527] Stack:
[  102.181545]  ffff880078307cc0 ffffffff810926b2 ffffffff81098c60 ffff8800772cf008
[  102.189005]  ffff880079b96f00 000000000000069c ffff880079b96ee8 0000000000014400
[  102.196464]  ffff88007d094400 ffff880078307db0 ffffffff8109f773 ffff88007cc10480
[  102.203920] Call Trace:
[  102.206372]  [<ffffffff810926b2>] walk_tg_tree_from+0x32/0xe0
[  102.212118]  [<ffffffff81098c60>] ? task_waking_fair+0x20/0x20
[  102.217955]  [<ffffffff8109f773>] load_balance+0x2a3/0x7d0
[  102.223444]  [<ffffffff8108fa7c>] ? update_rq_clock.part.67+0x1c/0x170
[  102.229977]  [<ffffffff810a0142>] idle_balance+0x182/0x2f0
[  102.235468]  [<ffffffff8160f1ac>] __schedule+0x7bc/0x7d0
[  102.240786]  [<ffffffff8160f1e9>] schedule+0x29/0x70
[  102.245756]  [<ffffffff8107f404>] worker_thread+0x1b4/0x3d0
[  102.251332]  [<ffffffff8107f250>] ? __alloc_workqueue_key+0x500/0x500
[  102.257777]  [<ffffffff81084240>] kthread+0xc0/0xd0
[  102.262662]  [<ffffffff81084180>] ? insert_kthread_work+0x40/0x40
[  102.268761]  [<ffffffff816192ac>] ret_from_fork+0x7c/0xb0
[  102.274164]  [<ffffffff81084180>] ? insert_kthread_work+0x40/0x40
[  102.280262] Code: 00 00 00 00 48 8b 14 f0 48 8b 0c f1 48 8b 82 98 00 00 00 48 0f af 01 48 8b 0a 31 d2 48 83 c1 01 48 f7 f1 48 8b 57 48 4a 8b 14 02 <48> 89 82 98 00 00 00 31 c0 5d c3 66 0f 1f 84 00 00 00 00 00 48 
[  102.300226] RIP  [<ffffffff81098cac>] tg_load_down+0x4c/0x80
[  102.305906]  RSP <ffff880078307c78>
[  102.309394] CR2: 0000000000000098
[  102.312710] ---[ end trace ba964230a74993fe ]---
[  102.312714] BUG: unable to handle kernel NULL pointer dereference at 0000000000000098
[  102.312719] IP: [<ffffffff81098cac>] tg_load_down+0x4c/0x80
[  102.312720] PGD 0 
[  102.312723] Oops: 0002 [#2] SMP 
[  102.312752] Modules linked in: lockd sunrpc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_MASQUERADE ip6table_nat nf_nat_ipv6 ip6table_mangle ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 iptable_nat nf_nat_ipv4 nf_nat iptable_mangle ipt_REJECT nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ip_tables sg snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_intel snd_hda_codec snd_hwdep snd_seq snd_seq_device snd_pcm hp_wmi sparse_keymap rfkill iTCO_wdt e1000e pcspkr snd_page_alloc iTCO_vendor_support mei ptp pps_core lpc_ich i2c_i801 snd_timer mfd_core microcode(+) snd soundcore xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif nouveau video mxm_wmi i2c_algo_bit drm_kms_helper ahci ata_generic ttm libahci pata_acpi drm i2c_core libata wmi dm_mirror dm_region_hash dm_log dm_mod
[  102.312758] CPU 0 
[  102.312758] Pid: 78, comm: kworker/0:2 Tainted: G      D      3.9.4 #1 Hewlett-Packard HP Z210 Workstation/1587h
[  102.312762] RIP: 0010:[<ffffffff81098cac>]  [<ffffffff81098cac>] tg_load_down+0x4c/0x80
[  102.312763] RSP: 0018:ffff880036c25c78  EFLAGS: 00010002
[  102.312764] RAX: 0001f2b5a618ed0f RBX: 0000000000000001 RCX: 000000000000068a
[  102.312765] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8800772ceee8
[  102.312766] RBP: ffff880036c25c78 R08: 0000000000000008 R09: ffff88007d094400
[  102.312767] R10: 0000000000000344 R11: 0000000000000001 R12: ffffffff81c78560
[  102.312768] R13: ffffffff8108c460 R14: 0000000000000000 R15: ffff8800772ceee8
[  102.312770] FS:  0000000000000000(0000) GS:ffff88007d000000(0000) knlGS:0000000000000000
[  102.312771] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  102.312772] CR2: 0000000000000098 CR3: 00000000018fa000 CR4: 00000000000407f0
[  102.312773] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[  102.312774] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[  102.312776] Process kworker/0:2 (pid: 78, threadinfo ffff880036c24000, task ffff8800773c9ac0)
[  102.312776] Stack:
[  102.312779]  ffff880036c25cc0 ffffffff810926b2 ffffffff81098c60 ffff8800772cf008
[  102.312781]  ffff880079b96f00 000000000000069c ffff880079b96ee8 0000000000014400
[  102.312783]  ffff88007d094400 ffff880036c25db0 ffffffff8109f773 ffff88007cc10080
[  102.312783] Call Trace:
[  102.312787]  [<ffffffff810926b2>] walk_tg_tree_from+0x32/0xe0
[  102.312789]  [<ffffffff81098c60>] ? task_waking_fair+0x20/0x20
[  102.312792]  [<ffffffff8109f773>] load_balance+0x2a3/0x7d0
[  102.312795]  [<ffffffff8108fa7c>] ? update_rq_clock.part.67+0x1c/0x170
[  102.312798]  [<ffffffff810a0142>] idle_balance+0x182/0x2f0
[  102.312801]  [<ffffffff8160f1ac>] __schedule+0x7bc/0x7d0
[  102.312803]  [<ffffffff8160f1e9>] schedule+0x29/0x70
[  102.312806]  [<ffffffff8107f404>] worker_thread+0x1b4/0x3d0
[  102.312809]  [<ffffffff8107f250>] ? __alloc_workqueue_key+0x500/0x500
[  102.312811]  [<ffffffff81084240>] kthread+0xc0/0xd0
[  102.312813]  [<ffffffff81084180>] ? insert_kthread_work+0x40/0x40
[  102.312816]  [<ffffffff816192ac>] ret_from_fork+0x7c/0xb0
[  102.312818]  [<ffffffff81084180>] ? insert_kthread_work+0x40/0x40
[  102.312837] Code: 00 00 00 00 48 8b 14 f0 48 8b 0c f1 48 8b 82 98 00 00 00 48 0f af 01 48 8b 0a 31 d2 48 83 c1 01 48 f7 f1 48 8b 57 48 4a 8b 14 02 <48> 89 82 98 00 00 00 31 c0 5d c3 66 0f 1f 84 00 00 00 00 00 48 
[  102.312839] RIP  [<ffffffff81098cac>] tg_load_down+0x4c/0x80
[  102.312840]  RSP <ffff880036c25c78>
[  102.312840] CR2: 0000000000000098
[  102.312842] ---[ end trace ba964230a74993ff ]---
[  102.312866] general protection fault: 0000 [#3] SMP 
[  102.312896] Modules linked in: lockd sunrpc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_MASQUERADE ip6table_nat nf_nat_ipv6 ip6table_mangle ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 iptable_nat nf_nat_ipv4 nf_nat iptable_mangle ipt_REJECT nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ip_tables sg snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_intel snd_hda_codec snd_hwdep snd_seq snd_seq_device snd_pcm hp_wmi sparse_keymap rfkill iTCO_wdt e1000e pcspkr snd_page_alloc iTCO_vendor_support mei ptp pps_core lpc_ich i2c_i801 snd_timer mfd_core microcode(+) snd soundcore xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif nouveau video mxm_wmi i2c_algo_bit drm_kms_helper ahci ata_generic ttm libahci pata_acpi drm i2c_core libata wmi dm_mirror dm_region_hash dm_log dm_mod
[  102.312903] CPU 1 
[  102.312903] Pid: 1999, comm: attr Tainted: G      D      3.9.4 #1 Hewlett-Packard HP Z210 Workstation/1587h
[  102.312908] RIP: 0010:[<ffffffff810983a9>]  [<ffffffff810983a9>] irqtime_account_process_tick.isra.2+0x239/0x3c0
[  102.312909] =============================================================================
[  102.312910] RSP: 0018:ffff88007d083e08  EFLAGS: 00010003
[  102.312912] BUG kmalloc-1024 (Tainted: G      D     ): Padding overwritten. 0xffff88005b4e7ec0-0xffff88005b4e7fff
[  102.312913] RAX: ffff88005b656288 RBX: ffff880079b43c80 RCX: 00000000000000a7
[  102.312914] -----------------------------------------------------------------------------
[  102.312914] 
[  102.312915] RDX: 6b6b6b6b6b6b6b6b RSI: 0000000000000000 RDI: 0000000000000086
[  102.312917] INFO: Slab 0xffffea00016d3800 objects=24 used=24 fp=0x          (null) flags=0x10000000004080
[  102.312918] RBP: ffff88007d083e40 R08: 0000000000000000 R09: 0000000225c17d03
[  102.312921] Pid: 518, comm: in:imklog Tainted: G    B D      3.9.4 #1
[  102.312922] R10: 0000000000000000 R11: 0000000000000001 R12: ffff88007d08e800
[  102.312922] Call Trace:
[  102.312923] R13: ffff880058063580 R14: 0000000000000000 R15: ffff88007d094c88
[  102.312926] FS:  00007f237404a740(0000) GS:ffff88007d080000(0000) knlGS:0000000000000000
[  102.312931]  [<ffffffff81181ed2>] slab_err+0xc2/0xf0
[  102.312932] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  102.312934] CR2: 00007f2373c352a0 CR3: 0000000063ea9000 CR4: 00000000000407e0
[  102.312937]  [<ffffffff81018018>] ? write_ok_or_segv+0x88/0x90
[  102.312938] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[  102.312939] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[  102.312943]  [<ffffffff81020c2b>] ? save_stack_trace+0x2b/0x50
[  102.312944] Process attr (pid: 1999, threadinfo ffff88005753e000, task ffff880058063580)
[  102.312945] Stack:
[  102.312947]  [<ffffffff81180911>] ? set_track+0x61/0x1b0
[  102.312949]  ffff88007d08e800
[  102.312952]  [<ffffffff8118176d>] ? init_object+0x3d/0x70
[  102.312953]  0000000000000086 ffff88007d094400
[  102.312956]  [<ffffffff81181ff5>] slab_pad_check.part.41+0xf5/0x170
[  102.312957]  0000000000014400
[  102.312967]  [<ffffffff81063bdc>] ? do_syslog+0x23c/0x5c0
[  102.312968]  ffff880058063580 0000000000000000
[  102.312969]  [<ffffffff811820e3>] check_slab+0x73/0x100
[  102.312970]  00000017979e99b3 ffff88007d083e78
[  102.312970] 
[  102.312972]  [<ffffffff81606b50>] alloc_debug_processing+0x21/0x118
[  102.312973]  ffffffff8109874c
[  102.312974]  [<ffffffff8160772f>] __slab_alloc+0x3b8/0x4a2
[  102.312974]  ffff880058063580 0000000000000000
[  102.312976]  [<ffffffff81063bdc>] ? do_syslog+0x23c/0x5c0
[  102.312976]  0000000000000001
[  102.312977] Call Trace:
[  102.312978]  [<ffffffff810915aa>] ? finish_task_switch+0xba/0xe0
[  102.312979]  <IRQ> 
[  102.312980]  [<ffffffff81184dd1>] kmem_cache_alloc_trace+0x1b1/0x200
[  102.312981]  [<ffffffff8109874c>] account_process_tick+0x11c/0x1d0
[  102.312983]  [<ffffffff81063bdc>] do_syslog+0x23c/0x5c0
[  102.312985]  [<ffffffff81071f0d>] update_process_times+0x2d/0x80
[  102.312986]  [<ffffffff810850e0>] ? wake_up_bit+0x30/0x30
[  102.312988]  [<ffffffff810ba1c5>] tick_sched_handle.isra.13+0x25/0x60
[  102.312991]  [<ffffffff8120ca94>] kmsg_read+0x44/0x60
[  102.312992]  [<ffffffff810ba241>] tick_sched_timer+0x41/0x60
[  102.312994]  [<ffffffff811ffa9a>] proc_reg_read+0x6a/0xa0
[  102.312996]  [<ffffffff81087af4>] __run_hrtimer+0x74/0x1d0
[  102.312997]  [<ffffffff8119c56c>] vfs_read+0x9c/0x170
[  102.312998]  [<ffffffff810ba200>] ? tick_sched_handle.isra.13+0x60/0x60
[  102.313000]  [<ffffffff8119c939>] sys_read+0x49/0xa0
[  102.313001]  [<ffffffff810882d7>] hrtimer_interrupt+0xe7/0x220
[  102.313004]  [<ffffffff810e0ef6>] ? __audit_syscall_exit+0x1f6/0x2a0
[  102.313005]  [<ffffffff8161b0e9>] smp_apic_timer_interrupt+0x69/0x9c
[  102.313007]  [<ffffffff81619359>] system_call_fastpath+0x16/0x1b
[  102.313008]  [<ffffffff81619fdd>] apic_timer_interrupt+0x6d/0x80
[  102.313009] Padding ffff88005b4e7ec0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  102.313010] Padding ffff88005b4e7ed0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  102.313010]  <EOI> 
[  102.313011] Padding ffff88005b4e7ee0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  102.313013]  [<ffffffff813022f6>] ? memmove+0x46/0x1a0
[  102.313013] Padding ffff88005b4e7ef0: 00 00 00 00 00 00 00 00 00 00 00 00 00 29 01 00  .............)..
[  102.313014] Padding ffff88005b4e7f00: 07 1b 04 73 65 6c 69 6e 75 78 73 79 73 74 65 6d  ...selinuxsystem
[  102.313015] Padding ffff88005b4e7f10: 5f 75 3a 6f 62 6a 65 63 74 5f 72 3a 75 73 72 5f  _u:object_r:usr_
[  102.313032]  [<ffffffffa02801a1>] ? xfs_attr_leaf_moveents.isra.2+0x91/0x280 [xfs]
[  102.313032] Padding ffff88005b4e7f20: 74 3a 73 30 00 00 00 00 49 4e 81 a4 02 02 00 00  t:s0....IN......
[  102.313033] Padding ffff88005b4e7f30: 00 00 00 00 00 00 00 00 00 00 00 01 00 00 00 00  ................
[  102.313033] Padding ffff88005b4e7f40: 00 00 00 00 00 00 00 02 51 47 09 00 00 00 00 00  ........QG......
[  102.313042]  [<ffffffffa0280467>] xfs_attr_leaf_compact+0xd7/0x130 [xfs]
[  102.313043] Padding ffff88005b4e7f50: 51 47 09 00 00 00 00 00 51 ac 1e 27 21 f1 4e ad  QG......Q..'!.N.
[  102.313043] Padding ffff88005b4e7f60: 00 00 00 00 00 00 00 f2 00 00 00 00 00 00 00 01  ................
[  102.313044] Padding ffff88005b4e7f70: 00 00 00 00 00 00 00 01 00 00 0e 01 00 00 00 00  ................
[  102.313053]  [<ffffffffa0281a2e>] xfs_attr_leaf_add+0xce/0x170 [xfs]
[  102.313053] Padding ffff88005b4e7f80: 00 00 00 00 c1 6d 78 44 ff ff ff ff 00 00 00 00  .....mxD........
[  102.313054] Padding ffff88005b4e7f90: 00 00 00 00 00 00 08 10 36 a0 00 01 00 00 00 00  ........6.......
[  102.313062]  [<ffffffffa027d850>] xfs_attr_leaf_addname+0xc0/0x3d0 [xfs]
[  102.313062] Padding ffff88005b4e7fa0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  102.313063] Padding ffff88005b4e7fb0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  102.313072]  [<ffffffffa028bd4e>] ? xfs_bmap_one_block+0x3e/0xa0 [xfs]
[  102.313072] Padding ffff88005b4e7fc0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  102.313073] Padding ffff88005b4e7fd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  102.313074] Padding ffff88005b4e7fe0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  102.313081]  [<ffffffffa027e78c>] xfs_attr_set_int+0x30c/0x420 [xfs]
[  102.313082] Padding ffff88005b4e7ff0: 00 00 00 00 00 00 00 00 00 00 00 00 00 29 01 00  .............)..
[  102.313085]  [<ffffffff811be9f4>] ? setxattr+0xa4/0x1c0
[  102.313085] FIX kmalloc-1024: Restoring 0xffff88005b4e7ec0-0xffff88005b4e7fff=0x5a
[  102.313085] 
[  102.313086] ==========
CAI Qian
> 	3. if you can't reproduce it like that, does it reproduce on
> 	  an xfstest run on a pristine system? If so, what command
> 	  line are you running, and what are the filesystem
> 	  configurations?
> 	4. if you cannot reproduce it just with xfstests and you need
> 	  to run LTP first, then can you just run the xfstest that
> 	  is failing after running LTP and see if that triggers the
> 	  problem. If it does, please take a metadump of the
> 	  filesystems after LTP has run, save them, and if the
> 	  single test then fails send me the metadumps and your
> 	  xfstests command line.
> 	5. If all else fails, bisect the kernel to identify the
> 	  commit that introduces the problem....
> 
> Cheers,
> 
> Dave.
> 
> --
> Dave Chinner
> david@xxxxxxxxxxxxx
> 
> --
> To unsubscribe, send a message with 'unsubscribe linux-mm' in
> the body to majordomo@xxxxxxxxx.  For more info on Linux MM,
> see: http://www.linux-mm.org/ .
> Don't email: <a href=mailto:"dont@xxxxxxxxx";> email@xxxxxxxxx </a>
> 
--
To unsubscribe from this list: send the line "unsubscribe stable" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html




[Index of Archives]     [Linux Kernel]     [Kernel Development Newbies]     [Linux USB Devel]     [Video for Linux]     [Linux Audio Users]     [Yosemite Hiking]     [Linux Kernel]     [Linux SCSI]