In our test labs we discovered a bug with the latest 3.10-rt kernel.
When investigating, I found that it was actually a bug in the 3.10.18
kernel that we based on. With that, I bisected it down to this commit:
commit 506cdb8909a1a739c7585c680c6bd4b3d1247564
Author: Nicolas Dichtel <nicolas.dichtel@xxxxxxxxx>
Date: Tue Oct 1 18:05:00 2013 +0200
ip6tnl: allow to use rtnl ops on fb tunnel
[ Upstream commit bb8140947a247b9aa15652cc24dc555ebb0b64b0 ]
rtnl ops where introduced by c075b13098b3 ("ip6tnl: advertise tunnel param
rtnl"), but I forget to assign rtnl ops to fb tunnels.
Now that it is done, we must remove the explicit call to
unregister_netdevice_queue(), because the fallback tunnel is added to the
in ip6_tnl_destroy_tunnels() when checking rtnl_link_ops of all netdevices
is valid since commit 0bd8762824e7 ("ip6tnl: add x-netns support")).
The bug we see is caused by simply loading and unloading the ip6_tunnel
module.
modprobe ip6_tunnel; rmmod ip6_tunnel
Which causes the following oops:
[ 43.423028] BUG: unable to handle kernel NULL pointer dereference at 0000000000000008
[ 43.424010] IP: [<ffffffffa0534f51>] ip6_tnl_exit_net+0x71/0x93 [ip6_tunnel]
[ 43.424010] PGD 776f4067 PUD 7810a067 PMD 0
[ 43.424010] Oops: 0000 [#1] PREEMPT SMP
[ 43.424010] Modules linked in: ip6_tunnel(-) tunnel6 ipt_MASQUERADE sunrpc ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ip6_tables uinput snd_hda_codec_idt kvm_i
ntel snd_hda_intel snd_hda_codec kvm snd_hwdep snd_seq snd_seq_device snd_pcm snd_page_alloc snd_timer snd shpchp i2c_i801 soundcore microcode pata_acpi firewire_ohci firewire_core
crc_itu_t ata_generic i915 drm_kms_helper drm i2c_algo_bit i2c_core video
[ 43.424010] CPU: 1 PID: 2731 Comm: rmmod Not tainted 3.10.15-test+ #105
[ 43.424010] Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./To be filled by O.E.M., BIOS SDBLI944.86P 05/08/2007
[ 43.424010] task: ffff880078b01460 ti: ffff880077bf4000 task.ti: ffff880077bf4000
[ 43.424010] RIP: 0010:[<ffffffffa0534f51>] [<ffffffffa0534f51>] ip6_tnl_exit_net+0x71/0x93 [ip6_tunnel]
[ 43.424010] RSP: 0018:ffff880077bf5e08 EFLAGS: 00010246
[ 43.424010] RAX: 0000000000000000 RBX: 0000000000000100 RCX: 0000000000000003
[ 43.424010] RDX: ffff88007d480000 RSI: ffff880077bf5e08 RDI: ffff880077bf4000
[ 43.424010] RBP: ffff880077bf5e38 R08: ffff880077bf5d68 R09: ffffffff81aa20d0
[ 43.424010] R10: ffffffff81aa20d0 R11: ffffffff81aa20d0 R12: 0000000000000000
[ 43.424010] R13: ffff88007794b400 R14: ffff880077bf5e08 R15: 0000000000000001
[ 43.424010] FS: 00007fbc2ee27700(0000) GS:ffff88007d480000(0000) knlGS:0000000000000000
[ 43.424010] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 43.424010] CR2: 0000000000000008 CR3: 0000000077bd0000 CR4: 00000000000007e0
[ 43.424010] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 43.424010] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 43.424010] Stack:
[ 43.424010] ffff880077bf5e08 ffff880077bf5e08 ffffffffa0536f50 ffffffff81aa0900
[ 43.424010] ffff880077bf5e78 0000000000000000 ffff880077bf5e68 ffffffff81408df4
[ 43.424010] 0000000000000000 ffffffffa0536f50 ffffffff81aa1820 ffff880077bf5e78
[ 43.424010] Call Trace:
[ 43.424010] [<ffffffff81408df4>] ops_exit_list+0x27/0x50
[ 43.424010] [<ffffffff814090ba>] unregister_pernet_operations+0x61/0x93
[ 43.424010] [<ffffffff81409122>] unregister_pernet_device+0x36/0x47
[ 43.424010] [<ffffffffa05367d4>] ip6_tunnel_cleanup+0x70/0x72 [ip6_tunnel]
[ 43.424010] [<ffffffff81083ef5>] SyS_delete_module+0x20b/0x27d
[ 43.424010] [<ffffffff81244cae>] ? trace_hardirqs_on_thunk+0x3a/0x3c
[ 43.424010] [<ffffffff81503902>] system_call_fastpath+0x16/0x1b
[ 43.424010] Code: 24 08 4c 89 f6 e8 8c 88 ed e0 4d 8b 24 24 4d 85 e4 75 ea 48 83 c3 08 48 81 fb 00 01 00 00 75 d6 49 8b 85 08 01 00 00 48 8d 75 d0 <48> 8b 78 08 e8 62 88 ed e0 48 8d 7d d0 e8 84 77 ed e0 e8 90 62
[ 43.424010] RIP [<ffffffffa0534f51>] ip6_tnl_exit_net+0x71/0x93 [ip6_tunnel]
[ 43.424010] RSP <ffff880077bf5e08>
[ 43.424010] CR2: 0000000000000008
[ 43.708059] ---[ end trace ea2c125633de7c64 ]---
(gdb) li *ip6_tnl_exit_net+0x71
0xf51 is in ip6_tnl_exit_net (/home/rostedt/work/git/linux-trace.git/net/ipv6/ip6_tunnel.c:1715).
1710 t = rtnl_dereference(t->next);
1711 }
1712 }
1713
1714 t = rtnl_dereference(ip6n->tnls_wc[0]);
1715 unregister_netdevice_queue(t->dev, &list);
1716 unregister_netdevice_many(&list);
1717 }
1718
1719 static int __net_init ip6_tnl_init_net(struct net *net)
Thus, this got called with ip6n->tnsl_wc[0] as NULL.
I ran the following trace command on this:
# modprobe ip6_tunnel
# trace-cmd start -p function_graph -g SyS_delete_module
# rmmod ip6_tunnel
and traced the flow of functions that lead up to the crash: Full dump
can be found here: http://rostedt.homelinux.com/private/ip6_tunnel.trace
ip6_tnl_dev_uninit() which is called by rollback_registered_many() sets
tnls_wc[0] to NULL. Later unregistered_pernet_device() gets called,
which eventually calls ip6_tnl_exit_net() which references the
tnls_wc[0] unconditionally. This looks to be where the bug happens.
Finally, after digging through all this, I looked at the original
commit that was backported to 3.10 and noticed that the backport
doesn't include the entire change. It also has:
+++ b/net/ipv6/ip6_tunnel.c
@@ -1731,8 +1731,6 @@ static void __net_exit ip6_tnl_destroy_tunnels(struct ip
}
}
- t = rtnl_dereference(ip6n->tnls_wc[0]);
- unregister_netdevice_queue(t->dev, &list);
unregister_netdevice_many(&list);
}
Which, when applied to 3.10.18, fixes the bug. Was there a reason that
this part of the commit wasn't backported? or was this just an oversight?
-- Steve
--
To unsubscribe from this list: send the line "unsubscribe stable" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html