Re: [Bug] Kernel Panic on Deletion of the network-namespace containing the SocketCAN interface

[Oliver Hartkopp <socketcan@xxxxxxxxxxxx>]

I found it!

diff --git a/net/core/dev.c b/net/core/dev.c
index 90b59fc50dc9..add15461a9e2 100644
--- a/net/core/dev.c
+++ b/net/core/dev.c
@@ -10517,7 +10517,7 @@ static void __net_exit 
default_device_exit(struct net *net)
                        continue;

                /* Leave virtual devices for the generic cleanup */
-               if (dev->rtnl_link_ops)
+               if ((dev->rtnl_link_ops) && (dev->type != ARPHRD_CAN))
                        continue;

                /* Push remaining network devices to init_net */

The problem is, that 'real' CAN interfaces use the rtnl_link_ops to 
configure bitrates and other CAN controller specific settings.

But the fact that rtnl_link_ops are available makes somebody thinking 
this is only a virtual interface - and therefore the transition of the 
interface back to the root namespace is skipped.

The patch above fixes the issue but I'm not sure if we need a more 
general solution here.

Best,
Oliver

On 17.07.20 13:02, Oliver Hartkopp wrote:
> On 17.07.20 09:55, Stéphane Grosjean wrote:
> > There's a first WARNING kernel message just when the namespace is 
> > deleted. The WARNING turns into a BUG (kernel NULL pointer 
> > dereference) when removing the interface itself (for example, when the 
> > driver module is removed from memory). Note that the issue occurs with 
> > all our internal as well as USB CAN interfaces.
> >
> > Obviously, the problem doesn't appear when you put the interface back 
> > in the root namespace before the destruction, or when the interface is 
> > a true Ethernet network interface.
>
> Yes. I checked that with an USB Ethernet interface I have at hand here - 
> and when deleting the test namespace it just emerges in the root 
> namespace again.
>
> So I wonder what's missing in our configuration for CAN interfaces that 
> this transition is not performed :-/
>
> Best regards,
> Oliver
>
>
> > Context:
> >
> > $ uname -a
> > Linux linux-dev 5.4.0-39-generic #43-Ubuntu SMP Fri Jun 19 10:28:31 
> > UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
> >
> > $ dmesg | grep peak_pci
> > [   19.028048] peak_pci 0000:0a:00.0: enabling device (0100 -> 0102)
> > [   19.034283] peak_pci 0000:0a:00.0: can6 at 
> > reg_base=0x00000000136b8b0b cfg_base=0x00000000b826597c irq=27
> > [   19.034378] peak_pci 0000:0a:00.0: can7 at 
> > reg_base=0x000000006b3de9a0 cfg_base=0x00000000b826597c irq=27
> >
> > # ip netns add test
> > # ip link set dev can6 netns test
> > # ip netns delete test
> >
> > [ 1755.805241] ------------[ cut here ]------------
> > [ 1755.805251] WARNING: CPU: 8 PID: 2635 at net/core/dev.c:10039 
> > netdev_exit+0x44/0x50
> > [ 1755.805252] Modules linked in: vboxnetadp(OE) vboxnetflt(OE) 
> > vboxdrv(OE) md4 nls_utf8 cifs libarc4 fscache libdes cfg80211 
> > nls_iso8859_1 intel_rapl_msr intel_rapl_common x86_pkg_temp_thermal 
> > intel_powerclamp coretemp kvm_intel peak_usb 8812au(OE) joydev 
> > plin(OE) kvm input_leds pcan(OE) eeepc_wmi snd_hda_codec_hdmi 
> > snd_hda_codec_realtek nouveau pcmcia snd_hda_codec_generic asus_wmi 
> > pcmcia_core crct10dif_pclmul ghash_clmulni_intel peak_pci sja1000 
> > peak_pciefd ledtrig_audio ttm aesni_intel drm_kms_helper snd_hda_intel 
> > snd_intel_dspcfg fb_sys_fops syscopyarea sysfillrect sysimgblt can_dev 
> > snd_hda_codec snd_hda_core crypto_simd snd_hwdep cryptd snd_pcm 
> > glue_helper snd_seq_midi snd_seq_midi_event snd_rawmidi snd_seq 
> > snd_seq_device snd_timer sparse_keymap intel_cstate intel_rapl_perf 
> > snd video mei_me mei soundcore wmi_bmof intel_wmi_thunderbolt mxm_wmi 
> > mac_hid sch_fq_codel parport_pc ppdev lp parport drm ip_tables 
> > x_tables autofs4 hid_generic usbhid uas usb_storage hid crc32_pclmul 
> > igb e1000e
> > [ 1755.805306]  ahci i2c_i801 i2c_algo_bit lpc_ich libahci dca wmi
> > [ 1755.805315] CPU: 8 PID: 2635 Comm: kworker/u24:0 Tainted: 
> > G           OE     5.4.0-39-generic #43-Ubuntu
> > [ 1755.805316] Hardware name: ASUS All Series/X99-E WS, BIOS 4001 
> > 05/27/2019
> > [ 1755.805319] Workqueue: netns cleanup_net
> > [ 1755.805324] RIP: 0010:netdev_exit+0x44/0x50
> > [ 1755.805327] Code: 8b bb 30 01 00 00 e8 8b 9d 97 ff 48 81 fb 00 21 
> > be b5 74 13 48 8b 83 90 00 00 00 48 81 c3 90 00 00 00 48 39 c3 75 03 
> > 5b 5d c3 <0f> 0b eb f9 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 89 
> > e5 41
> > [ 1755.805329] RSP: 0018:ffffb12d0169fdc8 EFLAGS: 00010287
> > [ 1755.805331] RAX: ffff9624c8a70050 RBX: ffff96247642a710 RCX: 
> > 000000008010000b
> > [ 1755.805333] RDX: 000000008010000c RSI: 0000000000000001 RDI: 
> > ffff9624cdc06a00
> > [ 1755.805334] RBP: ffffb12d0169fdd0 R08: 0000000000000000 R09: 
> > ffffffffb4d2e300
> > [ 1755.805335] R10: ffff9624a0d84000 R11: 0000000000000001 R12: 
> > ffffb12d0169fe20
> > [ 1755.805337] R13: ffffffffb5be3f20 R14: ffffffffb5be3f28 R15: 
> > ffff962498ea39d8
> > [ 1755.805339] FS:  0000000000000000(0000) GS:ffff9624cfa00000(0000) 
> > knlGS:0000000000000000
> > [ 1755.805341] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > [ 1755.805342] CR2: 000055fbb59cd8e8 CR3: 000000004aa0a006 CR4: 
> > 00000000001606e0
> > [ 1755.805344] Call Trace:
> > [ 1755.805350]  ops_exit_list.isra.0+0x3b/0x70
> > [ 1755.805353]  cleanup_net+0x1f0/0x300
> > [ 1755.805359]  process_one_work+0x1eb/0x3b0
> > [ 1755.805363]  worker_thread+0x4d/0x400
> > [ 1755.805367]  kthread+0x104/0x140
> > [ 1755.805370]  ? process_one_work+0x3b0/0x3b0
> > [ 1755.805373]  ? kthread_park+0x90/0x90
> > [ 1755.805378]  ret_from_fork+0x35/0x40
> > [ 1755.805382] ---[ end trace 832a75ad96f8105e ]---
> > [ 1755.805410] ------------[ cut here ]------------
> > [ 1755.805416] WARNING: CPU: 8 PID: 2635 at fs/proc/proc_sysctl.c:1714 
> > retire_sysctl_set+0x14/0x18
> > [ 1755.805417] Modules linked in: vboxnetadp(OE) vboxnetflt(OE) 
> > vboxdrv(OE) md4 nls_utf8 cifs libarc4 fscache libdes cfg80211 
> > nls_iso8859_1 intel_rapl_msr intel_rapl_common x86_pkg_temp_thermal 
> > intel_powerclamp coretemp kvm_intel peak_usb 8812au(OE) joydev 
> > plin(OE) kvm input_leds pcan(OE) eeepc_wmi snd_hda_codec_hdmi 
> > snd_hda_codec_realtek nouveau pcmcia snd_hda_codec_generic asus_wmi 
> > pcmcia_core crct10dif_pclmul ghash_clmulni_intel peak_pci sja1000 
> > peak_pciefd ledtrig_audio ttm aesni_intel drm_kms_helper snd_hda_intel 
> > snd_intel_dspcfg fb_sys_fops syscopyarea sysfillrect sysimgblt can_dev 
> > snd_hda_codec snd_hda_core crypto_simd snd_hwdep cryptd snd_pcm 
> > glue_helper snd_seq_midi snd_seq_midi_event snd_rawmidi snd_seq 
> > snd_seq_device snd_timer sparse_keymap intel_cstate intel_rapl_perf 
> > snd video mei_me mei soundcore wmi_bmof intel_wmi_thunderbolt mxm_wmi 
> > mac_hid sch_fq_codel parport_pc ppdev lp parport drm ip_tables 
> > x_tables autofs4 hid_generic usbhid uas usb_storage hid crc32_pclmul 
> > igb e1000e
> > [ 1755.805454]  ahci i2c_i801 i2c_algo_bit lpc_ich libahci dca wmi
> > [ 1755.805460] CPU: 8 PID: 2635 Comm: kworker/u24:0 Tainted: G        
> > W  OE     5.4.0-39-generic #43-Ubuntu
> > [ 1755.805461] Hardware name: ASUS All Series/X99-E WS, BIOS 4001 
> > 05/27/2019
> > [ 1755.805463] Workqueue: netns cleanup_net
> > [ 1755.805467] RIP: 0010:retire_sysctl_set+0x14/0x18
> > [ 1755.805469] Code: 00 00 00 00 49 c7 40 48 00 00 00 00 49 c7 40 50 
> > 00 00 00 00 c3 90 0f 1f 44 00 00 55 48 8b 47 58 48 89 e5 48 85 c0 75 
> > 02 5d c3 <0f> 0b 5d c3 0f 1f 44 00 00 55 48 89 e5 48 83 ec 60 48 89 4c 
> > 24 48
> > [ 1755.805471] RSP: 0018:ffffb12d0169fdc0 EFLAGS: 00010282
> > [ 1755.805473] RAX: ffff9624a697ad58 RBX: ffff96247642a680 RCX: 
> > 0000000080150009
> > [ 1755.805475] RDX: ffff96247642a6b0 RSI: ffffffffb5bfeb48 RDI: 
> > ffff96247642a730
> > [ 1755.805476] RBP: ffffb12d0169fdc0 R08: 0000000000000000 R09: 
> > ffffffffb4776d00
> > [ 1755.805477] R10: ffff9624cad0a9c0 R11: 0000000000000001 R12: 
> > ffffb12d0169fe20
> > [ 1755.805479] R13: ffffffffb5bfeb40 R14: ffffffffb5bfeb48 R15: 
> > ffff962498ea39d8
> > [ 1755.805481] FS:  0000000000000000(0000) GS:ffff9624cfa00000(0000) 
> > knlGS:0000000000000000
> > [ 1755.805482] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > [ 1755.805483] CR2: 000055fbb59cd8e8 CR3: 000000004aa0a006 CR4: 
> > 00000000001606e0
> > [ 1755.805485] Call Trace:
> > [ 1755.805490]  sysctl_net_exit+0x15/0x20
> > [ 1755.805493]  ops_exit_list.isra.0+0x3b/0x70
> > [ 1755.805496]  cleanup_net+0x1f0/0x300
> > [ 1755.805500]  process_one_work+0x1eb/0x3b0
> > [ 1755.805503]  worker_thread+0x4d/0x400
> > [ 1755.805507]  kthread+0x104/0x140
> > [ 1755.805510]  ? process_one_work+0x3b0/0x3b0
> > [ 1755.805512]  ? kthread_park+0x90/0x90
> > [ 1755.805517]  ret_from_fork+0x35/0x40
> > [ 1755.805520] ---[ end trace 832a75ad96f8105f ]---
> >
> >
> >
> > — Stéphane
> >
> > -----Message d'origine-----
> > De : linux-can-owner@xxxxxxxxxxxxxxx <linux-can-owner@xxxxxxxxxxxxxxx> 
> > De la part de Oliver Hartkopp
> > Envoyé : jeudi 16 juillet 2020 20:38
> > À : Philipp Lehmann <leph1016@xxxxxxxxxxxxxxx>; wg@xxxxxxxxxxxxxx; 
> > mkl@xxxxxxxxxxxxxx
> > Cc : linux-can@xxxxxxxxxxxxxxx; christian.sauer.w@xxxxxxxxxxxxxxxx
> > Objet : Re: [Bug] Kernel Panic on Deletion of the network-namespace 
> > containing the SocketCAN interface
> >
> > Hi Philipp,
> >
> > thanks for the report and its reproducer!
> >
> > I assumed the interfaces - at least in the case of 'real' hardware CAN 
> > interfaces - to me moved back to the root name space ... well.
> >
> > I'll take a look at it.
> >
> > Best regards,
> > Oliver
> >
> > On 16.07.20 18:46, Philipp Lehmann wrote:
> > > If a SocketCAN Interface (Tested with a PCAN-USB adapter) is moved 
> > > into a network-namespace and the network namespace is deleted 
> > > afterwards, without moving the device out of the namespace prior to 
> > > the deletion. The device could not be found in any of the network 
> > > namespaces afterwards, only a reboot of the system fixes this. If the 
> > > device is instead removed from the USB-Bus without a restart, a 
> > > kernel panic is the result.
> > >
> > >
> > > Output of uname -r [Linux cpc4x 5.4.0-40-generic #44-Ubuntu SMP Tue
> > > Jun 23 00:01:04 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux]
> > >
> > >
> > > The bug could be reproduced with the following steps:
> > >
> > >
> > > 1. Connect the (USB)-SocketCAN device to the host
> > >
> > > 2. Add a new network namespace [sudo ip netns add test] 3. Move the
> > > CAN-interface to the network name-space [sudo ip link set dev can0
> > > netns test] 4. Delete the namespace [sudo ip netns delete test] 5.
> > > Remove the adapter from the USB-Bus. In most cases this should result
> > > in a kernel panic
> >
> > --
> > PEAK-System Technik GmbH
> > Sitz der Gesellschaft Darmstadt - HRB 9183
> > Geschaeftsfuehrung: Alexander Gach / Uwe Wilhelm
> > Unsere Datenschutzerklaerung mit wichtigen Hinweisen
> > zur Behandlung personenbezogener Daten finden Sie unter
> > www.peak-system.com/Datenschutz.483.0.html