Thanks Steve, I was able to get hold of 4.19 build 134. I am also seeing the following 2 NPE in CIFS on kernel 4.16.2.1 and was hoping to see if these 2 are also fixed in 4.19 kernel ? [255120.498118] BUG: unable to handle kernel NULL pointer dereference at 0000000000000038 [255120.498520] IP: cifs_negotiate_protocol+0x15/0xd0 [cifs] [255120.498805] PGD 800000042a54c067 P4D 800000042a54c067 PUD 4276f0067 PMD 0 [255120.499119] Oops: 0000 [#1] SMP PTI [255120.499398] Modules linked in: rpcsec_gss_krb5 cmac nfsv4 nfs fscache arc4 md4 nls_utf8 cifs ccm dns_resolver nf_conntrack_netbios_ns nf_conntrack_broadcast xt_CT ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock ppdev sb_edac crct10dif_pclmul crc32_pclmul ghash_clmulni_intel pcbc vmw_balloon aesni_intel crypto_simd glue_helper cryptd intel_rapl_perf joydev input_leds pcspkr sg parport_pc parport vmw_vmci shpchp [255120.503180] i2c_piix4 nfsd nfs_acl lockd auth_rpcgss grace sunrpc ip_tables xfs libcrc32c sr_mod cdrom sd_mod ata_generic pata_acpi serio_raw crc32c_intel floppy vmxnet3 vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix drm libata mptspi mptscsih mptbase scsi_transport_spi dm_mirror dm_region_hash dm_log dm_mod dax [255120.505469] CPU: 0 PID: 6419 Comm: kworker/0:0 Not tainted 4.16.2-1.el7.elrepo.x86_64 #1 [255120.506098] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/05/2016 [255120.507412] Workqueue: cifsiod smb2_reconnect_server [cifs] [255120.508111] RIP: 0010:cifs_negotiate_protocol+0x15/0xd0 [cifs] [255120.508827] RSP: 0018:ffffc900113e7d40 EFLAGS: 00010246 [255120.509536] RAX: 0000000000000000 RBX: 0000000000000000 RCX: ffffffffa061a340 [255120.510256] RDX: ffff88042d725d00 RSI: ffff88042bb9ae00 RDI: 0000000000000000 [255120.510987] RBP: ffffc900113e7d58 R08: 0000000000000271 R09: 0000000000000000 [255120.511724] R10: 0000000000000001 R11: 000000000000026b R12: ffff88042bb9ae00 [255120.512476] R13: ffff880399229568 R14: ffffffffa061a000 R15: 0000000000000001 [255120.513246] FS: 0000000000000000(0000) GS:ffff88043fc00000(0000) knlGS:0000000000000000 [255120.514048] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [255120.514832] CR2: 0000000000000038 CR3: 000000042a5b2000 CR4: 00000000000406f0 [255120.515696] Call Trace: [255120.516545] smb2_reconnect+0x108/0x420 [cifs] [255120.517410] ? remove_wait_queue+0x60/0x60 [255120.518249] smb2_reconnect_server+0x1d4/0x340 [cifs] [255120.519102] process_one_work+0x15f/0x370 [255120.519962] worker_thread+0x4d/0x3e0 [255120.520808] kthread+0x105/0x140 [255120.521642] ? max_active_store+0x80/0x80 [255120.522456] ? kthread_bind+0x20/0x20 [255120.523254] ? do_syscall_64+0x79/0x1b0 [255120.524035] ret_from_fork+0x35/0x40 [255120.524791] Code: 64 a0 e8 ff 08 ab e0 5b 41 5c 41 5d 5d c3 0f 1f 84 00 00 00 00 00 66 66 66 66 90 55 48 89 e5 41 55 41 54 49 89 f4 53 48 8b 5e 48 <48> 8b 43 38 48 8b 90 b8 00 00 00 48 85 d2 0f 84 97 00 00 00 48 [255120.527052] RIP: cifs_negotiate_protocol+0x15/0xd0 [cifs] RSP: ffffc900113e7d40 [255120.527807] CR2: 0000000000000038 [366603.058360] BUG: unable to handle kernel NULL pointer dereference at 0000000000000000 [366603.058730] IP: cifs_reconnect+0x286/0x620 [cifs] [366603.059062] PGD 8000000426291067 P4D 8000000426291067 PUD 42af59067 PMD 0 [366603.059418] Oops: 0000 [#1] SMP PTI [366603.059803] Modules linked in: rpcsec_gss_krb5 nfsv4 nfs cmac fscache arc4 md4 nls_utf8 cifs ccm dns_resolver nf_conntrack_netbios_ns nf_conntrack_broadcast xt_CT ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sb_edac crct10dif_pclmul crc32_pclmul ghash_clmulni_intel pcbc ppdev aesni_intel crypto_simd glue_helper cryptd vmw_balloon intel_rapl_perf pcspkr joydev input_leds sg parport_pc parport i2c_piix4 [366603.064108] vmw_vmci shpchp nfsd nfs_acl lockd auth_rpcgss grace sunrpc ip_tables xfs libcrc32c sr_mod cdrom sd_mod ata_generic pata_acpi crc32c_intel serio_raw vmxnet3 mptspi mptscsih mptbase scsi_transport_spi vmwgfx ata_piix drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm libata drm floppy dm_mirror dm_region_hash dm_log dm_mod dax [366603.066558] CPU: 6 PID: 5920 Comm: cifsd Tainted: G W 4.16.2-1.el7.elrepo.x86_64 #1 [366603.067888] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/05/2016 [366603.069252] RIP: 0010:cifs_reconnect+0x286/0x620 [cifs] [366603.069947] RSP: 0018:ffffc90005a53d70 EFLAGS: 00010203 [366603.070641] RAX: 0000000000000001 RBX: ffff88042b599c00 RCX: 0000000000000000 [366603.071413] RDX: 0000000000000000 RSI: 0000000000000246 RDI: 0000000000000246 [366603.072144] RBP: ffffc90005a53dc0 R08: 0000000000000005 R09: ffff88042d6c8000 [366603.072934] R10: 00000000000002d4 R11: 000000000000b6bf R12: 0000000000000000 [366603.073669] R13: ffffc90005a53d80 R14: ffff88035d086680 R15: ffff88035d086f00 [366603.074406] FS: 0000000000000000(0000) GS:ffff88043fd80000(0000) knlGS:0000000000000000 [366603.075162] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [366603.076031] CR2: 0000000000000000 CR3: 0000000427c88000 CR4: 00000000000406e0 [366603.076896] Call Trace: [366603.077706] cifs_readv_from_socket+0x1b4/0x200 [cifs] [366603.078621] cifs_read_from_socket+0x52/0x70 [cifs] [366603.079458] cifs_demultiplex_thread+0x11b/0xa90 [cifs] [366603.080313] kthread+0x105/0x140 [366603.081182] ? cifs_handle_standard+0x190/0x190 [cifs] [366603.082048] ? kthread_bind+0x20/0x20 [366603.082896] ? do_syscall_64+0x79/0x1b0 [366603.083716] ret_from_fork+0x35/0x40 [366603.084541] Code: e7 da e0 84 c0 74 0e 49 8b 17 49 8b 47 08 48 89 42 08 48 89 10 49 8b 47 38 4d 89 3f 4c 89 ff 4d 89 7f 08 e8 4d 3f 5e e1 4d 39 ec <49> 8b 04 24 75 c4 65 4c 8b 2c 25 40 5c 01 00 eb 31 f6 05 66 0a [366603.087050] RIP: cifs_reconnect+0x286/0x620 [cifs] RSP: ffffc90005a53d70 [366603.087856] CR2: 0000000000000000 Vidhesh Ramesh From: Steve French <smfrench@xxxxxxxxx> Sent: Friday, July 17, 2020 2:44 PM To: Vidhesh Ramesh <vidhesh.ramesh@xxxxxxxxxxxx> Cc: Kenneth Dsouza <kdsouza@xxxxxxxxxx>; CIFS <linux-cifs@xxxxxxxxxxxxxxx> Subject: Re: Oops in cifs_match_super() - linux kernel 4.16.2-1.el7.elrepo.x86_64 I see only 13 'cc:stable' patches missing from 4.19.113 for cifs.ko. See below: smfrench@smfrench-ThinkPad-P52:~/linux-stable-rc$ git log --oneline v4.19.113.. fs/cifs 38bcc785c2eb cifs: update ctime and mtime during truncate e1afc2c1baa5 cifs: Fix the target file was deleted when rename failed. 6f9e471d62ae SMB3: Honor lease disabling for multiuser mounts da6845a443dc SMB3: Honor persistent/resilient handle flags for multiuser mounts c0651cd829ee SMB3: Honor 'seal' flag for multiuser mounts 1fa012710599 SMB3: Honor 'posix' flag for multiuser mounts 39dad7304057 cifs/smb3: Fix data inconsistent when zero file range f4c710c4a39b cifs/smb3: Fix data inconsistent when punch hole 4d9248f989ec cifs: Fix null pointer check in cifs_read ad149b6e08f1 cifs: fix leaked reference on requeued write 6c662c519277 cifs: protect updating server->dstaddr with a spinlock 731a3bc2be26 cifs: Allocate encryption header through kmalloc 9bc022589575 CIFS: Fix bug which the return value by asynchronous read is error There are 113 cifs (cc:stable) patches in the 4.19.133 tree (out of the 628 total patches to cifs.ko between 4.19 and now, 5.8-rc). See attached On Fri, Jul 17, 2020 at 4:23 PM Vidhesh Ramesh <vidhesh.ramesh@xxxxxxxxxxxx> wrote: > > Thanks Steve. > We have few more bug fixes that we need in 4.19 LT kernel. However I am unable to get a built kernel RPM of latest build 133 of kernel 4.19. > The latest I could get is build 113 of kernel 4.19 at https://buildlogs.centos.org/c7-kernels.x86_64/kernel/20200330213326/4.19.113-300.el8.x86_64/kernel-4.19.113-300.el7.x86_64.rpm > > Vidhesh Ramesh > > From: Steve French <smfrench@xxxxxxxxx> > Sent: Friday, July 17, 2020 7:05 AM > To: Vidhesh Ramesh <vidhesh.ramesh@xxxxxxxxxxxx> > Cc: Kenneth Dsouza <kdsouza@xxxxxxxxxx>; CIFS <linux-cifs@xxxxxxxxxxxxxxx> > Subject: Re: Oops in cifs_match_super() - linux kernel 4.16.2-1.el7.elrepo.x86_64 > > I am not familiar with your distro but a Google search matched on this as a more recent kernel rpm download with similar sounding name http://lists.elrepo.org/pipermail/elrepo/2018-May/004282.html > > But there should be more recent kernels than that are eg based on newer 4.6.18 stable branch https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/log/?h=linux-4.16.y > > On Fri, Jul 17, 2020, 04:12 Vidhesh Ramesh <vidhesh.ramesh@xxxxxxxxxxxx> wrote: > Thanks Steve and Kenneth for your responses. > > I did look at the commits and found that this has been fixed in a patch of 4.19. Is there an archive where I can download latest 4.19 kernel rpm so that I can verify the fix ? > > Vidhesh Ramesh > > > > > > > From: Steve French <smfrench@xxxxxxxxx> > Sent: Tuesday, July 14, 2020 10:49 AM > To: Kenneth Dsouza <kdsouza@xxxxxxxxxx> > Cc: Vidhesh Ramesh <vidhesh.ramesh@xxxxxxxxxxxx>; linux-cifs@xxxxxxxxxxxxxxx <linux-cifs@xxxxxxxxxxxxxxx> > Subject: Re: Oops in cifs_match_super() - linux kernel 4.16.2-1.el7.elrepo.x86_64 > > If you have a particular stable kernel used for your distro you can > email stable@xxxxxxxxxxxxxxx with the commit id and version you want > it backported for. > > On Tue, Jul 14, 2020 at 12:43 PM Kenneth Dsouza <kdsouza@xxxxxxxxxx> wrote: > > > > You are hitting a known bug which has been fixed by below patch and is > > part of v5.6 and above. > > > > commit fe1292686333d1dadaf84091f585ee903b9ddb84 > > Author: Ronnie Sahlberg <lsahlber@xxxxxxxxxx> > > Date: Wed Jan 22 11:07:56 2020 +1000 > > > > cifs: fix NULL dereference in match_prepath > > > > RHBZ: 1760879 > > > > Fix an oops in match_prepath() by making sure that the prepath string is not > > NULL before we pass it into strcmp(). > > > > This is similar to other checks we make for example in cifs_root_iget() > > > > Signed-off-by: Ronnie Sahlberg <lsahlber@xxxxxxxxxx> > > Signed-off-by: Steve French <stfrench@xxxxxxxxxxxxx> > > > > On Tue, Jul 14, 2020 at 11:05 PM Vidhesh Ramesh > > <vidhesh.ramesh@xxxxxxxxxxxx> wrote: > > > > > > Adding linux-cifs mailing list. > > > > > > Vidhesh Ramesh > > > > > > > > > From: Vidhesh Ramesh <vidhesh.ramesh@xxxxxxxxxxxx> > > > Sent: Tuesday, July 14, 2020 10:18 AM > > > To: Steve French <sfrench@xxxxxxxxx>; linux-cifs@xxxxxxxxxxxxxxx <linux-cifs@xxxxxxxxxxxxxxx>; samba-technical@xxxxxxxxxxxxxxx <samba-technical@xxxxxxxxxxxxxxx> > > > Cc: Ameya Usgaonkar <ameya.usgaonkar@xxxxxxxxxxxx> > > > Subject: Re: Oops in cifs_match_super() - linux kernel 4.16.2-1.el7.elrepo.x86_64 > > > > > > Hi Steve et all, > > > > > > Resending this as a gentle reminder if anyone got a chance to look at the below mentioned oops kernel panic. > > > > > > > > > Vidhesh Ramesh > > > > > > > > > From: Vidhesh Ramesh <vidhesh.ramesh@xxxxxxxxxxxx> > > > Sent: Thursday, July 2, 2020 12:21 AM > > > To: Steve French <sfrench@xxxxxxxxx>; linux-cifs@xxxxxxxxxxxxxxx <linux-cifs@xxxxxxxxxxxxxxx>; samba-technical@xxxxxxxxxxxxxxx <samba-technical@xxxxxxxxxxxxxxx> > > > Cc: Ameya Usgaonkar <ameya.usgaonkar@xxxxxxxxxxxx> > > > Subject: Re: Oops in cifs_match_super() - linux kernel 4.16.2-1.el7.elrepo.x86_64 > > > > > > Hi Steve et all, > > > > > > Resending this as a gentle reminder if anyone got a chance to look at the below mentioned oops kernel panic. > > > > > > Vidhesh Ramesh > > > > > > > > > > > > > > > > > > > > > > > > From: Ameya Usgaonkar <ameya.usgaonkar@xxxxxxxxxxxx> > > > Sent: Wednesday, June 24, 2020 10:26 PM > > > To: Steve French <sfrench@xxxxxxxxx>; linux-cifs@xxxxxxxxxxxxxxx <linux-cifs@xxxxxxxxxxxxxxx>; samba-technical@xxxxxxxxxxxxxxx <samba-technical@xxxxxxxxxxxxxxx> > > > Cc: Vidhesh Ramesh <vidhesh.ramesh@xxxxxxxxxxxx> > > > Subject: Oops in cifs_match_super() - linux kernel 4.16.2-1.el7.elrepo.x86_64 > > > > > > Hi Steve et al, > > > My name is Ameya and I work for www.komprise.com. The linux kernel with above mentioned version has been panicing, though randomly, but the stack trace appears is consistent. You can find more details in the attachments. Below is the brief description of the problem - > > > > > > [1.] One line summary of the problem: > > > oops kernel panic > > > [2.] Full description of the problem/report: > > > ESX VM hangs with a kernel panic when cifs shares are mounted. Unable to ssh to the VM and the console of the VM is also not responding. With kdump service running and core collected the VM restarts successfully. > > > [3.] Keywords (i.e., modules, networking, kernel): > > > cifs, kernel, panic, strcmp, mount > > > [4.] Kernel information > > > [4.1.] Kernel version (from /proc/version): > > > Linux version 4.16.2-1.el7.elrepo.x86_64 (mockbuild@Build64R7) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-28) (GCC)) #1 SMP Thu Apr 12 09:08:05 EDT 2018 > > > [4.2.] Kernel .config file: > > > Please check the file uploaded > > > [5.] Most recent kernel version which did not have the bug: > > > [6.] Output of Oops.. message (if applicable) with symbolic information > > > resolved (see Documentation/admin-guide/oops-tracing.rst) > > > [442282.069937] BUG: unable to handle kernel NULL pointer dereference at 0000000000000000 > > > [442282.071474] IP: strcmp+0xe/0x30 > > > [442282.072892] PGD 0 P4D 0 > > > [442282.074196] Oops: 0000 [#1] SMP PTI > > > [442282.075561] Modules linked in: binfmt_misc fuse cmac rpcsec_gss_krb5 nfsv4 arc4 md4 nls_utf8 cifs ccm dns_resolver nfsv3 nfs fscache nf_conntrack_netbios_ns nf_conntrack_broadcast xt_CT ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sb_edac crct10dif_pclmul crc32_pclmul ghash_clmulni_intel pcbc ppdev aesni_intel vmw_balloon crypto_simd glue_helper cryptd sg intel_rapl_perf input_leds pcspkr joydev shpchp > > > [442282.090386] parport_pc parport i2c_piix4 vmw_vmci nfsd nfs_acl lockd auth_rpcgss grace sunrpc ip_tables xfs libcrc32c sr_mod cdrom ata_generic sd_mod pata_acpi crc32c_intel serio_raw vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm mptspi scsi_transport_spi ata_piix mptscsih vmxnet3 drm mptbase libata floppy dm_mirror dm_region_hash dm_log dm_mod dax > > > [442282.095332] CPU: 4 PID: 9273 Comm: mount.cifs Tainted: G W 4.16.2-1.el7.elrepo.x86_64 #1 > > > [442282.097802] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/05/2016 > > > [442282.100363] RIP: 0010:strcmp+0xe/0x30 > > > [442282.101645] RSP: 0018:ffffc9001bff7c88 EFLAGS: 00010202 > > > [442282.102919] RAX: 0000000000000001 RBX: ffff8802ad7c2400 RCX: 0000000001240404 > > > [442282.104207] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 0000000000000001 > > > [442282.105497] RBP: ffffc9001bff7c88 R08: 0000000001240404 R09: 0000000000000001 > > > [442282.106765] R10: ffff88017d2c2b40 R11: ffff8801744c0e50 R12: ffff88011f793000 > > > [442282.108038] R13: ffff88042b105800 R14: ffffc9001bff7d98 R15: ffff8801b63c0f00 > > > [442282.109318] FS: 00007f3e90ee6780(0000) GS:ffff88043fd00000(0000) knlGS:0000000000000000 > > > [442282.110630] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > > > [442282.111970] CR2: 0000000000000000 CR3: 000000011b2ea000 CR4: 00000000000406e0 > > > [442282.113405] Call Trace: > > > [442282.114849] cifs_match_super+0x123/0x210 [cifs] > > > [442282.116211] ? cifs_prune_tlinks+0xe0/0xe0 [cifs] > > > [442282.117541] sget_userns+0x88/0x4a0 > > > [442282.118877] ? cifs_kill_sb+0x30/0x30 [cifs] > > > [442282.120203] ? cifs_prune_tlinks+0xe0/0xe0 [cifs] > > > [442282.121502] sget+0x7d/0xa0 > > > [442282.122781] ? cifs_kill_sb+0x30/0x30 [cifs] > > > [442282.124072] cifs_do_mount+0x168/0x5a0 [cifs] > > > [442282.125364] mount_fs+0x3e/0x150 > > > [442282.126627] vfs_kern_mount+0x67/0x130 > > > [442282.127850] do_mount+0x1f5/0xca0 > > > [442282.129047] SyS_mount+0x83/0xd0 > > > [442282.130257] do_syscall_64+0x79/0x1b0 > > > [442282.131583] entry_SYSCALL_64_after_hwframe+0x3d/0xa2 > > > [442282.132687] RIP: 0033:0x7f3e907fdfea > > > [442282.133754] RSP: 002b:00007ffee9139868 EFLAGS: 00000202 ORIG_RAX: 00000000000000a5 > > > [442282.134842] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f3e907fdfea > > > [442282.135909] RDX: 000055d13f5953b2 RSI: 000055d13f5953f9 RDI: 00007ffee913bb51 > > > [442282.136944] RBP: 00007ffee913bb46 R08: 000055d1415e80d0 R09: 0000000000001000 > > > [442282.137962] R10: 0000000000000000 R11: 0000000000000202 R12: 00007f3e90ef0000 > > > [442282.138957] R13: 000055d1415e80d0 R14: 00007f3e90ef290f R15: 0000000000000000 > > > [442282.139918] Code: 80 3a 00 75 f7 48 83 c6 01 0f b6 4e ff 48 83 c2 01 84 c9 88 4a ff 75 ed 5d c3 0f 1f 00 55 48 89 e5 eb 04 84 c0 74 18 48 83 c7 01 <0f> b6 47 ff 48 83 c6 01 3a 46 ff 74 eb 19 c0 83 c8 01 5d c3 31 > > > [442282.142829] RIP: strcmp+0xe/0x30 RSP: ffffc9001bff7c88 > > > [442282.143759] CR2: 0000000000000000 > > > [7.] A small shell script or example program which triggers the > > > problem (if possible) > > > > > > My colleague Vidhesh (CC'ed) will be glad to share further details on the test scenario and/or in-house reproduction. Please let us know of workarounds, if any. > > > > > > Thanks and Regards, > > > = Ameya > > > > > > > > > > > > > > -- > Thanks, > > Steve -- Thanks, Steve
