Hi Jeff,
thanks for the fast reply. I think before opening a report I will try to collect more information. On our system it seems to affect only one specific workload and the error is triggered very rarely. It might be possible to change the workload to prevent this from happening. Is this related to file name lengths? Is it utf8 encoding/decoding? Could we simply ask the user to rename some files/directories as a workaround?
Thanks!
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
________________________________________
From: Jeff Layton <jlayton@xxxxxxxxxx>
Sent: 14 February 2022 16:21:56
To: Frank Schilder; Ilya Dryomov
Cc: ceph-users
Subject: Re: kernel BUG at include/linux/ceph/decode.h:262
On Mon, 2022-02-14 at 15:05 +0000, Frank Schilder wrote:
> Hi Ilya and Jeff,
>
> it happened again and this time we got a little bit more. I'm not sure it is complete though:
>
> Feb 13 19:27:56 sn311 kernel: ------------[ cut here ]------------
> Feb 13 19:27:56 sn311 kernel: kernel BUG at include/linux/ceph/decode.h:262!
> Feb 13 19:27:56 sn311 kernel: invalid opcode: 0000 [#1] SMP
> Feb 13 19:27:56 sn311 kernel: Modules linked in: squashfs loop overlay(T) 8021q garp mrp stp llc nfsv3 nfs_acl nfs lockd grace fscache beegfs(OE) ceph libceph libcrc32c dns_resolver rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib
> (OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) dcdbas mlx4_core(OE) amd64_edac_mod edac_mce_amd kvm irqbypass crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ab
> lk_helper cryptd pcspkr sg mgag200 ahci ttm libahci drm_kms_helper libata k10temp syscopyarea sysfillrect ipmi_si sysimgblt ipmi_devintf fb_sys_fops ipmi_msghandler drm drm_panel_orientation_quirks i2c_piix4 ccp acpi_power_meter acpi_cp
> ufreq sunrpc knem(OE) ip_tables mlx5_ib(OE) megaraid_sas ib_uverbs(OE) ib_core(OE) mlx5_core(OE) mlxfw(OE) devlink mlx_compat(OE)
> Feb 13 19:27:56 sn311 kernel: igb i2c_algo_bit ixgbe dca ptp pps_core mdio sd_mod crc_t10dif crct10dif_common
> Feb 13 19:27:56 sn311 kernel: CPU: 22 PID: 127515 Comm: octave-cli Tainted: G OE ------------ T 3.10.0-957.12.2.el7.x86_64 #1
> Feb 13 19:27:56 sn311 kernel: Hardware name: Dell Inc. PowerEdge R7425/02MJ3T, BIOS 1.8.7 04/02/2019
> Feb 13 19:27:56 sn311 kernel: task: ffff9d43adee2080 ti: ffff9d43aeb60000 task.ti: ffff9d43aeb60000
> Feb 13 19:27:56 sn311 kernel: RIP: 0010:[<ffffffffc09ddca8>] [<ffffffffc09ddca8>] ceph_encode_filepath.part.26+0x4/0x6 [ceph]
> Feb 13 19:27:56 sn311 kernel: RSP: 0018:ffff9d43aeb63ae8 EFLAGS: 00010293
> Feb 13 19:27:56 sn311 kernel: RAX: ffff9d3f91f46748 RBX: ffff9d43ad4ac800 RCX: ffff9d3f91f4673b
> Feb 13 19:27:56 sn311 kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff9d3f91f46725
> Feb 13 19:27:56 sn311 kernel: RBP: ffff9d43aeb63ae8 R08: 0000000000000000 R09: 0000000000000000
> Feb 13 19:27:56 sn311 kernel: R10: ffff9d34bfc075c0 R11: 000000000000002c R12: ffff9d43adc81c20
> Feb 13 19:27:56 sn311 kernel: R13: 0000000000000000 R14: ffff9d4af08c2000 R15: ffff9d3f91f466c0
> Feb 13 19:27:56 sn311 kernel: FS: 00002aaab4a24d00(0000) GS:ffff9d43afa80000(0000) knlGS:00000000f7bf7700
> Feb 13 19:27:56 sn311 kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> Feb 13 19:27:56 sn311 kernel: CR2: 00002aaaad488800 CR3: 000000106c85c000 CR4: 00000000003407e0
> Feb 13 19:27:56 sn311 kernel: Call Trace:
> Feb 13 19:27:56 sn311 kernel: [<ffffffffc09d6fab>] __prepare_send_request+0x7cb/0x830 [ceph]
> Feb 13 19:27:56 sn311 kernel: [<ffffffffc09d7352>] __do_request+0x342/0x430 [ceph]
> Feb 13 19:27:56 sn311 kernel: [<ffffffffc09d8a8d>] ceph_mdsc_do_request+0x9d/0x280 [ceph]
>
> The log ends here. We don't really have much options with this deployment. It is slim HPC compute nodes and we can't change their configuration easily. I'm afraid we have to wait for an event with more information pushed to logs.
>
> Best regards,
> =================
> Frank Schilder
> AIT Risø Campus
> Bygning 109, rum S14
>
>
>
Thanks, that helps. It crashed while trying to encode a file path in a
request to the MDS. The specific BUG_ON is:
BUG_ON(*p + 1 + sizeof(ino) + sizeof(len) + len > end);
...so that means that it didn't allocate enough buffer space for this
request. This may be a manifestation of commit 1bcb344086f3e (ceph: only
use d_name directly when parent is locked), but it's hard to say with
only a partial stack trace. There have been a number of patches in this
area, and the kernel you're using is quite old.
You may want to open a bug a bugzilla.redhat.com for this. It's not
clear to me that this is critical enough to patch RHEL7, but if it is
then opening a bug there would be the first step.
> ________________________________________
> From: Jeff Layton <jlayton@xxxxxxxxxx>
> Sent: 01 February 2022 13:19:34
> To: Frank Schilder; Ilya Dryomov
> Cc: ceph-users
> Subject: Re: kernel BUG at include/linux/ceph/decode.h:262
>
> Sounds good. We unfortunately can't tell much from the info below. You
> may want to consider turning on kdump. You'll sacrifice a little RAM to
> run it, but it would allow you to collect a core dump if the machine
> crashes.
>
> Also, fwiw, the RHEL7 ceph client is significantly behind where the
> RHEL8 client is. If you're doing any sort of significant work with ceph,
> you may want to migrate if you're able.
>
> -- Jeff
>
> On Tue, 2022-02-01 at 11:49 +0000, Frank Schilder wrote:
> > Hi Ilya,
> >
> > I'm afraid this is all we have from the time of the crash:
> >
> > Jan 25 21:18:42 sn319 kernel: beegfs: enabling unsafe global rkey
> > Jan 25 23:33:47 sn319 kernel: CPU: 12 PID: 123399 Comm: octave-cli Tainted: G OE ------------ T 3.10.0-957.1
> > 2.2.el7.x86_64 #1
> > Jan 25 23:33:47 sn319 kernel: ------------[ cut here ]------------
> > Jan 25 23:33:47 sn319 kernel: Hardware name: Dell Inc. PowerEdge R7425/08V001, BIOS 1.15.0 09/11/2020
> > Jan 25 23:33:47 sn319 kernel: igb i2c_algo_bit ixgbe dca ptp pps_core mdio sd_mod crc_t10dif crct10dif_common
> > Jan 25 23:33:47 sn319 kernel: invalid opcode: 0000 [#1] SMP
> > Jan 25 23:33:47 sn319 kernel: kernel BUG at include/linux/ceph/decode.h:262!
> > Jan 25 23:33:47 sn319 kernel: Modules linked in: squashfs loop overlay(T) 8021q garp mrp stp llc nfsv3 nfs_acl nfs lockd grace fscache beegfs(OE) ceph libceph libcrc32c dns_resolver rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) dcdbas mlx4_core(OE) amd64_edac_mod edac_mce_amd kvm irqbypass crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr sg ahci libahci mgag200 ttm libata ccp i2c_piix4 drm_kms_helper k10temp syscopyarea sysfillrect sysimgblt fb_sys_fops drm ipmi_si ipmi_devintf ipmi_msghandler drm_panel_orientation_quirks acpi_power_meter acpi_cpufreq sunrpc knem(OE) ip_tables mlx5_ib(OE) megaraid_sas ib_uverbs(OE) ib_core(OE) mlx5_core(OE) mlxfw(OE) devlink mlx_compat(OE)
> > Jan 29 15:33:18 sn319.hpc.ait.dtu.dk wwlogger: Running provision script: adhoc-pre
> >
> > This is a stateless deployment and the node crashed hard. syslog was not able to push more lines to the log server, either due to network coming off-line or everything got stopped. Next time we see this problem, we will try to access a crashed node and hope we can pull out more information.
> >
> > Best regards,
> > =================
> > Frank Schilder
> > AIT Risø Campus
> > Bygning 109, rum S14
> >
> > ________________________________________
> > From: Ilya Dryomov <idryomov@xxxxxxxxx>
> > Sent: 31 January 2022 17:16:01
> > To: Frank Schilder
> > Cc: ceph-users; Jeff Layton
> > Subject: Re: kernel BUG at include/linux/ceph/decode.h:262
> >
> > On Mon, Jan 31, 2022 at 5:07 PM Frank Schilder <frans@xxxxxx> wrote:
> > >
> > > Hi all,
> > >
> > > we observed server crashes with these possibly related error messages in the log showing up:
> > >
> > > Jan 26 10:07:53 sn180 kernel: kernel BUG at include/linux/ceph/decode.h:262!
> > > Jan 25 23:33:47 sn319 kernel: kernel BUG at include/linux/ceph/decode.h:262!
> > > Jan 25 16:32:37 sn323 kernel: kernel BUG at include/linux/ceph/decode.h:262!
> > > Jan 25 14:05:07 sn328 kernel: kernel BUG at include/linux/ceph/decode.h:262!
> > > Jan 26 18:47:40 sn369 kernel: kernel BUG at include/linux/ceph/decode.h:262!
> > > Jan 27 21:43:25 sn376 kernel: kernel BUG at include/linux/ceph/decode.h:262!
> > > Jan 28 09:11:00 sn424 kernel: kernel BUG at include/linux/ceph/decode.h:262!
> >
> > The BUG appears to be
> >
> > BUG_ON(*p + 1 + sizeof(ino) + sizeof(len) + len > end);
> >
> > in ceph_encode_filepath().
> >
> > >
> > > The crash repost says:
> > >
> > > Jan 25 23:33:47 sn319 kernel: ------------[ cut here ]------------
> > > Jan 25 23:33:47 sn319 kernel: kernel BUG at include/linux/ceph/decode.h:262!
> > > Jan 25 23:33:47 sn319 kernel: invalid opcode: 0000 [#1] SMP
> > > Jan 25 23:33:47 sn319 kernel: Modules linked in: squashfs loop overlay(T) 8021q garp mrp stp llc nfsv3 nfs_acl nfs lockd grace fscache beegfs(OE) ceph libceph libcrc32c dns_resolver rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib
> > > (OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) dcdbas mlx4_core(OE) amd64_edac_mod edac_mce_amd kvm irqbypass crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ab
> > > lk_helper cryptd pcspkr sg ahci libahci mgag200 ttm libata ccp i2c_piix4 drm_kms_helper k10temp syscopyarea sysfillrect sysimgblt fb_sys_fops drm ipmi_si ipmi_devintf ipmi_msghandler drm_panel_orientation_quirks acpi_power_meter acpi_cp
> > > ufreq sunrpc knem(OE) ip_tables mlx5_ib(OE) megaraid_sas ib_uverbs(OE) ib_core(OE) mlx5_core(OE) mlxfw(OE) devlink mlx_compat(OE)
> > > Jan 25 23:33:47 sn319 kernel: igb i2c_algo_bit ixgbe dca ptp pps_core mdio sd_mod crc_t10dif crct10dif_common
> > > Jan 25 23:33:47 sn319 kernel: CPU: 12 PID: 123399 Comm: octave-cli Tainted: G OE ------------ T 3.10.0-957.12.2.el7.x86_64 #1
> > > Jan 25 23:33:47 sn319 kernel: Hardware name: Dell Inc. PowerEdge R7425/08V001, BIOS 1.15.0 09/11/2020
> >
> > What about a stack trace that should follow here?
> >
> > Thanks,
> >
> > Ilya
>
> --
> Jeff Layton <jlayton@xxxxxxxxxx>
--
Jeff Layton <jlayton@xxxxxxxxxx>
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
