https://bugzilla.kernel.org/show_bug.cgi?id=219504 Bug ID: 219504 Summary: XFS crashes with kernel Version > 6.1.91. Perhaps Changes in kernel 6.1.92 for XFS/iomap causing the problems? Product: File System Version: 2.5 Hardware: AMD OS: Linux Status: NEW Severity: normal Priority: P3 Component: XFS Assignee: filesystem_xfs@xxxxxxxxxxxxxxxxxxxxxx Reporter: speedcracker@xxxxxxxxxxx Regression: No Hi there, the reason for this message is, with changes in Kernel 6.1.92, I've got some trouble with high I/O by using the XFS filesystem. I think the reason for the kernel-backtrace, which is mentioned later, is one of the following commits (in Kernel 6.1.92): e811fec51c66a0056459daa1ac834aea7d8d98f5, ea67e73129fceffd40b9193da93544c34d81b9c2, 54a37e5d07478358dcbf6e73b6c7e40e50a6f375, 580f40b4c956f38e83f66ebed4d81bbe4a7d82fb, 12339ec6fe4d41e69a81a13ca5e1c443fbe5bcba... and so on. By using kernel 6.1.91 or lower version, no problems occur. By using kernel 6.1.92 or up (latest tests I've done with 6.1.105 - then I gave up) and high I/O workload, the problem occur. I'm using XFS as a underlaying file-system with a 32bit kernel for exporting LUN's by using SCST (https://github.com/SCST-project/scst) through fileio (not blockio). I'm using the latest SCST git release. One of the scst developers also sees the problem here as being more in the direction of the Linux kernel, high I/O and XFS file system. Here is the kernel-backtrace: [Tue Aug 20 00:02:00 2024] ------------[ cut here ]------------ [Tue Aug 20 00:02:00 2024] WARNING: CPU: 5 PID: 2048 at fs/iomap/buffered-io.c:980 iomap_file_buffered_write_punch_delalloc+0x3b0/0x440 [Tue Aug 20 00:02:00 2024] Modules linked in: iscsi_scst(O) scst_vdisk(O) scst(O) dlm quota_v2 quota_tree autofs4 tcp_bbr sch_fq udf crc_itu_t input_leds led_class ses enclosure hid_generic wmi_bmof edac_mce_amd crc32_pclmul usbhid aesni_intel crypto_simd uas hid usb_storage rapl r8169 bnx2 i2c_piix4 mpt3sas ccp i2c_core sha1_generic k10temp pcspkr video wmi backlight [Tue Aug 20 00:02:00 2024] CPU: 5 PID: 2048 Comm: disk042_0 Tainted: G S O 6.1.106_LFS_FILE01 #1 [Tue Aug 20 00:02:00 2024] Hardware name: To Be Filled By O.E.M. X370 Pro4/X370 Pro4, BIOS P10.08 01/22/2024 [Tue Aug 20 00:02:00 2024] EIP: iomap_file_buffered_write_punch_delalloc+0x3b0/0x440 [Tue Aug 20 00:02:00 2024] Code: 26 00 89 c6 89 d8 e8 4f e3 ed ff f0 ff 4b 1c 75 9c 89 d8 e8 22 0a ef ff 66 90 eb 91 8d b6 00 00 00 00 0f 0b e9 f5 fd ff ff 90 <0f> 0b e9 df fd ff ff 90 0f 0b 8b 45 ec 8b 55 f0 89 f9 39 c6 19 d1 [Tue Aug 20 00:02:00 2024] EAX: a733e000 EBX: 0007f000 ECX: fffffef2 EDX: 0000010e [Tue Aug 20 00:02:00 2024] ESI: a733f000 EDI: 00000000 EBP: 86df3bd0 ESP: 86df3b80 [Tue Aug 20 00:02:00 2024] DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068 EFLAGS: 00010293 [Tue Aug 20 00:02:00 2024] CR0: 80050033 CR2: 37f87000 CR3: 0185a6e0 CR4: 00350ef0 [Tue Aug 20 00:02:00 2024] Call Trace: [Tue Aug 20 00:02:00 2024] ? show_regs.cold+0x16/0x1b [Tue Aug 20 00:02:00 2024] ? iomap_file_buffered_write_punch_delalloc+0x3b0/0x440 [Tue Aug 20 00:02:00 2024] ? __warn+0x87/0xe0 [Tue Aug 20 00:02:00 2024] ? iomap_file_buffered_write_punch_delalloc+0x3b0/0x440 [Tue Aug 20 00:02:00 2024] ? iomap_file_buffered_write_punch_delalloc+0x3b0/0x440 [Tue Aug 20 00:02:00 2024] ? report_bug+0xe5/0x170 [Tue Aug 20 00:02:00 2024] ? exc_overflow+0x60/0x60 [Tue Aug 20 00:02:00 2024] ? handle_bug+0x2a/0x50 [Tue Aug 20 00:02:00 2024] ? exc_invalid_op+0x1e/0x70 [Tue Aug 20 00:02:00 2024] ? handle_exception+0x101/0x101 [Tue Aug 20 00:02:00 2024] ? exc_overflow+0x60/0x60 [Tue Aug 20 00:02:00 2024] ? iomap_file_buffered_write_punch_delalloc+0x3b0/0x440 [Tue Aug 20 00:02:00 2024] ? exc_overflow+0x60/0x60 [Tue Aug 20 00:02:00 2024] ? iomap_file_buffered_write_punch_delalloc+0x3b0/0x440 [Tue Aug 20 00:02:00 2024] ? xfs_dax_write_iomap_end+0xa0/0xa0 [Tue Aug 20 00:02:00 2024] xfs_buffered_write_iomap_end+0x52/0xc0 [Tue Aug 20 00:02:00 2024] ? xfs_buffered_write_iomap_end+0xc0/0xc0 [Tue Aug 20 00:02:00 2024] iomap_iter+0xce/0x4b0 [Tue Aug 20 00:02:00 2024] ? xfs_dax_write_iomap_end+0xa0/0xa0 [Tue Aug 20 00:02:00 2024] iomap_file_buffered_write+0xa9/0x420 [Tue Aug 20 00:02:00 2024] xfs_file_buffered_write+0x9d/0x2e0 [Tue Aug 20 00:02:00 2024] xfs_file_write_iter+0xc9/0x100 [Tue Aug 20 00:02:00 2024] fileio_exec_async+0x25e/0x3a0 [scst_vdisk] [Tue Aug 20 00:02:00 2024] fileio_exec_write+0x2ce/0x400 [scst_vdisk] [Tue Aug 20 00:02:00 2024] ? __switch_to_asm+0xdd/0xf0 [Tue Aug 20 00:02:00 2024] ? __switch_to_asm+0xd7/0xf0 [Tue Aug 20 00:02:00 2024] ? __switch_to_asm+0xd1/0xf0 [Tue Aug 20 00:02:00 2024] ? __switch_to_asm+0xcb/0xf0 [Tue Aug 20 00:02:00 2024] vdev_do_job+0x36/0xe0 [scst_vdisk] [Tue Aug 20 00:02:00 2024] ? __switch_to_asm+0x8f/0xf0 [Tue Aug 20 00:02:00 2024] fileio_exec+0x1f/0x30 [scst_vdisk] [Tue Aug 20 00:02:00 2024] scst_do_real_exec+0x51/0x130 [scst] [Tue Aug 20 00:02:00 2024] scst_exec_check_blocking+0xa8/0x220 [scst] [Tue Aug 20 00:02:00 2024] scst_process_active_cmd+0x200/0x18f0 [scst] [Tue Aug 20 00:02:00 2024] scst_cmd_thread+0x15c/0x500 [scst] [Tue Aug 20 00:02:00 2024] ? prepare_to_wait_event+0x160/0x160 [Tue Aug 20 00:02:00 2024] kthread+0xd2/0x100 [Tue Aug 20 00:02:00 2024] ? scst_cmd_done_local+0x90/0x90 [scst] [Tue Aug 20 00:02:00 2024] ? kthread_complete_and_exit+0x20/0x20 [Tue Aug 20 00:02:00 2024] ret_from_fork+0x1c/0x28 [Tue Aug 20 00:02:00 2024] ---[ end trace 0000000000000000 ]--- [Tue Aug 20 00:02:00 2024] ------------[ cut here ]------------ [Tue Aug 20 00:02:00 2024] WARNING: CPU: 5 PID: 2048 at fs/iomap/buffered-io.c:993 iomap_file_buffered_write_punch_delalloc+0x2f0/0x440 [Tue Aug 20 00:02:00 2024] Modules linked in: iscsi_scst(O) scst_vdisk(O) scst(O) dlm quota_v2 quota_tree autofs4 tcp_bbr sch_fq udf crc_itu_t input_leds led_class ses enclosure hid_generic wmi_bmof edac_mce_amd crc32_pclmul usbhid aesni_intel crypto_simd uas hid usb_storage rapl r8169 bnx2 i2c_piix4 mpt3sas ccp i2c_core sha1_generic k10temp pcspkr video wmi backlight [Tue Aug 20 00:02:00 2024] CPU: 5 PID: 2048 Comm: disk042_0 Tainted: G S W O 6.1.106_LFS_FILE01 #1 [Tue Aug 20 00:02:00 2024] Hardware name: To Be Filled By O.E.M. X370 Pro4/X370 Pro4, BIOS P10.08 01/22/2024 [Tue Aug 20 00:02:00 2024] EIP: iomap_file_buffered_write_punch_delalloc+0x2f0/0x440 [Tue Aug 20 00:02:00 2024] Code: 8b 7d f0 01 c2 c1 e2 0c c7 45 d8 00 00 00 00 89 55 d4 39 d6 89 f9 83 d9 00 0f 8d 1e ff ff ff 89 75 d4 89 7d d8 e9 13 ff ff ff <0f> 0b 39 45 dc 8b 4d e4 19 d1 0f 8c b8 00 00 00 8b 45 ec 8b 7d dc [Tue Aug 20 00:02:00 2024] EAX: a733f000 EBX: 00000000 ECX: a733f000 EDX: 00000000 [Tue Aug 20 00:02:00 2024] ESI: a733f000 EDI: 00000000 EBP: 86df3bd0 ESP: 86df3b80 [Tue Aug 20 00:02:00 2024] DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068 EFLAGS: 00010246 [Tue Aug 20 00:02:00 2024] CR0: 80050033 CR2: 37f87000 CR3: 0185a6e0 CR4: 00350ef0 [Tue Aug 20 00:02:00 2024] Call Trace: [Tue Aug 20 00:02:00 2024] ? show_regs.cold+0x16/0x1b [Tue Aug 20 00:02:00 2024] ? iomap_file_buffered_write_punch_delalloc+0x2f0/0x440 [Tue Aug 20 00:02:00 2024] ? __warn+0x87/0xe0 [Tue Aug 20 00:02:00 2024] ? iomap_file_buffered_write_punch_delalloc+0x2f0/0x440 [Tue Aug 20 00:02:00 2024] ? iomap_file_buffered_write_punch_delalloc+0x2f0/0x440 [Tue Aug 20 00:02:00 2024] ? report_bug+0xe5/0x170 [Tue Aug 20 00:02:00 2024] ? exc_overflow+0x60/0x60 [Tue Aug 20 00:02:00 2024] ? handle_bug+0x2a/0x50 [Tue Aug 20 00:02:00 2024] ? exc_invalid_op+0x1e/0x70 [Tue Aug 20 00:02:00 2024] ? handle_exception+0x101/0x101 [Tue Aug 20 00:02:00 2024] ? exc_overflow+0x60/0x60 [Tue Aug 20 00:02:00 2024] ? iomap_file_buffered_write_punch_delalloc+0x2f0/0x440 [Tue Aug 20 00:02:00 2024] ? exc_overflow+0x60/0x60 [Tue Aug 20 00:02:00 2024] ? iomap_file_buffered_write_punch_delalloc+0x2f0/0x440 [Tue Aug 20 00:02:00 2024] ? xfs_dax_write_iomap_end+0xa0/0xa0 [Tue Aug 20 00:02:00 2024] xfs_buffered_write_iomap_end+0x52/0xc0 [Tue Aug 20 00:02:00 2024] ? xfs_buffered_write_iomap_end+0xc0/0xc0 [Tue Aug 20 00:02:00 2024] iomap_iter+0xce/0x4b0 [Tue Aug 20 00:02:00 2024] ? xfs_dax_write_iomap_end+0xa0/0xa0 [Tue Aug 20 00:02:00 2024] iomap_file_buffered_write+0xa9/0x420 [Tue Aug 20 00:02:00 2024] xfs_file_buffered_write+0x9d/0x2e0 [Tue Aug 20 00:02:00 2024] xfs_file_write_iter+0xc9/0x100 [Tue Aug 20 00:02:00 2024] fileio_exec_async+0x25e/0x3a0 [scst_vdisk] [Tue Aug 20 00:02:00 2024] fileio_exec_write+0x2ce/0x400 [scst_vdisk] [Tue Aug 20 00:02:00 2024] ? __switch_to_asm+0xdd/0xf0 [Tue Aug 20 00:02:00 2024] ? __switch_to_asm+0xd7/0xf0 [Tue Aug 20 00:02:00 2024] ? __switch_to_asm+0xd1/0xf0 [Tue Aug 20 00:02:00 2024] ? __switch_to_asm+0xcb/0xf0 [Tue Aug 20 00:02:00 2024] vdev_do_job+0x36/0xe0 [scst_vdisk] [Tue Aug 20 00:02:00 2024] ? __switch_to_asm+0x8f/0xf0 [Tue Aug 20 00:02:00 2024] fileio_exec+0x1f/0x30 [scst_vdisk] [Tue Aug 20 00:02:00 2024] scst_do_real_exec+0x51/0x130 [scst] [Tue Aug 20 00:02:00 2024] scst_exec_check_blocking+0xa8/0x220 [scst] [Tue Aug 20 00:02:00 2024] scst_process_active_cmd+0x200/0x18f0 [scst] [Tue Aug 20 00:02:00 2024] scst_cmd_thread+0x15c/0x500 [scst] [Tue Aug 20 00:02:00 2024] ? prepare_to_wait_event+0x160/0x160 [Tue Aug 20 00:02:00 2024] kthread+0xd2/0x100 [Tue Aug 20 00:02:00 2024] ? scst_cmd_done_local+0x90/0x90 [scst] [Tue Aug 20 00:02:00 2024] ? kthread_complete_and_exit+0x20/0x20 [Tue Aug 20 00:02:00 2024] ret_from_fork+0x1c/0x28 [Tue Aug 20 00:02:00 2024] ---[ end trace 0000000000000000 ]--- Now my question is: Where we have to search for the problem? At the kernel-maintaining staff or at the maintainer @SCST? Thanks in advance and for investigation, Mike -- You may reply to this email to add a comment. You are receiving this mail because: You are watching the assignee of the bug.
