> > > > I can't reproduce this here, fwiw. Ming, something you've seen? > > I just test against the latest for-next/block(-rc4 based), and still can't > reproduce it. There was such RH internal report before, and maybe not > ublk related. > > Changhui, if the issue can be reproduced in your machine, care to share > your machine for me to investigate a bit? > > Thanks, > Ming > I still can reproduce this issue on my machine, and I shared machine to Ming,he can do more investigation for this issue, [ 1244.207092] running generic/006 [ 1246.456896] blk_print_req_error: 77 callbacks suppressed [ 1246.456907] I/O error, dev ublkb1, sector 2395864 op 0x1:(WRITE) flags 0x8800 phys_seg 1 prio class 0 [ 1246.465887] I/O error, dev ublkb1, sector 2551168 op 0x1:(WRITE) flags 0x8800 phys_seg 1 prio class 0 [ 1246.473176] I/O error, dev ublkb1, sector 2395928 op 0x1:(WRITE) flags 0x8800 phys_seg 4 prio class 0 [ 1246.473185] I/O error, dev ublkb1, sector 2395888 op 0x1:(WRITE) flags 0x8800 phys_seg 2 prio class 0 [ 1246.483487] I/O error, dev ublkb1, sector 2550936 op 0x1:(WRITE) flags 0x8800 phys_seg 1 prio class 0 [ 1246.493775] I/O error, dev ublkb1, sector 2395984 op 0x1:(WRITE) flags 0x8800 phys_seg 3 prio class 0 [ 1246.493778] I/O error, dev ublkb1, sector 2395960 op 0x1:(WRITE) flags 0x8800 phys_seg 1 prio class 0 [ 1246.504071] I/O error, dev ublkb1, sector 2221344 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 [ 1246.514367] I/O error, dev ublkb1, sector 2408312 op 0x1:(WRITE) flags 0x8800 phys_seg 2 prio class 0 [ 1246.524665] I/O error, dev ublkb1, sector 2223680 op 0x1:(WRITE) flags 0x8800 phys_seg 2 prio class 0 [ 1258.012089] blk_print_req_error: 214 callbacks suppressed [ 1258.012097] I/O error, dev ublkb1, sector 2191768 op 0x0:(READ) flags 0x0 phys_seg 2 prio class 0 [ 1258.019007] I/O error, dev ublkb1, sector 2123752 op 0x0:(READ) flags 0x0 phys_seg 5 prio class 0 [ 1258.028079] I/O error, dev ublkb1, sector 2195728 op 0x1:(WRITE) flags 0x8800 phys_seg 1 prio class 0 [ 1258.037993] I/O error, dev ublkb1, sector 2127424 op 0x1:(WRITE) flags 0x8800 phys_seg 1 prio class 0 [ 1258.048276] I/O error, dev ublkb1, sector 2191920 op 0x0:(READ) flags 0x0 phys_seg 2 prio class 0 [ 1258.058567] I/O error, dev ublkb1, sector 2127384 op 0x1:(WRITE) flags 0x8800 phys_seg 2 prio class 0 [ 1258.068467] I/O error, dev ublkb1, sector 2200624 op 0x1:(WRITE) flags 0x8800 phys_seg 1 prio class 0 [ 1258.078763] I/O error, dev ublkb1, sector 2127656 op 0x1:(WRITE) flags 0x8800 phys_seg 1 prio class 0 [ 1258.089058] I/O error, dev ublkb1, sector 2195360 op 0x1:(WRITE) flags 0x8800 phys_seg 1 prio class 0 [ 1258.089061] I/O error, dev ublkb1, sector 2191784 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 [ 1269.576153] blk_print_req_error: 237 callbacks suppressed [ 1269.576161] I/O error, dev ublkb1, sector 2061112 op 0x0:(READ) flags 0x0 phys_seg 2 prio class 0 [ 1269.592138] I/O error, dev ublkb1, sector 2177016 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 [ 1269.602051] I/O error, dev ublkb1, sector 2184568 op 0x1:(WRITE) flags 0x8800 phys_seg 1 prio class 0 [ 1269.612347] I/O error, dev ublkb1, sector 2061304 op 0x0:(READ) flags 0x0 phys_seg 2 prio class 0 [ 1269.622258] I/O error, dev ublkb1, sector 2059360 op 0x1:(WRITE) flags 0x8800 phys_seg 2 prio class 0 [ 1269.632556] I/O error, dev ublkb1, sector 2061320 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 [ 1269.642466] I/O error, dev ublkb1, sector 2059080 op 0x1:(WRITE) flags 0x8800 phys_seg 2 prio class 0 [ 1269.652763] I/O error, dev ublkb1, sector 2061128 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 [ 1269.662672] I/O error, dev ublkb1, sector 2059112 op 0x1:(WRITE) flags 0x8800 phys_seg 2 prio class 0 [ 1269.672969] I/O error, dev ublkb1, sector 2061136 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 [ 1299.615464] blk_print_req_error: 74 callbacks suppressed [ 1299.615472] I/O error, dev ublkb1, sector 2023992 op 0x1:(WRITE) flags 0x8800 phys_seg 1 prio class 0 [ 1299.631730] I/O error, dev ublkb1, sector 2387840 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 [ 1299.641647] I/O error, dev ublkb1, sector 2017904 op 0x0:(READ) flags 0x0 phys_seg 2 prio class 0 [ 1299.651551] I/O error, dev ublkb1, sector 2386584 op 0x1:(WRITE) flags 0x8800 phys_seg 2 prio class 0 [ 1299.661852] I/O error, dev ublkb1, sector 2017872 op 0x0:(READ) flags 0x0 phys_seg 3 prio class 0 [ 1299.671769] I/O error, dev ublkb1, sector 2387944 op 0x0:(READ) flags 0x0 phys_seg 3 prio class 0 [ 1299.681689] I/O error, dev ublkb1, sector 2386672 op 0x1:(WRITE) flags 0x8800 phys_seg 2 prio class 0 [ 1299.691991] I/O error, dev ublkb1, sector 2023968 op 0x1:(WRITE) flags 0x8800 phys_seg 3 prio class 0 [ 1299.702289] I/O error, dev ublkb1, sector 2386864 op 0x1:(WRITE) flags 0x8800 phys_seg 1 prio class 0 [ 1299.712586] I/O error, dev ublkb1, sector 2018072 op 0x0:(READ) flags 0x0 phys_seg 2 prio class 0 [ 1310.309570] blk_print_req_error: 118 callbacks suppressed [ 1310.309579] I/O error, dev ublkb1, sector 1939168 op 0x0:(READ) flags 0x0 phys_seg 2 prio class 0 [ 1310.310554] I/O error, dev ublkb1, sector 2575992 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 [ 1310.315649] I/O error, dev ublkb1, sector 1939704 op 0x1:(WRITE) flags 0x8800 phys_seg 2 prio class 0 [ 1310.325550] I/O error, dev ublkb1, sector 2582152 op 0x1:(WRITE) flags 0x8800 phys_seg 1 prio class 0 [ 1310.335456] I/O error, dev ublkb1, sector 1939776 op 0x1:(WRITE) flags 0x8800 phys_seg 1 prio class 0 [ 1310.345751] I/O error, dev ublkb1, sector 2576008 op 0x0:(READ) flags 0x0 phys_seg 4 prio class 0 [ 1310.356048] I/O error, dev ublkb1, sector 1939456 op 0x1:(WRITE) flags 0x8800 phys_seg 2 prio class 0 [ 1310.356051] I/O error, dev ublkb1, sector 1939424 op 0x1:(WRITE) flags 0x8800 phys_seg 1 prio class 0 [ 1310.356054] I/O error, dev ublkb1, sector 1939272 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 [ 1310.356057] I/O error, dev ublkb1, sector 1939440 op 0x1:(WRITE) flags 0x8800 phys_seg 1 prio class 0 [ 1321.569744] ------------[ cut here ]------------ [ 1321.574931] WARNING: CPU: 15 PID: 679 at io_uring/io_uring.c:2835 io_ring_exit_work+0x2b6/0x2e0 [ 1321.584673] Modules linked in: ext4 mbcache jbd2 rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace netfs rfkill sunrpc dm_multipath intel_rapl_msr intel_rapl_common intel_uncore_frequency intel_uncore_frequency_common i10nm_edac nfit libnvdimm x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm ipmi_ssif rapl intel_cstate mgag200 i2c_algo_bit dax_hmem iTCO_wdt cxl_acpi drm_shmem_helper iTCO_vendor_support cxl_core mei_me drm_kms_helper ioatdma i2c_i801 isst_if_mmio intel_uncore isst_if_mbox_pci mei einj isst_if_common pcspkr intel_pch_thermal i2c_smbus intel_vsec ipmi_si dca ipmi_devintf ipmi_msghandler acpi_power_meter acpi_pad drm fuse xfs libcrc32c sd_mod t10_pi sg crct10dif_pclmul crc32_pclmul crc32c_intel ahci libahci libata tg3 ghash_clmulni_intel wmi dm_mirror dm_region_hash dm_log dm_mod [ 1321.665566] CPU: 15 PID: 679 Comm: kworker/u96:9 Not tainted 6.9.0-rc3+ #1 [ 1321.673251] Hardware name: Lenovo ThinkSystem SR650 V2/7Z73CTO1WW, BIOS AFE120G-1.40 09/20/2022 [ 1321.682970] Workqueue: iou_exit io_ring_exit_work [ 1321.688235] RIP: 0010:io_ring_exit_work+0x2b6/0x2e0 [ 1321.693689] Code: 89 e7 e8 6d de ff ff 48 8b 44 24 58 65 48 2b 04 25 28 00 00 00 75 2e 48 8d 65 d8 5b 41 5c 41 5d 41 5e 41 5f 5d c3 cc cc cc cc <0f> 0b 41 be 60 ea 00 00 e9 45 fe ff ff 0f 0b e9 1d ff ff ff e8 c1 [ 1321.714654] RSP: 0018:ff37e1e101257dd0 EFLAGS: 00010293 [ 1321.720494] RAX: 00000001000f97d6 RBX: ff3771afdd8bec48 RCX: 0000000000000000 [ 1321.728468] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ff3771afdd8be840 [ 1321.736437] RBP: ff37e1e101257e60 R08: 0000000000000000 R09: ffffffffa6c54ca0 [ 1321.744406] R10: 0000000000000000 R11: 0000000000000000 R12: ff3771afdd8be800 [ 1321.752376] R13: ff3771afdd8be840 R14: 0000000000000032 R15: 0000000000000000 [ 1321.760354] FS: 0000000000000000(0000) GS:ff3771b2efb80000(0000) knlGS:0000000000000000 [ 1321.769396] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1321.775808] CR2: 00007f5334002088 CR3: 00000003ba220001 CR4: 0000000000771ef0 [ 1321.783783] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 1321.791757] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 1321.799733] PKRU: 55555554 [ 1321.802760] Call Trace: [ 1321.805498] <TASK> [ 1321.807847] ? __warn+0x7f/0x130 [ 1321.811460] ? io_ring_exit_work+0x2b6/0x2e0 [ 1321.816238] ? report_bug+0x18a/0x1a0 [ 1321.820340] ? handle_bug+0x3c/0x70 [ 1321.824247] ? exc_invalid_op+0x14/0x70 [ 1321.828537] ? asm_exc_invalid_op+0x16/0x20 [ 1321.833222] ? io_ring_exit_work+0x2b6/0x2e0 [ 1321.837998] ? finish_task_switch.isra.0+0x8e/0x2a0 [ 1321.843457] process_one_work+0x193/0x3d0 [ 1321.847945] worker_thread+0x2fc/0x410 [ 1321.852140] ? __pfx_worker_thread+0x10/0x10 [ 1321.856907] kthread+0xdc/0x110 [ 1321.860421] ? __pfx_kthread+0x10/0x10 [ 1321.864617] ret_from_fork+0x2d/0x50 [ 1321.868622] ? __pfx_kthread+0x10/0x10 [ 1321.872812] ret_from_fork_asm+0x1a/0x30 [ 1321.877208] </TASK> [ 1321.879655] ---[ end trace 0000000000000000 ]---