I reproduced this issue even when system boot with the latest linux-block/for-next, will try to bisect it later. 43f3ae1898c9 (HEAD -> for-next, origin/for-next) Merge branch 'for-6.2/writeback' into for-next d6798bc243fa writeback: Add asserts for adding freed inode to lists [ 24.183829] list_add corruption. prev->next should be next (ffff9a1d9f337f68), but was ffff9a1a02119e70. (prev=ffff9a1a02119e70). [ 24.195478] ------------[ cut here ]------------ [ 24.200088] kernel BUG at lib/list_debug.c:30! [ 24.204532] invalid opcode: 0000 [#1] PREEMPT SMP NOPTI [ 24.209751] CPU: 4 PID: 167 Comm: kworker/4:1 Not tainted 6.1.0-rc6+ #1 [ 24.216365] Hardware name: Dell Inc. PowerEdge R6515/07PXPY, BIOS 2.8.5 08/18/2022 [ 24.223930] Workqueue: cgwb_release cgwb_release_workfn [ 24.229157] RIP: 0010:__list_add_valid.cold+0x3a/0x5b [ 24.234208] Code: f2 4c 89 c1 48 89 fe 48 c7 c7 20 23 65 a8 e8 d2 a2 fe ff 0f 0b 48 89 d1 4c 89 c6 4c 89 ca 48 c7 c7 c8 22 65 a8 e8 bb a2 fe ff <0f> 0b 4c 89 c1 48 c7 c7 70 22 65 a8 e8 aa a2 fe ff 0f 0b 48 c7 c7 [ 24.252953] RSP: 0018:ffffb035407e7da8 EFLAGS: 00010046 [ 24.258172] RAX: 0000000000000075 RBX: ffff9a1a02119e68 RCX: 0000000000000000 [ 24.265303] RDX: 0000000000000000 RSI: ffff9a1d9f31f840 RDI: ffff9a1d9f31f840 [ 24.272428] RBP: ffff9a1d9f337f00 R08: 0000000000000000 R09: 00000000ffff7fff [ 24.279560] R10: ffffb035407e7c50 R11: ffffffffa8be75e8 R12: ffff9a1d9f337f68 [ 24.286683] R13: ffff9a1a02119e70 R14: ffff9a1a02119e70 R15: ffff9a1d9f330340 [ 24.293808] FS: 0000000000000000(0000) GS:ffff9a1d9f300000(0000) knlGS:0000000000000000 [ 24.301894] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 24.307641] CR2: 000055b5f1f28050 CR3: 0000000104e38000 CR4: 0000000000350ee0 [ 24.314772] Call Trace: [ 24.314774] <TASK> [ 24.314774] insert_work+0x46/0xc0 [ 24.314780] __queue_work+0x1d5/0x380 [ 24.326376] queue_work_on+0x24/0x30 [ 24.329955] blkcg_unpin_online+0x1b5/0x1c0 [ 24.334143] cgwb_release_workfn+0x6a/0x200 [ 24.338327] process_one_work+0x1e5/0x3b0 [ 24.342342] ? rescuer_thread+0x390/0x390 [ 24.346352] worker_thread+0x50/0x3a0 [ 24.350019] ? rescuer_thread+0x390/0x390 [ 24.354030] kthread+0xd9/0x100 [ 24.357177] ? kthread_complete_and_exit+0x20/0x20 [ 24.361970] ret_from_fork+0x22/0x30 [ 24.365550] </TASK> [ 24.367742] Modules linked in: sunrpc intel_rapl_msr intel_rapl_common amd64_edac edac_mce_amd ipmi_ssif kvm_amd kvm mgag200 ledtrig_audio rfkill video i2c_algo_bit drm_shmem_helper dcdbas drm_kms_helper irqbypass dell_smbios rapl dell_wmi_descriptor wmi_bmof pcspkr syscopyarea acpi_ipmi sysfillrect sysimgblt fb_sys_fops ipmi_si ipmi_devintf ptdma i2c_piix4 k10temp ipmi_msghandler acpi_power_meter vfat fat acpi_cpufreq drm fuse xfs libcrc32c sd_mod sg ahci crct10dif_pclmul crc32_pclmul libahci crc32c_intel ghash_clmulni_intel mpt3sas nvme tg3 libata nvme_core ccp raid_class nvme_common t10_pi sp5100_tco scsi_transport_sas wmi dm_mirror dm_region_hash dm_log dm_mod [ 24.426475] ---[ end trace 0000000000000000 ]--- [ 24.505278] RIP: 0010:__list_add_valid.cold+0x3a/0x5b [ 24.510331] Code: f2 4c 89 c1 48 89 fe 48 c7 c7 20 23 65 a8 e8 d2 a2 fe ff 0f 0b 48 89 d1 4c 89 c6 4c 89 ca 48 c7 c7 c8 22 65 a8 e8 bb a2 fe ff <0f> 0b 4c 89 c1 48 c7 c7 70 22 65 a8 e8 aa a2 fe ff 0f 0b 48 c7 c7 [ 24.510332] RSP: 0018:ffffb035407e7da8 EFLAGS: 00010046 [ 24.510333] RAX: 0000000000000075 RBX: ffff9a1a02119e68 RCX: 0000000000000000 [ 24.510334] RDX: 0000000000000000 RSI: ffff9a1d9f31f840 RDI: ffff9a1d9f31f840 [ 24.510335] RBP: ffff9a1d9f337f00 R08: 0000000000000000 R09: 00000000ffff7fff [ 24.510337] R10: ffffb035407e7c50 R11: ffffffffa8be75e8 R12: ffff9a1d9f337f68 [ 24.562805] R13: ffff9a1a02119e70 R14: ffff9a1a02119e70 R15: ffff9a1d9f330340 [ 24.569929] FS: 0000000000000000(0000) GS:ffff9a1d9f300000(0000) knlGS:0000000000000000 [ 24.578017] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 24.578018] CR2: 000055b5f1f28050 CR3: 0000000104e38000 CR4: 0000000000350ee0 [ 24.578019] Kernel panic - not syncing: Fatal exception [ 24.578653] Kernel Offset: 0x26200000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff) [ 24.682013] ---[ end Kernel panic - not syncing: Fatal exception ]--- [ 24.339396] r[-- MARK -- Fri Nov 25 06:25:00 2022] On Thu, Nov 24, 2022 at 11:00 PM Bruno Goncalves <bgoncalv@xxxxxxxxxx> wrote: > > On Wed, 23 Nov 2022 at 14:46, Jens Axboe <axboe@xxxxxxxxx> wrote: > > > > On 11/23/22 1:48 AM, Bruno Goncalves wrote: > > > Hello, > > > > > > We recently started to hit the following panic when testing the block > > > tree (for-next branch). > > > > > > [ 5076.172749] list_add corruption. prev->next should be next > > > (ffff91cd6f7fa568), but was ffff91c991ca6670. (prev=ffff91c991ca6670). > > > [ 5076.173863] ------------[ cut here ]------------ > > > [ 5076.174853] kernel BUG at lib/list_debug.c:30! > > > [ 5076.175523] invalid opcode: 0000 [#1] PREEMPT SMP PTI > > > [ 5076.175853] CPU: 15 PID: 16415 Comm: kworker/15:13 Tainted: G > > > I 6.1.0-rc6 #1 > > > [ 5076.176799] Hardware name: HP ProLiant DL360p Gen8, BIOS P71 05/24/2019 > > > [ 5076.177198] Workqueue: cgwb_release cgwb_release_workfn > > > [ 5076.177497] RIP: 0010:__list_add_valid.cold+0x3a/0x5b > > > [ 5076.177788] Code: f2 48 89 c1 48 89 fe 48 c7 c7 48 d8 76 ad e8 5a > > > 8f fd ff 0f 0b 48 89 d1 48 89 c6 4c 89 c2 48 c7 c7 f0 d7 76 ad e8 43 > > > 8f fd ff <0f> 0b 48 89 c1 48 c7 c7 98 d7 76 ad e8 32 8f fd ff 0f 0b 48 > > > c7 c7 > > > [ 5076.179173] RSP: 0018:ffffa1c98a6afdb0 EFLAGS: 00010082 > > > [ 5076.179472] RAX: 0000000000000075 RBX: ffff91c991ca6668 RCX: 0000000000000000 > > > [ 5076.180241] RDX: 0000000000000002 RSI: ffffffffad752ad3 RDI: 00000000ffffffff > > > [ 5076.181069] RBP: ffff91cd6f7fa500 R08: 0000000000000000 R09: ffffa1c98a6afc60 > > > [ 5076.182209] R10: 0000000000000003 R11: ffff91cd7ff42fe8 R12: ffff91cd6f7fa568 > > > [ 5076.183002] R13: ffff91c991ca6670 R14: ffff91c991ca6670 R15: ffff91cd6f7f1440 > > > [ 5076.183902] FS: 0000000000000000(0000) GS:ffff91cd6f7c0000(0000) > > > knlGS:0000000000000000 > > > [ 5076.184377] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > > > [ 5076.185084] CR2: 0000560ff67e11b8 CR3: 000000020d010005 CR4: 00000000000606e0 > > > [ 5076.185945] Call Trace: > > > [ 5076.186110] <TASK> > > > [ 5076.186916] insert_work+0x46/0xc0 > > > [ 5076.187533] __queue_work+0x1d4/0x460 > > > [ 5076.187788] queue_work_on+0x37/0x40 > > > [ 5076.187993] blkcg_unpin_online+0x1ad/0x1b0 > > > [ 5076.188244] cgwb_release_workfn+0x6a/0x200 > > > [ 5076.188464] process_one_work+0x1c7/0x380 > > > [ 5076.188675] worker_thread+0x4d/0x380 > > > [ 5076.188881] ? rescuer_thread+0x380/0x380 > > > [ 5076.189089] kthread+0xe9/0x110 > > > [ 5076.189716] ? kthread_complete_and_exit+0x20/0x20 > > > [ 5076.190407] ret_from_fork+0x22/0x30 > > > [ 5076.190677] </TASK> > > > [ 5076.190816] Modules linked in: nvme nvme_core nvme_common loop tls > > > rfkill intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal > > > intel_powerclamp coretemp sunrpc kvm_intel kvm iTCO_wdt iapl > > > intel_cstate intel_uncore pcspkr lpc_ich ipmi_ssif hpilo tg3 acpi_ipmi > > > ioatdma ipmi_si ipmi_devintf dca ipmi_msghandler acpi_power_meter fuse > > > zram xfs crct10dif_pclmul crc32_pclmul crc32c_intel polyval_clmulni > > > polyval_generic ghash_clmulni_intel sha512_ssse3 serio_raw hpsa > > > mgag200 scsi_transport_sas [last unloaded: scsi_debug] > > > [ 5076.293315] ---[ end trace 0000000000000000 ]--- > > > [ 5076.295226] RIP: 0010:__list_add_valid.cold+0x3a/0x5b > > > [ 5076.295587] Code: f2 48 89 c1 48 89 fe 48 c7 c7 48 d8 76 ad e8 5a > > > 8f fd ff 0f 0b 48 89 d1 48 89 c6 4c 89 c2 48 c7 c7 f0 d7 76 ad e8 43 > > > 8f fd ff <0f> 0b 48 89 c1 48 c7 c7 98 d7 76 ad e8 32 8f fd ff 0f 0b 48 > > > c7 c7 > > > [ 5076.296921] RSP: 0018:ffffa1c98a6afdb0 EFLAGS: 00010082 > > > [ 5076.297239] RAX: 0000000000000075 RBX: ffff91c991ca6668 RCX: 0000000000000000 > > > [ 5076.297983] RDX: 0000000000000002 RSI: ffffffffad752ad3 RDI: 00000000ffffffff > > > [ 5076.298768] RBP: ffff91cd6f7fa500 R08: 0000000000000000 R09: ffffa1c98a6afc60 > > > [ 5076.299525] R10: 0000S: 0000000000000000(0000) > > > GS:ffff91cd6f7c0000(0000) knlGS:0000000000000000 > > > [ 5076.700351] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > > > [ 5076.701046] CR2: 0000560ff67e11b8 CR3: 000000020d010005 CR4: 00000000000606e0 > > > [ 5076ernel panic - not syncing: Fatal exception > > > [ 5077.924713] Shutting down cpus with NMI > > > [ 5077.924986] Kernel Offset: 0x2b000000 from 0xffffffff81000000 > > > (relocation range: 0xffffffff80000000-0xffffffffbfffffff) > > > [ 5077.927946] ---[ end Kernel panic - not syncing: Fatal exception ]--- > > > > > > It seems to happen often during different tests. > > > > > > full console.log: > > > https://s3.us-east-1.amazonaws.com/arr-cki-prod-datawarehouse-public/datawarehouse-public/2022/11/21/redhat:700955106/build_x86_64_redhat:700955106_x86_64/tests/1/results_0001/console.log/console.log > > > > > > kernel tarball: > > > https://s3.amazonaws.com/arr-cki-prod-trusted-artifacts/trusted-artifacts/700955106/publish%20x86_64/3356091217/artifacts/kernel-block-redhat_700955106_x86_64.tar.gz > > > > > > kernel config: https://s3.amazonaws.com/arr-cki-prod-trusted-artifacts/trusted-artifacts/700955106/build%20x86_64/3356091207/artifacts/kernel-block-redhat_700955106_x86_64.config > > > > > > test logs: https://datawarehouse.cki-project.org/kcidb/tests/6061677 > > > > > > We didn't bisect, but the first commit we hit the problem was > > > "f65d92c600fe6eecdbd6e7fab7893c9c094dfcbf > > > (io_uring-6.1-2022-11-18-2180-gf65d92c600fe)" and the last one where > > > we didn't hit the problem was > > > "40fa774af7fd04d06014ac74947c351649b6f64f > > > (io_uring-6.1-2022-11-11-1843-g40fa774af7fd)" > > > > > > test logs: https://datawarehouse.cki-project.org/kcidb/tests/6061677 > > > cki issue tracker: https://datawarehouse.cki-project.org/issue/1732 > > > > Please just try and clone for-6.2/block from the block tree and bisect > > it? > > > > Hi, > I've tried with commit 93c68cc46a070775cc6675e3543dd909eb9f6c9e (drbd: > use consistent license), but I was not able to hit the panic with it. > > > Bruno > > > -- > > Jens Axboe > > > > > -- Best Regards, Yi Zhang
