[Adding Joanne, Willy and linux-mm].
On Thu, 6 Feb 2025 at 11:54, Christian Heusel <christian@xxxxxxxxx> wrote:
>
> Hello everyone,
>
> we have recently received [a report][0] on the Arch Linux Gitlab about
> multiple users having system crashes when using Flatpak programs and
> related FUSE errors in their dmesg logs.
>
> We have subsequently bisected the issue within the mainline kernel tree
> to the following commit:
>
> 3eab9d7bc2f4 ("fuse: convert readahead to use folios")
>
> The error is still present in the latest mainline release 6.14-rc1 and
> sadly testing a revert is not trivially possible due to conflicts.
>
> I have attached a dmesg output from a boot where the failure occurs and
> I'm happy to test any debug patches with the help of the other reporters
> on our GitLab.
>
> We also noticed that there already was [a discussion][1] about a related
> commit but the fix for the issue back then 7a4f54187373 ("fuse: fix
> direct io folio offset and length calculation") was already included in
> the revisions we have tested.
>
> Cheers,
> Christian
>
> [0]: https://gitlab.archlinux.org/archlinux/packaging/packages/linux/-/issues/110
> [1]: https://lore.kernel.org/all/p3iss6hssbvtdutnwmuddvdadubrhfkdoosgmbewvo674f7f3y@cwnwffjqltzw/
Thanks for the report.
Seems like page allocation gets an inconsistent page (mapcount != -1)
in the report below.
Any ideas why this could be happening?
Thanks,
Miklos
> Feb 06 08:54:47 archvm kernel: BUG: Bad page state in process rnote pfn:67587
> Feb 06 08:54:47 archvm kernel: page: refcount:-1 mapcount:0 mapping:0000000000000000 index:0x0 pfn:0x67587
> Feb 06 08:54:47 archvm kernel: flags: 0xfffffc8000020(lru|node=0|zone=1|lastcpupid=0x1fffff)
> Feb 06 08:54:47 archvm kernel: raw: 000fffffc8000020 dead000000000100 dead000000000122 0000000000000000
> Feb 06 08:54:47 archvm kernel: raw: 0000000000000000 0000000000000000 ffffffffffffffff 0000000000000000
> Feb 06 08:54:47 archvm kernel: page dumped because: PAGE_FLAGS_CHECK_AT_PREP flag(s) set
> Feb 06 08:54:47 archvm kernel: Modules linked in: snd_seq_dummy snd_hrtimer snd_seq snd_seq_device rfkill vfat fat intel_rapl_msr intel_rapl_common kvm_amd ccp snd_hda_codec_hdmi snd_hda_codec_generic snd_hda_intel snd_intel_dspcfg kvm snd_intel_sdw_acpi snd_hda_codec polyval_clmulni snd_hda_core polyval_generic ghash_clmulni_intel snd_hwdep iTCO_wdt sha512_ssse3 intel_pmc_bxt sha256_ssse3 snd_pcm joydev iTCO_vendor_support sha1_ssse3 snd_timer aesni_intel snd crypto_simd i2c_i801 psmouse cryptd pcspkr i2c_smbus soundcore lpc_ich i2c_mux mousedev mac_hid crypto_user loop dm_mod nfnetlink vsock_loopback vmw_vsock_virtio_transport_common vmw_vsock_vmci_transport vsock vmw_vmci qemu_fw_cfg ip_tables x_tables ext4 crc16 mbcache jbd2 nouveau drm_ttm_helper ttm video gpu_sched i2c_algo_bit drm_gpuvm serio_raw drm_exec atkbd mxm_wmi wmi libps2 vivaldi_fmap drm_display_helper virtio_net net_failover cec intel_agp virtio_input virtio_rng virtio_console failover virtio_blk i8042 intel_gtt serio
> Feb 06 08:54:47 archvm kernel: CPU: 0 UID: 1000 PID: 1962 Comm: rnote Not tainted 6.14.0-rc1-1-mainline #1 715c0460cf5d3cc18e3178ef3209cee42e97ae1c
> Feb 06 08:54:47 archvm kernel: Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS unknown 02/02/2022
> Feb 06 08:54:47 archvm kernel: Call Trace:
> Feb 06 08:54:47 archvm kernel:
> Feb 06 08:54:47 archvm kernel: dump_stack_lvl+0x5d/0x80
> Feb 06 08:54:47 archvm kernel: bad_page.cold+0x7a/0x91
> Feb 06 08:54:47 archvm kernel: __rmqueue_pcplist+0x200/0xc50
> Feb 06 08:54:47 archvm kernel: get_page_from_freelist+0x2ae/0x1740
> Feb 06 08:54:47 archvm kernel: ? srso_return_thunk+0x5/0x5f
> Feb 06 08:54:47 archvm kernel: ? __pm_runtime_suspend+0x69/0xc0
> Feb 06 08:54:47 archvm kernel: ? srso_return_thunk+0x5/0x5f
> Feb 06 08:54:47 archvm kernel: ? __seccomp_filter+0x303/0x520
> Feb 06 08:54:47 archvm kernel: ? srso_return_thunk+0x5/0x5f
> Feb 06 08:54:47 archvm kernel: __alloc_frozen_pages_noprof+0x184/0x330
> Feb 06 08:54:47 archvm kernel: alloc_pages_mpol+0x7d/0x160
> Feb 06 08:54:47 archvm kernel: folio_alloc_mpol_noprof+0x14/0x40
> Feb 06 08:54:47 archvm kernel: vma_alloc_folio_noprof+0x69/0xb0
> Feb 06 08:54:47 archvm kernel: do_anonymous_page+0x32a/0x8b0
> Feb 06 08:54:47 archvm kernel: ? srso_return_thunk+0x5/0x5f
> Feb 06 08:54:47 archvm kernel: ? ___pte_offset_map+0x1b/0x180
> Feb 06 08:54:47 archvm kernel: __handle_mm_fault+0xb5e/0xfe0
> Feb 06 08:54:47 archvm kernel: handle_mm_fault+0xe2/0x2c0
> Feb 06 08:54:47 archvm kernel: do_user_addr_fault+0x217/0x620
> Feb 06 08:54:47 archvm kernel: exc_page_fault+0x81/0x1b0
> Feb 06 08:54:47 archvm kernel: asm_exc_page_fault+0x26/0x30
> Feb 06 08:54:47 archvm kernel: RIP: 0033:0x7fcfc31c8cf9
> Feb 06 08:54:47 archvm kernel: Code: 34 19 49 39 d4 49 89 74 24 60 0f 95 c2 48 29 d8 48 83 c1 10 0f b6 d2 48 83 c8 01 48 c1 e2 02 48 09 da 48 83 ca 01 48 89 51 f8 <48> 89 46 08 e9 22 ff ff ff 48 8d 3d 07 ed 10 00 e8 62 c3 ff ff 48
> Feb 06 08:54:47 archvm kernel: RSP: 002b:00007fff1f931850 EFLAGS: 00010206
> Feb 06 08:54:47 archvm kernel: RAX: 000000000000bee1 RBX: 0000000000000140 RCX: 000056541d491ff0
> Feb 06 08:54:47 archvm kernel: RDX: 0000000000000141 RSI: 000056541d492120 RDI: 0000000000000000
> Feb 06 08:54:47 archvm kernel: RBP: 00007fff1f9318a0 R08: 0000000000000140 R09: 0000000000000001
> Feb 06 08:54:47 archvm kernel: R10: 0000000000000004 R11: 0000565419567488 R12: 00007fcfc3308ac0
> Feb 06 08:54:47 archvm kernel: R13: 0000000000000130 R14: 00007fcfc3308b20 R15: 0000000000000140
> Feb 06 08:54:47 archvm kernel:
> Feb 06 08:54:47 archvm kernel: Disabling lock debugging due to kernel taint
> Feb 06 08:54:47 archvm kernel: Oops: general protection fault, probably for non-canonical address 0xdead000000000122: 0000 [#1] PREEMPT SMP NOPTI
> Feb 06 08:54:47 archvm kernel: CPU: 0 UID: 1000 PID: 1962 Comm: rnote Tainted: G B 6.14.0-rc1-1-mainline #1 715c0460cf5d3cc18e3178ef3209cee42e97ae1c
> Feb 06 08:54:47 archvm kernel: Tainted: [B]=BAD_PAGE
> Feb 06 08:54:47 archvm kernel: Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS unknown 02/02/2022
> Feb 06 08:54:47 archvm kernel: RIP: 0010:__rmqueue_pcplist+0xb0/0xc50
> Feb 06 08:54:47 archvm kernel: Code: 00 4c 01 f0 48 89 7c 24 30 48 89 44 24 20 49 8b 04 24 49 39 c4 0f 84 6c 01 00 00 49 8b 14 24 48 8b 42 08 48 8b 0a 48 8d 5a f8 <48> 3b 10 0f 85 8d 0b 00 00 48 3b 51 08 0f 85 d5 0f be ff 48 89 41
> Feb 06 08:54:47 archvm kernel: RSP: 0000:ffffab3b84a2faa0 EFLAGS: 00010297
> Feb 06 08:54:47 archvm kernel: RAX: dead000000000122 RBX: ffffdd38819d61c0 RCX: dead000000000100
> Feb 06 08:54:47 archvm kernel: RDX: ffffdd38819d61c8 RSI: ffff9b31fd2218c0 RDI: ffff9b31fd2218c0
> Feb 06 08:54:47 archvm kernel: RBP: 0000000000000010 R08: 0000000000000000 R09: ffffab3b84a2f920
> Feb 06 08:54:47 archvm kernel: R10: ffffffffbdeb44a8 R11: 0000000000000003 R12: ffff9b31fd23d4b0
> Feb 06 08:54:47 archvm kernel: R13: 0000000000000000 R14: ffff9b31fef21980 R15: ffff9b31fd23d480
> Feb 06 08:54:47 archvm kernel: FS: 00007fcfbead5140(0000) GS:ffff9b31fd200000(0000) knlGS:0000000000000000
> Feb 06 08:54:47 archvm kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> Feb 06 08:54:47 archvm kernel: CR2: 000056541d492128 CR3: 000000001ed94000 CR4: 00000000003506f0
> Feb 06 08:54:47 archvm kernel: Call Trace:
> Feb 06 08:54:47 archvm kernel:
> Feb 06 08:54:47 archvm kernel: ? __die_body.cold+0x19/0x27
> Feb 06 08:54:47 archvm kernel: ? die_addr+0x3c/0x60
> Feb 06 08:54:47 archvm kernel: ? exc_general_protection+0x17d/0x400
> Feb 06 08:54:47 archvm kernel: ? asm_exc_general_protection+0x26/0x30
> Feb 06 08:54:47 archvm kernel: ? __rmqueue_pcplist+0xb0/0xc50
> Feb 06 08:54:47 archvm kernel: get_page_from_freelist+0x2ae/0x1740
> Feb 06 08:54:47 archvm kernel: ? srso_return_thunk+0x5/0x5f
> Feb 06 08:54:47 archvm kernel: ? __pm_runtime_suspend+0x69/0xc0
> Feb 06 08:54:47 archvm kernel: ? srso_return_thunk+0x5/0x5f
> Feb 06 08:54:47 archvm kernel: ? __seccomp_filter+0x303/0x520
> Feb 06 08:54:47 archvm kernel: ? srso_return_thunk+0x5/0x5f
> Feb 06 08:54:47 archvm kernel: __alloc_frozen_pages_noprof+0x184/0x330
> Feb 06 08:54:47 archvm kernel: alloc_pages_mpol+0x7d/0x160
> Feb 06 08:54:47 archvm kernel: folio_alloc_mpol_noprof+0x14/0x40
> Feb 06 08:54:47 archvm kernel: vma_alloc_folio_noprof+0x69/0xb0
> Feb 06 08:54:47 archvm kernel: do_anonymous_page+0x32a/0x8b0
> Feb 06 08:54:47 archvm kernel: ? srso_return_thunk+0x5/0x5f
> Feb 06 08:54:47 archvm kernel: ? ___pte_offset_map+0x1b/0x180
> Feb 06 08:54:47 archvm kernel: __handle_mm_fault+0xb5e/0xfe0
> Feb 06 08:54:47 archvm kernel: handle_mm_fault+0xe2/0x2c0
> Feb 06 08:54:47 archvm kernel: do_user_addr_fault+0x217/0x620
> Feb 06 08:54:47 archvm kernel: exc_page_fault+0x81/0x1b0
> Feb 06 08:54:47 archvm kernel: asm_exc_page_fault+0x26/0x30
> Feb 06 08:54:47 archvm kernel: RIP: 0033:0x7fcfc31c8cf9
> Feb 06 08:54:47 archvm kernel: Code: 34 19 49 39 d4 49 89 74 24 60 0f 95 c2 48 29 d8 48 83 c1 10 0f b6 d2 48 83 c8 01 48 c1 e2 02 48 09 da 48 83 ca 01 48 89 51 f8 <48> 89 46 08 e9 22 ff ff ff 48 8d 3d 07 ed 10 00 e8 62 c3 ff ff 48
> Feb 06 08:54:47 archvm kernel: RSP: 002b:00007fff1f931850 EFLAGS: 00010206
> Feb 06 08:54:47 archvm kernel: RAX: 000000000000bee1 RBX: 0000000000000140 RCX: 000056541d491ff0
> Feb 06 08:54:47 archvm kernel: RDX: 0000000000000141 RSI: 000056541d492120 RDI: 0000000000000000
> Feb 06 08:54:47 archvm kernel: RBP: 00007fff1f9318a0 R08: 0000000000000140 R09: 0000000000000001
> Feb 06 08:54:47 archvm kernel: R10: 0000000000000004 R11: 0000565419567488 R12: 00007fcfc3308ac0
> Feb 06 08:54:47 archvm kernel: R13: 0000000000000130 R14: 00007fcfc3308b20 R15: 0000000000000140
> Feb 06 08:54:47 archvm kernel:
> Feb 06 08:54:47 archvm kernel: Modules linked in: snd_seq_dummy snd_hrtimer snd_seq snd_seq_device rfkill vfat fat intel_rapl_msr intel_rapl_common kvm_amd ccp snd_hda_codec_hdmi snd_hda_codec_generic snd_hda_intel snd_intel_dspcfg kvm snd_intel_sdw_acpi snd_hda_codec polyval_clmulni snd_hda_core polyval_generic ghash_clmulni_intel snd_hwdep iTCO_wdt sha512_ssse3 intel_pmc_bxt sha256_ssse3 snd_pcm joydev iTCO_vendor_support sha1_ssse3 snd_timer aesni_intel snd crypto_simd i2c_i801 psmouse cryptd pcspkr i2c_smbus soundcore lpc_ich i2c_mux mousedev mac_hid crypto_user loop dm_mod nfnetlink vsock_loopback vmw_vsock_virtio_transport_common vmw_vsock_vmci_transport vsock vmw_vmci qemu_fw_cfg ip_tables x_tables ext4 crc16 mbcache jbd2 nouveau drm_ttm_helper ttm video gpu_sched i2c_algo_bit drm_gpuvm serio_raw drm_exec atkbd mxm_wmi wmi libps2 vivaldi_fmap drm_display_helper virtio_net net_failover cec intel_agp virtio_input virtio_rng virtio_console failover virtio_blk i8042 intel_gtt serio
> Feb 06 08:54:47 archvm kernel: ---[ end trace 0000000000000000 ]---
> Feb 06 08:54:47 archvm kernel: RIP: 0010:__rmqueue_pcplist+0xb0/0xc50
> Feb 06 08:54:47 archvm kernel: Code: 00 4c 01 f0 48 89 7c 24 30 48 89 44 24 20 49 8b 04 24 49 39 c4 0f 84 6c 01 00 00 49 8b 14 24 48 8b 42 08 48 8b 0a 48 8d 5a f8 <48> 3b 10 0f 85 8d 0b 00 00 48 3b 51 08 0f 85 d5 0f be ff 48 89 41
> Feb 06 08:54:47 archvm kernel: RSP: 0000:ffffab3b84a2faa0 EFLAGS: 00010297
> Feb 06 08:54:47 archvm kernel: RAX: dead000000000122 RBX: ffffdd38819d61c0 RCX: dead000000000100
> Feb 06 08:54:47 archvm kernel: RDX: ffffdd38819d61c8 RSI: ffff9b31fd2218c0 RDI: ffff9b31fd2218c0
> Feb 06 08:54:47 archvm kernel: RBP: 0000000000000010 R08: 0000000000000000 R09: ffffab3b84a2f920
> Feb 06 08:54:47 archvm kernel: R10: ffffffffbdeb44a8 R11: 0000000000000003 R12: ffff9b31fd23d4b0
> Feb 06 08:54:47 archvm kernel: R13: 0000000000000000 R14: ffff9b31fef21980 R15: ffff9b31fd23d480
> Feb 06 08:54:47 archvm kernel: FS: 00007fcfbead5140(0000) GS:ffff9b31fd200000(0000) knlGS:0000000000000000
> Feb 06 08:54:47 archvm kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> Feb 06 08:54:47 archvm kernel: CR2: 000056541d492128 CR3: 000000001ed94000 CR4: 00000000003506f0
> Feb 06 08:54:47 archvm kernel: note: rnote[1962] exited with preempt_count 2
> Feb 06 08:54:50 archvm geoclue[844]: Service not used for 60 seconds. Shutting down..
> Feb 06 08:55:01 archvm systemd[990]: Starting Virtual filesystem metadata service...
> Feb 06 08:55:14 archvm kernel: watchdog: BUG: soft lockup - CPU#0 stuck for 26s! [kworker/0:3:370]
> Feb 06 08:55:14 archvm kernel: CPU#0 Utilization every 4s during lockup:
> Feb 06 08:55:14 archvm kernel: #1: 100% system, 0% softirq, 1% hardirq, 0% idle
> Feb 06 08:55:14 archvm kernel: #2: 100% system, 0% softirq, 1% hardirq, 0% idle
> Feb 06 08:55:14 archvm kernel: #3: 100% system, 0% softirq, 1% hardirq, 0% idle
> Feb 06 08:55:14 archvm kernel: #4: 100% system, 0% softirq, 1% hardirq, 0% idle
> Feb 06 08:55:14 archvm kernel: #5: 100% system, 0% softirq, 1% hardirq, 0% idle
> Feb 06 08:55:14 archvm kernel: Modules linked in: snd_seq_dummy snd_hrtimer snd_seq snd_seq_device rfkill vfat fat intel_rapl_msr intel_rapl_common kvm_amd ccp snd_hda_codec_hdmi snd_hda_codec_generic snd_hda_intel snd_intel_dspcfg kvm snd_intel_sdw_acpi snd_hda_codec polyval_clmulni snd_hda_core polyval_generic ghash_clmulni_intel snd_hwdep iTCO_wdt sha512_ssse3 intel_pmc_bxt sha256_ssse3 snd_pcm joydev iTCO_vendor_support sha1_ssse3 snd_timer aesni_intel snd crypto_simd i2c_i801 psmouse cryptd pcspkr i2c_smbus soundcore lpc_ich i2c_mux mousedev mac_hid crypto_user loop dm_mod nfnetlink vsock_loopback vmw_vsock_virtio_transport_common vmw_vsock_vmci_transport vsock vmw_vmci qemu_fw_cfg ip_tables x_tables ext4 crc16 mbcache jbd2 nouveau drm_ttm_helper ttm video gpu_sched i2c_algo_bit drm_gpuvm serio_raw drm_exec atkbd mxm_wmi wmi libps2 vivaldi_fmap drm_display_helper virtio_net net_failover cec intel_agp virtio_input virtio_rng virtio_console failover virtio_blk i8042 intel_gtt serio
> Feb 06 08:55:14 archvm kernel: CPU: 0 UID: 0 PID: 370 Comm: kworker/0:3 Tainted: G B D 6.14.0-rc1-1-mainline #1 715c0460cf5d3cc18e3178ef3209cee42e97ae1c
> Feb 06 08:55:14 archvm kernel: Tainted: [B]=BAD_PAGE, [D]=DIE
> Feb 06 08:55:14 archvm kernel: Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS unknown 02/02/2022
> Feb 06 08:55:14 archvm kernel: Workqueue: mm_percpu_wq vmstat_update
> Feb 06 08:55:14 archvm kernel: RIP: 0010:__pv_queued_spin_lock_slowpath+0x267/0x490
> Feb 06 08:55:14 archvm kernel: Code: 14 0f 85 5c fe ff ff 41 c6 45 00 03 4c 89 fe 4c 89 ef e8 8c 2d 2e ff e9 47 fe ff ff f3 90 4d 8b 3e 4d 85 ff 74 f6 eb c1 f3 90 <83> ea 01 75 8a 48 83 3c 24 00 41 c6 45 01 00 0f 84 de 01 00 00 41
> Feb 06 08:55:14 archvm kernel: RSP: 0018:ffffab3b80907c98 EFLAGS: 00000206
> Feb 06 08:55:14 archvm kernel: RAX: 0000000000000003 RBX: 0000000000040000 RCX: 0000000000000008
> Feb 06 08:55:14 archvm kernel: RDX: 00000000000053b7 RSI: 0000000000000003 RDI: ffff9b31fd23d480
> Feb 06 08:55:14 archvm kernel: RBP: 0000000000000001 R08: ffff9b31fd237bc0 R09: 0000000000000000
> Feb 06 08:55:14 archvm kernel: R10: 0000000000000000 R11: fefefefefefefeff R12: 0000000000000100
> Feb 06 08:55:14 archvm kernel: R13: ffff9b31fd23d480 R14: ffff9b31fd237bc0 R15: 0000000000000000
> Feb 06 08:55:14 archvm kernel: FS: 0000000000000000(0000) GS:ffff9b31fd200000(0000) knlGS:0000000000000000
> Feb 06 08:55:14 archvm kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> Feb 06 08:55:14 archvm kernel: CR2: 00007fa8ba718100 CR3: 0000000016022000 CR4: 00000000003506f0
> Feb 06 08:55:14 archvm kernel: Call Trace:
> Feb 06 08:55:14 archvm kernel:
> Feb 06 08:55:14 archvm kernel: ? watchdog_timer_fn.cold+0x226/0x22b
> Feb 06 08:55:14 archvm kernel: ? srso_return_thunk+0x5/0x5f
> Feb 06 08:55:14 archvm kernel: ? __pfx_watchdog_timer_fn+0x10/0x10
> Feb 06 08:55:14 archvm kernel: ? __hrtimer_run_queues+0x132/0x2a0
> Feb 06 08:55:14 archvm kernel: ? hrtimer_interrupt+0xff/0x230
> Feb 06 08:55:14 archvm kernel: ? __sysvec_apic_timer_interrupt+0x55/0x100
> Feb 06 08:55:14 archvm kernel: ? sysvec_apic_timer_interrupt+0x6c/0x90
> Feb 06 08:55:14 archvm kernel:
> Feb 06 08:55:14 archvm kernel:
> Feb 06 08:55:14 archvm kernel: ? asm_sysvec_apic_timer_interrupt+0x1a/0x20
> Feb 06 08:55:14 archvm kernel: ? __pv_queued_spin_lock_slowpath+0x267/0x490
> Feb 06 08:55:14 archvm kernel: ? __pv_queued_spin_lock_slowpath+0x2be/0x490
> Feb 06 08:55:14 archvm kernel: _raw_spin_lock+0x29/0x30
> Feb 06 08:55:14 archvm kernel: decay_pcp_high+0x63/0x90
> Feb 06 08:55:14 archvm kernel: refresh_cpu_vm_stats+0xf7/0x240
> Feb 06 08:55:14 archvm kernel: vmstat_update+0x13/0x50
> Feb 06 08:55:14 archvm kernel: process_one_work+0x17e/0x330
> Feb 06 08:55:14 archvm kernel: worker_thread+0x2ce/0x3f0
> Feb 06 08:55:14 archvm kernel: ? __pfx_worker_thread+0x10/0x10
> Feb 06 08:55:14 archvm kernel: kthread+0xef/0x230
> Feb 06 08:55:14 archvm kernel: ? __pfx_kthread+0x10/0x10
> Feb 06 08:55:14 archvm kernel: ret_from_fork+0x34/0x50
> Feb 06 08:55:14 archvm kernel: ? __pfx_kthread+0x10/0x10
> Feb 06 08:55:14 archvm kernel: ret_from_fork_asm+0x1a/0x30
> Feb 06 08:55:14 archvm kernel:
