Hi, After beginning the release cycle of the 6.4 kernel I noted that when I reboot or turn off the computer the last message which I see is a use-after-free bug found by kasan sanitizer. Here is photo: https://ibb.co/1fxMYjt Below photo transcripted to text form: [ 87.946202] ================================================================== [ 87.946247] BUG: KASAN: use-after-free in tasklet_action_common.isra.0+0x6a4/0x7a0 [ 87.9462811 Read of size 8 at addr ffff8882b46a6a88 by task ksoftirqd/2/29 [ 87.946306] [ 87.946315] CPU: 2 PID: 29 Comm: ksoftirqd/2 Tainted: G W L ------- --- 6.4.0-0.rc5.20230606gitf8dba31b0a82.42.fc39.x86_64+debug #1 [ 87.946359] Hardware name: Micro-Star International Co., Ltd. MS-7D73/MPG B650I EDGE WIFI (MS-7D73), BIOS 1.30 05/24/2023 [ 87.946396] Call Trace: [ 87.946408] <TASK> [ 87.9464191 dump_stack_lvl+0x76/0xd0 [ 87.946439] print_report+0xcf/0x670 [ 87.946459] ? tasklet_action_common.isra.0+0x6a4/0x7a0 [ 87.946481] ? tasklet action_common.isra.0+0x6a4/0x7a0 [ 87.946502] kasan_report+0xa8/0xe0 [ 87.946531] ? tasklet_action_common.isra.0+0x6a4/0x7a0 [ 87.946555] tasklet_action_common.isra.0+0x6a4/0x7a0 [ 87.946577] __do_softirq+0x218/0x8bb [ 87.946596] ? __pfx___do_softirq+0x10/0x10 [ 87.946614] ? run_ksoftirqd+Ox73/0x80 [ 87.946633] ? smpboot_thread_fn+0x5bc/0x9b0 [ 87.946651] run_ksoftirqd+0x4b/0x80 [ 87.946668] smpboot_thread_fn+0x5bc/0x9b0 [ 87.946687] ? __pfx_smpboot_thread_fn+0x10/0x10 [ 87.946706] kthread+0x2eb/0x3c0 [ 87.946722] ? __pfx_kthread+0x10/0x10 [ 87.946740] ret_from_fork+0x29/0x50 [ 87.946760] </TASK> [ 87.946771] [ 87.946778] The buggy address belongs to the physical page: [ 87.946799] page:000000008f30de24 refcount:0 mapcount:0 mapping:0000000000000000 index:0x0 pfn:0x2b46a6 [ 87.946833] flags: 0x17ffffc0000000(node=0|zone=2|lastcpupid=0x1fffff) [ 87.946857] page_type: Oxffffffff() [ 87.946873] raw: 0017ffffc0000000 0000000000000000 dead000000000122 0000000000000000 [ 87.946901] raw: 0000000000000000 0000000000000000 00000000ffffffff 0000000000000000 [ 87.946930] page dumped because: kasan: bad access detected [ 87.946949] r8169 0000:0e:00.0 enp14s0: Link is Down [ 87.946950] [ 87.946968] Memory state around the buggy address: [ 87.946970] ffff8882b46a6980: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff [ 87.946971] ffff8882b46a6a00: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff [ 87.946972] >ffff8882b46a6a80: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff [ 87.947093] ^ [ 87.947109] ffff8882b46a6b00: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff [ 87.947134] ffff8882b46a6b80: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff [ 87.947158] ================================================================== [ 87.947186] Disabling lock debugging due to kernel taint I suppose many users didn't notice it because all modern Linux distro use a plymouth screen which hides all kernel messages during boot and shutdown. And this bug message is not recording in journalctl, because at the stage when this message appears journalctl was already offline. I used git bisect for trying to find the problem commit: And answered: - "good" when the computer was finishing work without the use-after-free message. - "bad" every time when I saw use-after-free bug message. - "skip" when the computer was stucking at the shutdown. And I got such bisect log: git bisect start # status: waiting for both good and bad commits # good: [173ea743bf7a9eef04460e03b00ba267cc52aee2] Merge tag 'pull-nios2' of git://git.kernel.org/pub/scm/linux/kernel/git/viro/vfs git bisect good 173ea743bf7a9eef04460e03b00ba267cc52aee2 # status: waiting for bad commit, 1 good commit known # bad: [6e98b09da931a00bf4e0477d0fa52748bf28fcce] Merge tag 'net-next-6.4' of git://git.kernel.org/pub/scm/linux/kernel/git/netdev/net-next git bisect bad 6e98b09da931a00bf4e0477d0fa52748bf28fcce # good: [2c96606a0f8b7900387dbeb6532b59527999834d] Merge tag 'gpio-updates-for-v6.4' of git://git.kernel.org/pub/scm/linux/kernel/git/brgl/linux git bisect good 2c96606a0f8b7900387dbeb6532b59527999834d # bad: [ca288965801572fe41386560d4e6c5cc0e5cc56d] Merge tag 'wireless-next-2023-04-21' of git://git.kernel.org/pub/scm/linux/kernel/git/wireless/wireless-next git bisect bad ca288965801572fe41386560d4e6c5cc0e5cc56d # good: [d56417ad1133fc41752bb9fe37da7ae3187395a4] net: phy: smsc: clear edpd_enable if interrupt mode is used git bisect good d56417ad1133fc41752bb9fe37da7ae3187395a4 # good: [c54876cd5961ce0f8e74807f79a6739cd6b35ddf] net/sched: pass netlink extack to mqprio and taprio offload git bisect good c54876cd5961ce0f8e74807f79a6739cd6b35ddf # skip: [3288ee5844b74cebb94ed15bc9b5b9d3223ae038] Merge ath-next from git://git.kernel.org/pub/scm/linux/kernel/git/kvalo/ath.git git bisect skip 3288ee5844b74cebb94ed15bc9b5b9d3223ae038 # good: [b6d85cf5bd1433c5dd6bf6bb3a176537184c630c] net/ipv6: Initialise msg_control_is_user git bisect good b6d85cf5bd1433c5dd6bf6bb3a176537184c630c # skip: [d2a158d113cbfe37a5dd3f44dc96d008dd910a81] Merge tag 'mt76-for-kvalo-2023-04-18' of https://github.com/nbd168/wireless git bisect skip d2a158d113cbfe37a5dd3f44dc96d008dd910a81 # good: [02461d9368c59510ef51cc8a1db1f0f31cfbf9ad] wifi: rtw88: main: Reserve 8 bytes of extra TX headroom for SDIO cards git bisect good 02461d9368c59510ef51cc8a1db1f0f31cfbf9ad # good: [827145392a4aad635b93e5235b7d7fecc2fa31c7] net: enetc: only commit preemptible TCs to hardware when MM TX is active git bisect good 827145392a4aad635b93e5235b7d7fecc2fa31c7 # skip: [27db47ab1f47906c2392f9d246e244e412b19278] wifi: mt76: mt7996: enable mesh HW amsdu/de-amsdu support git bisect skip 27db47ab1f47906c2392f9d246e244e412b19278 # good: [22b68fc6d693e7a2b1c0eb852463f4a72522fa08] wifi: iwlwifi: mvm: fix RFKILL report when driver is going down git bisect good 22b68fc6d693e7a2b1c0eb852463f4a72522fa08 # good: [f94557154d9fc77c392844523388edd4661a27a3] wifi: wcn36xx: add support for pronto-v3 git bisect good f94557154d9fc77c392844523388edd4661a27a3 # good: [ccf73f6e69c0244a979e97eb6c38f80cd6cbc116] wifi: rtw88: add port switch for AP mode git bisect good ccf73f6e69c0244a979e97eb6c38f80cd6cbc116 # good: [a6f187f92bcc2b17821538b4a11d61764e68b091] wifi: rtw88: usb: fix priority queue to endpoint mapping git bisect good a6f187f92bcc2b17821538b4a11d61764e68b091 # skip: [61d1f54533496711e06fcfd42b93c5ded9e27c7a] wifi: mt76: move mcu_uni_event and mcu_reg_event in common code git bisect skip 61d1f54533496711e06fcfd42b93c5ded9e27c7a # good: [73175a042955e531ec355a8708585befa67a22db] sctp: delete the nested flexible array skip git bisect good 73175a042955e531ec355a8708585befa67a22db # good: [b9235aef84929e5330cb87125a6baf1cf7988983] wifi: ath12k: Remove redundant pci_clear_master git bisect good b9235aef84929e5330cb87125a6baf1cf7988983 # good: [6257c702264c44d74c6b71f0c62a7665da2dc356] wifi: ath11k: fix tx status reporting in encap offload mode git bisect good 6257c702264c44d74c6b71f0c62a7665da2dc356 # skip: [3b522cadedfe6e9e0e8193d7d4ab5aa8d0c73209] wifi: mt76: mt7996: fill txd by host driver git bisect skip 3b522cadedfe6e9e0e8193d7d4ab5aa8d0c73209 # skip: [f4d63a87b527de258eec5bd6e9547f063d472b79] wifi: mt76: dma: use napi_build_skb git bisect skip f4d63a87b527de258eec5bd6e9547f063d472b79 # skip: [09d4d6da1b65d09414e7bce61459593f3c80ead1] wifi: mt76: mt7921e: Set memory space enable in PCI_COMMAND if unset git bisect skip 09d4d6da1b65d09414e7bce61459593f3c80ead1 # skip: [230a167e094770fdcc104481719ef7b1a706fb27] wifi: mt76: set NL80211_EXT_FEATURE_CAN_REPLACE_PTK0 on supported drivers git bisect skip 230a167e094770fdcc104481719ef7b1a706fb27 # good: [49ce92fbee0b6bb8066dddf37489483b3b6b5c25] pds_core: add FW update feature to devlink git bisect good 49ce92fbee0b6bb8066dddf37489483b3b6b5c25 # skip: [12db28c3ef31f719bd18fa186a40bb152e6a527c] mt76: mt7921: fix kernel panic by accessing unallocated eeprom.data git bisect skip 12db28c3ef31f719bd18fa186a40bb152e6a527c # good: [45fd01f2fbf1119d083931b095ad6d0f13443d0e] net/mlx5e: Refactor duplicated code in mlx5e_ipsec_init_macs git bisect good 45fd01f2fbf1119d083931b095ad6d0f13443d0e # skip: [2631c5b6ef9d7c6707e020def6947464c8aa6f92] wifi: mt76: Replace zero-length array with flexible-array member git bisect skip 2631c5b6ef9d7c6707e020def6947464c8aa6f92 # good: [64822bdba456a145f7cb4c66d9939bf42c64ae62] dt-bindings: mt76: add active-low property for led git bisect good 64822bdba456a145f7cb4c66d9939bf42c64ae62 # skip: [6d6793cef6a491b8f6db5f40ef3334411293da32] wifi: mt76: mt7921: Replace fake flex-arrays with flexible-array members git bisect skip 6d6793cef6a491b8f6db5f40ef3334411293da32 # skip: [3d78c46423c6567ed25ca033e086865b1b4d5ae1] wifi: mt76: mt7921e: stop chip reset worker in unregister hook git bisect skip 3d78c46423c6567ed25ca033e086865b1b4d5ae1 # good: [b100722a777f6455d913666a376f81342b2cb995] wifi: ath11k: Remove disabling of 80+80 and 160 MHz git bisect good b100722a777f6455d913666a376f81342b2cb995 # skip: [03eb52dd78cab08f13925aeec8315fbdbcba3253] wifi: mt76: mt7921: add Netgear AXE3000 (A8000) support git bisect skip 03eb52dd78cab08f13925aeec8315fbdbcba3253 # good: [6a8b899df1562a46a8c55cebc7d35508a24300d3] wifi: mt76: add mt76_connac_gen_ppe_thresh utility routine git bisect good 6a8b899df1562a46a8c55cebc7d35508a24300d3 # skip: [15ee62e73705df447971613de4fa660dd71ed40e] wifi: mt76: mt7996: enable BSS_CHANGED_BASIC_RATES support git bisect skip 15ee62e73705df447971613de4fa660dd71ed40e # skip: [5c47cdebbaeb7724df6f9f46917c93e53f791547] wifi: mt76: mt7921: fix missing unwind goto in `mt7921u_probe` git bisect skip 5c47cdebbaeb7724df6f9f46917c93e53f791547 # good: [97c75e1adeda78b3794936c617d8b86e9ebd54f5] wifi: rtw88: set pkg_type correctly for specific rtw8821c variants git bisect good 97c75e1adeda78b3794936c617d8b86e9ebd54f5 # good: [59a3a312009723e3e5082899655fdcc420e2b47a] wifi: rtw88: Fix memory leak in rtw88_usb git bisect good 59a3a312009723e3e5082899655fdcc420e2b47a # skip: [3d2892e05086d09aecf14ea64b2debbf495e313c] wifi: mt76: connac: fix txd multicast rate setting git bisect skip 3d2892e05086d09aecf14ea64b2debbf495e313c # good: [c2171b068beea766311e4c2858ef8497504c6e6d] wifi: mt76: mt7996: enable configured beacon tx rate git bisect good c2171b068beea766311e4c2858ef8497504c6e6d # good: [cd85c8b059c54b00e3b509e83fb36c2798f50128] wifi: rtl8xxxu: Add rtl8xxxu_write{8,16,32}_{set,clear} git bisect good cd85c8b059c54b00e3b509e83fb36c2798f50128 # skip: [e12b2e99b8799f26432528934edc8677888ad72f] wifi: mt76: mt7615: increase eeprom size for mt7663 git bisect skip e12b2e99b8799f26432528934edc8677888ad72f # good: [6c6d62ae8271bd4b55dd2ba4b7ed552162823880] wifi: rtw88: Update spelling in main.h git bisect good 6c6d62ae8271bd4b55dd2ba4b7ed552162823880 # good: [dfc39d4026fb2432363c0f77543c4cf3adca4c7b] net/packet: support mergeable feature of virtio git bisect good dfc39d4026fb2432363c0f77543c4cf3adca4c7b # only skipped commits left to test # possible first bad commit: [ca288965801572fe41386560d4e6c5cc0e5cc56d] Merge tag 'wireless-next-2023-04-21' of git://git.kernel.org/pub/scm/linux/kernel/git/wireless/wireless-next # possible first bad commit: [3288ee5844b74cebb94ed15bc9b5b9d3223ae038] Merge ath-next from git://git.kernel.org/pub/scm/linux/kernel/git/kvalo/ath.git # possible first bad commit: [d2a158d113cbfe37a5dd3f44dc96d008dd910a81] Merge tag 'mt76-for-kvalo-2023-04-18' of https://github.com/nbd168/wireless # possible first bad commit: [3b522cadedfe6e9e0e8193d7d4ab5aa8d0c73209] wifi: mt76: mt7996: fill txd by host driver # possible first bad commit: [230a167e094770fdcc104481719ef7b1a706fb27] wifi: mt76: set NL80211_EXT_FEATURE_CAN_REPLACE_PTK0 on supported drivers # possible first bad commit: [f4d63a87b527de258eec5bd6e9547f063d472b79] wifi: mt76: dma: use napi_build_skb # possible first bad commit: [e12b2e99b8799f26432528934edc8677888ad72f] wifi: mt76: mt7615: increase eeprom size for mt7663 # possible first bad commit: [27db47ab1f47906c2392f9d246e244e412b19278] wifi: mt76: mt7996: enable mesh HW amsdu/de-amsdu support Unfortunately git bisect did not say which exact commit is culprit here, but anyway we got the result with eigh commits which can be reviewed. -- Best Regards, Mike Gavrilov.
