On Fri 16-07-21 10:03:25, Will Deacon wrote: > On Thu, Jul 15, 2021 at 11:55:34PM -0000, CKI Project wrote: > > All kernel binaries, config files, and logs are available for download here: > > > > https://arr-cki-prod-datawarehouse-public.s3.amazonaws.com/index.html?prefix=datawarehouse-public/2021/07/15/337870374 > > > > One or more kernel tests failed: > > > > aarch64: > > 💥 Podman system integration test - as user > > This is a NULL pointer dereference in the recently introduced > cleanup_offline_cgwbs_workfn() function [c22d70a162d3 ("writeback, cgroup: > release dying cgwbs by switching attached inodes")]. Adding some of those > folks to CC. Thanks for report! This bug was already reported here [1] and Roman already has a tenative fix. Stay tuned :) [1] https://lore.kernel.org/linux-fsdevel/YO5kCzI133B/fHiS@xxxxxxxxxxxxxxxxxxxxxxxxxxx Honza > addr2line gives: > > | linux/./include/linux/atomic-arch-fallback.h:2265 > | linux/./include/linux/atomic-arch-fallback.h:2290 > | linux/./include/asm-generic/atomic-instrumented.h:1149 > | linux/./include/asm-generic/atomic-long.h:491 > | linux/percpu-refcount.h:247 > | linux/percpu-refcount.h:266 > | linux/backing-dev-defs.h:227 > | linux/mm/backing-dev.c:679 > > and the crash (this is an arm64 machine) is: > > [ 4794.406516] Unable to handle kernel NULL pointer dereference at virtual address 0000000000000000 > [ 4794.415307] Mem abort info: > [ 4794.418088] ESR = 0x96000004 > [ 4794.421130] EC = 0x25: DABT (current EL), IL = 32 bits > [ 4794.426428] SET = 0, FnV = 0 > [ 4794.429470] EA = 0, S1PTW = 0 > [ 4794.432597] FSC = 0x04: level 0 translation fault > [ 4794.437461] Data abort info: > [ 4794.440329] ISV = 0, ISS = 0x00000004 > [ 4794.444151] CM = 0, WnR = 0 > [ 4794.447105] user pgtable: 4k pages, 48-bit VAs, pgdp=00000009072be000 > [ 4794.453534] [0000000000000000] pgd=0000000000000000, p4d=0000000000000000 > [ 4794.460312] Internal error: Oops: 96000004 [#1] SMP > [ 4794.465179] Modules linked in: xt_nat xt_addrtype xt_mark nft_chain_nat xt_MASQUERADE nf_nat xt_comment veth bridge stp llc loop tun af_key crypto_user scsi_transport_iscsi xt_multiport ip_gre ip_tunnel gre overlay xt_CONNSECMARK xt_SECMARK nft_counter xt_state xt_conntrack nft_compat ah6 ah4 nft_objref nft_ct nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nf_tables nfnetlink jfs sctp ip6_udp_tunnel udp_tunnel dm_log_writes dm_flakey rfkill mlx5_ib ib_uverbs ib_core sunrpc coresight_etm4x i2c_smbus coresight_replicator coresight_tpiu coresight_tmc joydev mlx5_core acpi_ipmi mlxfw ipmi_ssif psample ipmi_devintf ipmi_msghandler coresight_funnel coresight thunderx2_pmu vfat fat fuse zram ip_tables xfs crct10dif_ce ast ghash_ce i2c_algo_bit drm_vram_helper drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops cec drm_ttm_helper ttm drm gpio_xlp i2c_xlp9xx uas usb_storage aes_neon_bs [last unloaded: raid10] > [ 4794.545404] CPU: 86 PID: 1121023 Comm: kworker/u513:9 Tainted: G OE 5.14.0-rc1 #1 > [ 4794.554090] Hardware name: HPE Apollo 70 /C01_APACHE_MB , BIOS L50_5.13_1.15 05/08/2020 > [ 4794.563817] Workqueue: events_unbound cleanup_offline_cgwbs_workfn > [ 4794.569990] pstate: 004000c9 (nzcv daIF +PAN -UAO -TCO BTYPE=--) > [ 4794.575985] pc : cleanup_offline_cgwbs_workfn+0x2d8/0x350 > [ 4794.581372] lr : cleanup_offline_cgwbs_workfn+0xc8/0x350 > [ 4794.586672] sp : ffff800043ae3d40 > [ 4794.589973] x29: ffff800043ae3d40 x28: 0000000000000000 x27: 0000000000000001 > [ 4794.597097] x26: 0000000000000000 x25: 00000000000000e0 x24: ffff800011f01438 > [ 4794.604221] x23: ffff800043ae3db0 x22: ffff800011f01428 x21: ffff80001228c5f0 > [ 4794.611345] x20: ffff000817299800 x19: ffff000817299a80 x18: 000000000000000e > [ 4794.618469] x17: 0000000079c37e0f x16: 0000000000010000 x15: fffffc002084ac02 > [ 4794.625593] x14: 0000000000000000 x13: 0000000000000030 x12: 0000000000000040 > [ 4794.632716] x11: ffff000812333d98 x10: ffff000812333d9a x9 : ffff8000102dfd84 > [ 4794.639840] x8 : ffff000812331b88 x7 : 0000000000000000 x6 : ffff000812331d48 > [ 4794.646964] x5 : 0000000000000000 x4 : ffff00086f2a4a80 x3 : 0000000000000000 > [ 4794.654088] x2 : ffff800043ae3db0 x1 : ffff00086f2a4a80 x0 : 0000000000000003 > [ 4794.661212] Call trace: > [ 4794.663646] cleanup_offline_cgwbs_workfn+0x2d8/0x350 > [ 4794.668685] process_one_work+0x1f0/0x4ac > [ 4794.672686] worker_thread+0x180/0x400 > [ 4794.676423] kthread+0x110/0x114 > [ 4794.679641] ret_from_fork+0x10/0x18 > [ 4794.683209] Code: d63f0020 97f9c259 17ffffb1 f8588263 (f9400061) > [ 4794.689290] ---[ end trace 8c5d2f7872fef978 ]--- > > Will -- Jan Kara <jack@xxxxxxxx> SUSE Labs, CR