On Mon, Dec 18, 2023 at 10:01:34PM +0800, Zorro Lang wrote: > Hi, > > Recently I hit a crash [1] on s390x with 64k directory block size xfs > (-n size=65536 -m crc=1,finobt=1,reflink=1,rmapbt=0,bigtime=1,inobtcount=1), > even not panic, a assertion failure will happen. > > I found it from an old downstream kernel at first, then reproduced it > on latest upstream mainline linux (v6.7-rc6). Can't be sure how long > time this issue be there, just reported it at first. > > I hit and reproduced it several times by running generic/648 on s390x > with "-n size=65536" xfs many times. Not sure if s390x is needed, I'll > try to check other arches. XFS (loop3): Metadata corruption detected at __xfs_dir3_data_check+0x372/0x6c0 [xfs], xfs_dir3_block block 0x1020 Where in the source code is this ^^^^^^^^^^^^^^^^^^^^^^^^^^^? Also, does "xfs: update dir3 leaf block metadata after swap" fix it? (I suspect it won't because that's a dirent block, but for-next has a few directory fixes in it.) --D > > Thanks, > Zorro > > [1] > [ 899.271221] run fstests generic/648 at 2023-12-18 01:18:03 > [ 899.454444] XFS (loop1): Mounting V5 Filesystem 9be840e1-9bd1-468e-81c5-179f2ebdce56 > [ 899.455844] XFS (loop1): Ending clean mount > [ 899.463623] XFS (loop1): Unmounting Filesystem 9be840e1-9bd1-468e-81c5-179f2ebdce56 > [ 899.587365] XFS (dm-3): Mounting V5 Filesystem d636e675-dfc9-4e5d-b777-ff856af136a9 > [ 899.588501] XFS (dm-3): Ending clean mount > ... > ... > [ 974.217814] XFS (loop3): Mounting V5 Filesystem c1954438-a18d-4b4a-ad32-0e29c40713ed > [ 974.803146] XFS (loop3): Starting recovery (logdev: internal) > [ 974.804634] XFS (loop3): Ending recovery (logdev: internal) > [ 975.782734] Direct I/O collision with buffered writes! File: /p2/df/f4f Comm: fsstress > [ 976.831942] lo_write_bvec: 15 callbacks suppressed > [ 976.831946] loop: Write error at byte offset 1766129664, length 4096. > [ 976.831953] I/O error, dev loop3, sector 3449468 op 0x1:(WRITE) flags 0x9800 phys_seg 1 prio class 2 > [ 976.831957] I/O error, dev loop3, sector 3449468 op 0x1:(WRITE) flags 0x9800 phys_seg 1 prio class 2 > [ 976.831960] XFS (loop3): log I/O error -5 > [ 976.831963] XFS (loop3): Filesystem has been shut down due to log error (0x2). > [ 976.831964] XFS (loop3): Please unmount the filesystem and rectify the problem(s). > [ 976.839603] buffer_io_error: 6 callbacks suppressed > [ 976.839604] Buffer I/O error on dev dm-3, logical block 20971392, async page read > [ 976.839607] Buffer I/O error on dev dm-3, logical block 20971393, async page read > [ 976.839609] Buffer I/O error on dev dm-3, logical block 20971394, async page read > [ 976.839611] Buffer I/O error on dev dm-3, logical block 20971395, async page read > [ 976.839612] Buffer I/O error on dev dm-3, logical block 20971396, async page read > [ 976.839613] Buffer I/O error on dev dm-3, logical block 20971397, async page read > [ 976.839614] Buffer I/O error on dev dm-3, logical block 20971398, async page read > [ 976.839616] Buffer I/O error on dev dm-3, logical block 20971399, async page read > [ 977.419266] XFS (loop3): Unmounting Filesystem c1954438-a18d-4b4a-ad32-0e29c40713ed > [ 977.423981] iomap_finish_ioend: 15 callbacks suppressed > [ 977.423982] dm-3: writeback error on inode 131, offset 1766125568, sector 4059696 > [ 977.423988] I/O error, dev loop3, sector 0 op 0x1:(WRITE) flags 0x800 phys_seg 0 prio class 2 > [ 978.434124] XFS (dm-3): Unmounting Filesystem d636e675-dfc9-4e5d-b777-ff856af136a9 > [ 978.449156] XFS (dm-3): log I/O error -5 > [ 978.449159] XFS (dm-3): Filesystem has been shut down due to log error (0x2). > [ 978.449160] XFS (dm-3): Please unmount the filesystem and rectify the problem(s). > [ 978.449163] XFS (dm-3): log I/O error -5 > [ 978.449164] XFS (dm-3): log I/O error -5 > [ 978.449165] XFS (dm-3): log I/O error -5 > [ 978.449166] XFS (dm-3): log I/O error -5 > [ 978.449167] XFS (dm-3): log I/O error -5 > [ 978.498053] XFS (dm-3): Mounting V5 Filesystem d636e675-dfc9-4e5d-b777-ff856af136a9 > [ 978.513331] XFS (dm-3): Starting recovery (logdev: internal) > [ 978.584227] XFS (dm-3): Ending recovery (logdev: internal) > [ 978.587276] loop3: detected capacity change from 0 to 6876346 > [ 978.591588] XFS (loop3): Mounting V5 Filesystem c1954438-a18d-4b4a-ad32-0e29c40713ed > [ 979.216565] XFS (loop3): Starting recovery (logdev: internal) > [ 979.225078] XFS (loop3): Bad dir block magic! > [ 979.225081] XFS: Assertion failed: 0, file: fs/xfs/xfs_buf_item_recover.c, line: 414 > [ 979.225120] ------------[ cut here ]------------ > [ 979.225122] WARNING: CPU: 1 PID: 157902 at fs/xfs/xfs_message.c:104 assfail+0x4e/0x68 [xfs] > [ 979.225349] Modules linked in: tls loop lcs ctcm fsm qeth ccwgroup zfcp qdio scsi_transport_fc dasd_fba_mod dasd_eckd_mod dasd_mod rfkill sunrpc vfio_ccw mdev vfio_iommu_type1 vfio drm fuse i2c_core drm_panel_orientation_quirks xfs libcrc32c ghash_s390 prng des_s390 virtio_net net_failover sha3_512_s390 virtio_blk failover sha3_256_s390 dm_mirror dm_region_hash dm_log dm_mod pkey zcrypt aes_s390 > [ 979.225373] CPU: 1 PID: 157902 Comm: mount Kdump: loaded Not tainted 6.7.0-rc6 #1 > [ 979.225376] Hardware name: IBM 8561 LT1 400 (KVM/Linux) > [ 979.225377] Krnl PSW : 0704c00180000000 000003ff7ffead7a (assfail+0x52/0x68 [xfs]) > [ 979.225457] R:0 T:1 IO:1 EX:1 Key:0 M:1 W:0 P:0 AS:3 CC:0 PM:0 RI:0 EA:3 > [ 979.225460] Krnl GPRS: c000000000000021 000003ff800549b0 ffffffffffffffea 000000000000000a > [ 979.225462] 0000038000363418 0000000000000000 000003ff80162b74 000000030001c127 > [ 979.225464] 00000000ccc51d40 000000030001c127 00000000dfa03000 0000000163d19600 > [ 979.225465] 000003ffa1f2ef68 000003ff80147c28 000003ff7ffead68 00000380003634c0 > [ 979.225476] Krnl Code: 000003ff7ffead6e: 95001010 cli 16(%r1),0 > [ 979.225476] 000003ff7ffead72: a774000a brc 7,000003ff7ffead86 > [ 979.225476] #000003ff7ffead76: af000000 mc 0,0 > [ 979.225476] >000003ff7ffead7a: eb6ff0a80004 lmg %r6,%r15,168(%r15) > [ 979.225476] 000003ff7ffead80: 07fe bcr 15,%r14 > [ 979.225476] 000003ff7ffead82: 47000700 bc 0,1792 > [ 979.225476] 000003ff7ffead86: af000000 mc 0,0 > [ 979.225476] 000003ff7ffead8a: 0707 bcr 0,%r7 > [ 979.225487] Call Trace: > [ 979.225488] [<000003ff7ffead7a>] assfail+0x52/0x68 [xfs] > [ 979.225568] ([<000003ff7ffead68>] assfail+0x40/0x68 [xfs]) > [ 979.225648] [<000003ff80004b4e>] xlog_recover_validate_buf_type+0x2a6/0x5c8 [xfs] > [ 979.225727] [<000003ff80005eca>] xlog_recover_buf_commit_pass2+0x382/0x448 [xfs] > [ 979.225807] [<000003ff8001077a>] xlog_recover_items_pass2+0x72/0xf0 [xfs] > [ 979.225886] [<000003ff80011436>] xlog_recover_commit_trans+0x39e/0x3c0 [xfs] > [ 979.225965] [<000003ff80011598>] xlog_recovery_process_trans+0x140/0x148 [xfs] > [ 979.226045] [<000003ff80011660>] xlog_recover_process_ophdr+0xc0/0x180 [xfs] > [ 979.226124] [<000003ff80012126>] xlog_recover_process_data+0xb6/0x168 [xfs] > [ 979.226204] [<000003ff800122dc>] xlog_recover_process+0x104/0x150 [xfs] > [ 979.226285] [<000003ff800125a4>] xlog_do_recovery_pass+0x27c/0x748 [xfs] > [ 979.226366] [<000003ff80012ec8>] xlog_do_log_recovery+0x88/0xd8 [xfs] > [ 979.226446] [<000003ff80012f64>] xlog_do_recover+0x4c/0x218 [xfs] > [ 979.226526] [<000003ff8001446a>] xlog_recover+0xda/0x1a0 [xfs] > [ 979.226607] [<000003ff7fffa51e>] xfs_log_mount+0x11e/0x280 [xfs] > [ 979.226686] [<000003ff7ffec226>] xfs_mountfs+0x3e6/0x928 [xfs] > [ 979.226765] [<000003ff7fff3b4c>] xfs_fs_fill_super+0x40c/0x7d8 [xfs] > [ 979.226844] [<00000001ef53bfd4>] get_tree_bdev+0x144/0x1d0 > [ 979.226850] [<00000001ef539a08>] vfs_get_tree+0x38/0x110 > [ 979.226852] [<00000001ef56c432>] do_new_mount+0x17a/0x2d0 > [ 979.226855] [<00000001ef56d0ac>] path_mount+0x1ac/0x818 > [ 979.226857] [<00000001ef56d81c>] __s390x_sys_mount+0x104/0x148 > [ 979.226860] [<00000001efbb6e00>] __do_syscall+0x1d0/0x1f8 > [ 979.226864] [<00000001efbc7088>] system_call+0x70/0x98 > [ 979.226868] Last Breaking-Event-Address: > [ 979.226868] [<000003ff7ffeabc4>] xfs_printk_level+0xac/0xd8 [xfs] > [ 979.226949] ---[ end trace 0000000000000000 ]--- > [ 979.227613] XFS (loop3): Metadata corruption detected at __xfs_dir3_data_check+0x372/0x6c0 [xfs], xfs_dir3_block block 0x1020 > [ 979.227732] XFS (loop3): Unmount and run xfs_repair > [ 979.227733] XFS (loop3): First 128 bytes of corrupted metadata buffer: > [ 979.227736] 00000000: 58 44 42 33 00 00 00 00 00 00 00 00 00 00 10 20 XDB3........... > [ 979.227738] 00000010: 00 00 00 00 00 00 00 00 c1 95 44 38 a1 8d 4b 4a ..........D8..KJ > [ 979.227739] 00000020: ad 32 0e 29 c4 07 13 ed 00 00 00 00 00 00 00 80 .2.)............ > [ 979.227740] 00000030: 02 68 fc c8 00 00 00 00 00 00 00 00 00 00 00 00 .h.............. > [ 979.227741] 00000040: 00 00 00 00 00 00 00 80 01 2e 02 00 00 00 00 40 ...............@ > [ 979.227742] 00000050: 00 00 00 00 00 00 00 80 02 2e 2e 02 00 00 00 50 ...............P > [ 979.227743] 00000060: 00 00 00 00 00 00 00 83 08 66 73 73 32 36 30 38 .........fss2608 > [ 979.227745] 00000070: 36 01 00 00 00 00 00 60 00 00 00 00 00 00 00 84 6......`........ > [ 979.227748] XFS (loop3): Corruption of in-memory data (0x8) detected at __xfs_buf_submit+0x78/0x230 [xfs] (fs/xfs/xfs_buf.c:1558). Shutting down filesystem. > [ 979.227826] XFS (loop3): Please unmount the filesystem and rectify the problem(s) > [ 979.227912] XFS (loop3): log mount/recovery failed: error -117 > [ 979.228061] XFS (loop3): log mount failed > [ 980.224124] XFS (dm-3): Unmounting Filesystem d636e675-dfc9-4e5d-b777-ff856af136a9 > [ 980.318601] XFS (loop0): Unmounting Filesystem c8777546-cb21-46f2-9963-456588f60b85 > [ 980.353946] XFS (loop0): Mounting V5 Filesystem c8777546-cb21-46f2-9963-456588f60b85 > [ 980.355207] XFS (loop0): Ending clean mount > [ 982.585362] XFS (loop1): Mounting V5 Filesystem ec19bbbd-e925-41c9-befe-28218e74a23b > [ 982.586881] XFS (loop1): Ending clean mount > [ 982.588944] XFS (loop1): Unmounting Filesystem ec19bbbd-e925-41c9-befe-28218e74a23b > [ 982.603160] XFS (loop0): EXPERIMENTAL online scrub feature in use. Use at your own risk! > [ 982.620838] XFS (loop0): Unmounting Filesystem c8777546-cb21-46f2-9963-456588f60b85 > [ 982.636206] XFS (loop0): Mounting V5 Filesystem c8777546-cb21-46f2-9963-456588f60b85 > [ 982.637231] XFS (loop0): Ending clean mount > > >
