On Tue, Nov 07, 2023 at 11:13:14PM +0800, Zorro Lang wrote: > On Tue, Nov 07, 2023 at 07:13:39PM +1100, Dave Chinner wrote: > > On Tue, Nov 07, 2023 at 04:05:22PM +0800, Zorro Lang wrote: > > > On Tue, Nov 07, 2023 at 07:33:50AM +1100, Dave Chinner wrote: > > > > On Tue, Nov 07, 2023 at 03:26:27AM +0800, Zorro Lang wrote: > > > > > Thanks for your reply :) I tried to do a kernel bisect long time, but > > > > > find nothing ... Then suddently, I found it's failed from a xfsprogs > > > > > change [1]. > > > > > > > > > > Although that's not the root cause of this bug (on s390x), it just > > > > > enabled "nrext64" by default, which I never tested on s390x before. > > > > > For now, we know this's an issue about this feature, and only on > > > > > s390x for now. > > > > > > > > That's not good. Can you please determine if this is a zero-day bug > > > > with the nrext64 feature? I think it was merged in 5.19, so if you > > > > could try to reproduce it on a 5.18 and 5.19 kernels first, that > > > > would be handy. > > > > > > Unfortunately, it's a bug be there nearly from beginning. The linux v5.19 > > > can trigger this bug (with latest xfsprogs for-next branch): > > > > Ok. Can you grab the pahole output for the xfs_dinode and > > xfs_log_dinode for s390 from both 5.18 and 5.19 kernel builds? > > (i.e. 'pahole fs/xfs/xfs_inode.o |less' and search for the two > > structures). > > I can't find xfs_log_dinode in fs/xfs/xfs_inode.o, but I can find both structures > in fs/xfs/xfs_inode_item.o, so below output base on: > > # pahole fs/xfs/xfs_inode_item.o > > The output on v5.19 is [1], v5.18 output is [2], the diff of 5.18 and 5.19 is [3]. Ok, so there's nothing wrong with the on-disk format definition or the journal format - they both lay out in exactly the right shape so I think at this point we need metadumps from the broken filesystems. Can you pick one of the failing tests and grab metadumps from the shutdown filesystem (i.e. before it is recovered) and then another from after it is recovered and the problem tripped over? I know I won't be able to replay the log on x86-64, but knowing what is in the journal vs what ends up being recovered will tell us where to look next. -Dave. -- Dave Chinner david@xxxxxxxxxxxxx
