On Thu, Apr 28, 2022 at 07:25:38AM +1000, Dave Chinner wrote:
> On Wed, Apr 27, 2022 at 11:32:22AM -0700, Darrick J. Wong wrote:
> > On Tue, Apr 26, 2022 at 09:17:14AM +1000, Dave Chinner wrote:
> > > Hi folks,
> > >
> > > I just pushed out a new for-next branch for the XFS tree. It
> > > contains:
> > >
> > > - pending fixes for 5.18
> > > - various miscellaneous fixes
> > > - xlog_write() rework
> > > - conversions to unsigned for trace_printk flags
> > > - large on-disk extent counts
> > >
> > > This all passes my local regression testing, though further smoke
> > > testing in different environments would be appreaciated.
> > >
> > > I haven't pulled in fixes from late last week yet - I'll work
> > > through those in the next couple of days to get them into the tree
> > > as well.
> > >
> > > If I've missed anything you were expecting to see in this update,
> > > let me know and I'll get them sorted for the next update.
> >
> > Hmm. I saw the following crash on an arm64 VM with 64k page size and an
> > 8k blocksize:
>
> Ok, that's a whacky config I can't directly test here.
Heh. :/
> > run fstests xfs/502 at 2022-04-26 20:54:15
> > spectre-v4 mitigation disabled by command-line option
> > XFS (sda2): Mounting V5 Filesystem
> > XFS (sda2): Ending clean mount
> > XFS (sda3): Mounting V5 Filesystem
> > XFS (sda3): Ending clean mount
> > XFS (sda3): Quotacheck needed: Please wait.
> > XFS (sda3): Quotacheck: Done.
> > XFS (sda3): Injecting error (false) at file fs/xfs/xfs_inode.c, line 1876, on filesystem "sda3"
> > XFS (sda3): Injecting error (false) at file fs/xfs/xfs_inode.c, line 1876, on filesystem "sda3"
> > XFS (sda3): Injecting error (false) at file fs/xfs/xfs_inode.c, line 1876, on filesystem "sda3"
> > XFS (sda3): Injecting error (false) at file fs/xfs/xfs_inode.c, line 1876, on filesystem "sda3"
> > XFS (sda3): Injecting error (false) at file fs/xfs/xfs_inode.c, line 1876, on filesystem "sda3"
> > XFS (sda3): Injecting error (false) at file fs/xfs/xfs_inode.c, line 1876, on filesystem "sda3"
> > XFS (sda3): Injecting error (false) at file fs/xfs/xfs_inode.c, line 1876, on filesystem "sda3"
> > XFS (sda3): Injecting error (false) at file fs/xfs/xfs_inode.c, line 1876, on filesystem "sda3"
> > XFS (sda3): Injecting error (false) at file fs/xfs/xfs_inode.c, line 1876, on filesystem "sda3"
> > XFS (sda3): Injecting error (false) at file fs/xfs/xfs_inode.c, line 1876, on filesystem "sda3"
> > XFS: Assertion failed: IS_ALIGNED((unsigned long)lv->lv_buf, sizeof(uint64_t)), file: fs/xfs/xfs_log_cil.c, line: 431
>
> Huh. That implies that memory allocation for the shadow buffer
> wasn't 8 byte aligned because:
>
> lv->lv_buf = (char *)lv +
> xlog_cil_iovec_space(lv->lv_niovecs);
>
> And:
>
> static inline int
> xlog_cil_iovec_space(
> uint niovecs)
> {
> return round_up((sizeof(struct xfs_log_vec) +
> niovecs * sizeof(struct xfs_log_iovec)),
> sizeof(uint64_t));
> }
>
> So the length returned by xlog_cil_iovec_space() is always a
> multiple of 8 bytes. Hence the only way lv->lv_buf would not be
> aligned to 8 bytes is if the original allocation was not aligned.
>
> That buffer comes from:
>
> static inline void *
> xlog_cil_kvmalloc(
> size_t buf_size)
> {
> gfp_t flags = GFP_KERNEL;
> void *p;
>
> flags &= ~__GFP_DIRECT_RECLAIM;
> flags |= __GFP_NOWARN | __GFP_NORETRY;
> do {
> p = kmalloc(buf_size, flags);
> if (!p)
> p = vmalloc(buf_size);
> } while (!p);
>
> return p;
> }
>
> vmalloc() guarantees alignment of the returned buffer, so this must
> have come from a heap allocation via kmalloc() and that must have
> returned something that is not 8 byte aligned...
>
> Oh, after the XFS assert, a second oops happened, this time in
> kmalloc():
>
> > Unable to handle kernel paging request at virtual address 9ac7c01eb06874e8
> > Mem abort info:
> > ESR = 0x96000004
> > EC = 0x25: DABT (current EL), IL = 32 bits
> > SET = 0, FnV = 0
> > EA = 0, S1PTW = 0
> > FSC = 0x04: level 0 translation fault
> > Data abort info:
> > ISV = 0, ISS = 0x00000004
> > CM = 0, WnR = 0
> > [9ac7c01eb06874e8] address between user and kernel address ranges
> > Internal error: Oops: 96000004 [#1] PREEMPT SMP
> > Dumping ftrace buffer:
> > (ftrace buffer empty)
> > Modules linked in: xfs dm_delay dm_zero dm_flakey dm_snapshot dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio libcrc32c xt_REDIRECT iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 rputh_rpcgss xt_tcpudp ip_set_hash_ip ip_set_hash_net xt_set ip_set_hash_mac ip6table_filter ip6_tables bfq iptable_filter crct10dif_ce sch_fq_codel efivarfs ip_tables x_tables overlay nfsv4 [last unloaded: xfs]
> > CPU: 1 PID: 3168300 Comm: t_open_tmpfiles Tainted: G W 5.17.0-xfsa #5.17.0 0288cc936a4dc1878aaf6a4c6fa6235f949bf1e9
> > Hardware name: QEMU KVM Virtual Machine, BIOS 1.5.1 06/16/2021
> > pstate: a0401005 (NzCv daif +PAN -UAO -TCO -DIT +SSBS BTYPE=--)
> > pc : __kmalloc+0x120/0x3f0
> > lr : __kmalloc+0xe8/0x3f0
> > sp : fffffe00213ef840
> > x29: fffffe00213ef840 x28: 0000000000000002 x27: fffffe0008fe6000
> > x26: fffffc00e8740000 x25: fffffe00090e9000 x24: fffffe0001523c44
> > x23: 0000000000000150 x22: 00000000000128c0 x21: 0000000000000000
> > x20: 9ac7c01eb06874e8 x19: fffffc00e0010400 x18: 0000000000000000
> > x17: 0000000000000000 x16: 0000000000000000 x15: 0000000000000000
> > x14: 0000000000000000 x13: 0000f0ffffffffff x12: 0000000000000040
> > x11: fffffc0133b5c678 x10: fffffc0133b5c67a x9 : fffffe0008294814
> > x8 : 0000000000000001 x7 : fffffe01f68d0000 x6 : 8d4857517c5941da
> > x5 : 00000000019ff6ec x4 : 0000000000000100 x3 : 0000000000000000
> > x2 : e87468b01ec0c79a x1 : 0000000003a724a9 x0 : 9ac7c01eb06873e8
> > Call trace:
> > __kmalloc+0x120/0x3f0
> > xlog_cil_commit+0x144/0x9b0 [xfs afa05b7bcc3355e0f6d54fdf0bba6e6ddd5eafff]
>
> This smells like heap corruption has occurred and the CIL commit
> code was the unfortunate victim that tripped over it first. That
> would explain the unaligned object being returned from kmalloc().
>
> Did you have any memory debugging options turned on?
Nope. I'll run them again tonight with KASAN enabled and see if that
shakes anything out.
--D
> Cheers,
>
> Dave.
> --
> Dave Chinner
> david@xxxxxxxxxxxxx
