On Sun, Dec 04, 2016 at 02:07:18PM -0800, Cyril Peponnet wrote: > Hi here is the details. The issue is on the scratch RAID array > (used to store kvm snapshots). The other raid array is fine (no > snapshot storage). How do you know that? There's no indication which filesystem is generating the warnings.... FWIW, the only gluster snapshot proposal that I was aware of was this one: https://lists.gnu.org/archive/html/gluster-devel/2013-08/msg00004.html Which used LVM snapshots to take snapshots of the entire brick. I don't see any LVM in your config, so I'm not sure what snapshot implementation you are using here. What are you using to take the snapshots of your VM image files? Are you actually using the qemu qcow2 snapshot functionality rather than anything native to gluster? Also, can you attach the 'xfs_bmap -vp' output of some of these image files and their snapshots? > MemTotal: 65699268 kB 64GB RAM... > MemFree: 2058304 kB > MemAvailable: 62753028 kB > Buffers: 12 kB > Cached: 57664044 kB 56GB of cached file data. If you're getting high order allocation failures (which I suspect is the problem) then this is a memory fragmentation problem more than anything. > ---------------------------------------------------------------- > DG/VD TYPE State Access Consist Cache Cac sCC Size Name > ---------------------------------------------------------------- > 0/0 RAID0 Optl RW Yes RAWBC - ON 7.275 TB scratch > ---------------------------------------------------------------- > > Cac=CacheCade|Rec=Recovery|OfLn=OffLine|Pdgd=Partially Degraded|dgrd=Degraded > Optl=Optimal|RO=Read Only|RW=Read Write|HD=Hidden|B=Blocked|Consist=Consistent| > R=Read Ahead Always|NR=No Read Ahead|WB=WriteBack| > AWB=Always WriteBack|WT=WriteThrough|C=Cached IO|D=Direct IO|sCC=Scheduled IIRC, AWB means that if the cache goes into degraded/offline mode, you're vulnerable to corruption/loss on power failure... > xfs_info /export/raid/scratch/ > meta-data=/dev/sdb isize=256 agcount=32, agsize=61030368 blks > = sectsz=512 attr=2, projid32bit=1 > = crc=0 > data = bsize=4096 blocks=1952971776, imaxpct=5 > = sunit=32 swidth=128 blks > naming =version 2 bsize=4096 ascii-ci=0 ftype=0 > log =internal bsize=4096 blocks=521728, version=2 > = sectsz=512 sunit=32 blks, lazy-count=1 > realtime =none extsz=4096 blocks=0, rtextents=0 Nothing unusual there. > Nothing relevant in dmesg except several occurences of the following. > > [7649583.386283] XFS: possible memory allocation deadlock in kmem_alloc (mode:0x250) > [7649585.370830] XFS: possible memory allocation deadlock in kmem_alloc (mode:0x250) > [7649587.241290] XFS: possible memory allocation deadlock in kmem_alloc (mode:0x250) > [7649589.243881] XFS: possible memory allocation deadlock in kmem_alloc (mode:0x250) Ah, the kernel is old enough it doesn't have the added reporting to tell us what the process and size of the allocation being requested is. Hmm - it's an xfs_err() call, thet means we should be able to get a stack trace out of the kernel if we turn the error level up to 11. # echo 11 > /proc/sys/fs/xfs/error_level And wait for it to happena again. that should give a stack trace telling us where the issue is. Cheers, Dave. -- Dave Chinner david@xxxxxxxxxxxxx -- To unsubscribe from this list: send the line "unsubscribe linux-xfs" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
