We run the systemtap script in the background. The kmem_alloc message appeared in the /var/log/messages. The call stack is captured by the systemtap is follow.
0xffffffffa02a0030 : xfs_err+0x0/0x90 [xfs]
0xffffffffa02a853d : kmem_alloc+0xbd/0xf0 [xfs]
0xffffffffa02eeb6e : xfs_log_commit_cil+0x39e/0x4c0 [xfs]
0xffffffffa02a7d3c : xfs_trans_commit+0x11c/0x240 [xfs]
0xffffffffa02dd353 : xfs_remove+0x343/0x380 [xfs]
0xffffffffa029d93a : xfs_vn_unlink+0x5a/0xc0 [xfs]
0xffffffff811bde51
0xffffffff811be056 (inexact)
0xffffffff811b166e (inexact)
0xffffffff810823cc (inexact)
0xffffffff811c11b6 (inexact)
0xffffffff815f3219 (inexact)
Nov 5 18:13:21 csx-ceph1-003 kernel: XFS: possible memory allocation deadlock in kmem_alloc (mode:0x8250)
#! /usr/bin/env stap
probe module("xfs").function("xfs_err").call
{
print_backtrace();
}
# nohup stap -o /var/systemtap/kmem_alloc_bt.out backtrace.stp &
Redhat support case is https://access.redhat.com/support/cases/#/case/01500239
Thanks,
-Al
-----Original Message-----
From: Dave Chinner [mailto:david@xxxxxxxxxxxxx]
Sent: Tuesday, October 13, 2015 1:26 AM
To: Al Lau (alau2)
Cc: xfs@xxxxxxxxxxx
Subject: Re: mkfs.xfs -n size=65536
On Tue, Oct 13, 2015 at 07:28:48AM +0000, Al Lau (alau2) wrote:
> Are the xfs_db and filefrag the utilities to use to determine file fragmentation?
>
> # df -k /var/kmem_alloc
> Filesystem 1K-blocks Used Available Use% Mounted on
> /dev/sdf1 3905109820 3359385616 545724204 87% /var/kmem_alloc
> # xfs_db -r -c frag /dev/sdf1
> actual 438970, ideal 388168, fragmentation factor 11.57%
http://xfs.org/index.php/XFS_FAQ#Q:_The_xfs_db_.22frag.22_command_says_I.27m_over_50.25._Is_that_bad.3F
> # ls -l fragmented
> -rw-r--r--. 1 root root 3360239878657 Oct 13 07:25 fragmented #
> filefrag fragmented
> fragmented: 385533 extents found
That's a lot of extents, but for a 3TB sparse file that is being written in random 4k blocks, that's expected and there's little you can do about it. Preallocation of the file or use of extent size hints will reduce physical fragmentation, but you only want to use those if the file will eventually become non-sparse and sequential read IO performance is required...
i.e. the definition of "fragmented" really depends on the application, IO patterns and whether the current physical layout is acheiving the desired performance attributes of the file in question....
Cheers,
Dave.
--
Dave Chinner
david@xxxxxxxxxxxxx
_______________________________________________
xfs mailing list
xfs@xxxxxxxxxxx
http://oss.sgi.com/mailman/listinfo/xfs
