On Wed, Nov 14, 2018 at 12:42:49PM +0100, Michael Arndt wrote:
> Hello XFS Gurus,
>
> Problem: /bin/rm extremely slow on a major xfs (SSD based) HPC Storage
> slow == 90 seconds for unlink of an empty file without any extents
> strace says: time completely used for unlink call
>
> Question; Is there any issue resolution ?
>
> Information re XFS Version and OS at end of this Post
>
> Example of issue:
>
>
> [root@atgrzsl3150 DOM_0]# xfs_bmap -a .AN_720.0000122.fl3step_0.lock
>
> .AN_720.0000122.fl3step_0.lock: no extents
> [root@atgrzsl3150 DOM_0]# ls -laFtr .AN_720.0000122.fl3step_0.lock
>
> -rw-rw-r-- 1 user group 0 Oct 22 14:14 .AN_720.0000122.fl3step_0.lock
>
>
> strace -T -tt /bin/rm .AN_720.0000122.fl3step_0.lock
>
>
> 1:41:11.621005 execve("/bin/rm", ["/bin/rm", ".AN_720.0000122.fl3step_0.lock"], [/* 31 vars */]) = 0 <0.000169>11:41:11.621312 brk(NULL) = 0x6f5000 <0.000023>11:41:11.621378 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f4d96017000 <0.000058>
> …….
> 11:41:11.622485 newfstatat(AT_FDCWD, ".AN_720.0000122.fl3step_0.lock", {st_mode=S_IFREG|0664, st_size=0, ...}, AT_SYMLINK_NOFOLLOW) = 0 <0.000009>
> 11:41:11.622522 geteuid() = 0 <0.000009>
> -> 11:41:11.622546 unlinkat(AT_FDCWD, ".AN_720.0000122.fl3step_0.lock", 0) = 0 <89.612833>
> -> 11:42:41.235428 lseek(0, 0, SEEK_CUR) = -1 ESPIPE (Illegal seek) <0.000065>
> 11:42:41.235548 close(0) = 0 <0.000052>
> 11:42:41.235689 close(1) = 0 <0.000011>
> 11:42:41.235738 close(2) = 0 <0.000055>
> 11:42:41.235830 exit_group(0) = ?
> 11:42:41.235941 +++ exited with 0 +++
>
It might be useful to do something like:
trace-cmd record -e xfs:* <rm command>
... and either put the resulting trace.dat somewhere where it can be
downloaded or if not too large, run 'trace-cmd report' and copy the text
into a mail (without reformatting it).
Brian
> xfs_info /dev/mapper/vg_calc2-calc2
> meta-data=/dev/mapper/vg_calc2-calc2 isize=512 agcount=50, agsize=268435448 blks
> = sectsz=512 attr=2, projid32bit=1
> = crc=1 finobt=0 spinodes=0
> data = bsize=4096 blocks=13421711360, imaxpct=20
> = sunit=8 swidth=40 blks
> naming =version 2 bsize=4096 ascii-ci=0 ftype=1
> log =internal bsize=4096 blocks=521728, version=2
> = sectsz=512 sunit=8 blks, lazy-count=1
> realtime =none extsz=4096 blocks=0, rtextents=0
>
> Issue on:
>
> xfsprogs-4.5.0-18.el7.x86_64
> xfsdump-3.1.7-1.el7.x86_64
> Red Hat Enterprise Linux Server release 7.4 (Maipo)
> df -kh .
> Filesystem Size Used Avail Use% Mounted on
> /dev/mapper/vg_calc2 50T 20T 31T 40% /calc2
>
> Layers:
>
> SSD based commercial Storage exports many small LUN’s -> LUN#s striped via LVM2 for speed, xfs with default opts on top of LVM
> Currently no discard Option for mount and no fstrim manually called
>
> Mount Options used
> /dev/mapper/vg_calc2-calc2 /calc2 xfs noatime,delaylog,nobarrier,nodiratime,logbsize=256k,logbufs=8 0 0
>
> thanks for any tip / hint / question
> Micha
>
