Hello *,
Problem: /bin/rm extremely slow on a major xfs (SSD based) HPC Storage
slow == 90 seconds for unlink of an empty file without any extents
strace says: time completely used for unlink call
Question; Is there any issue resolution ?
Information re XFS Version and OS at end of this Post
Example of issue:
[root@atgrzsl3150 DOM_0]# xfs_bmap -a .AN_720.0000122.fl3step_0.lock
.AN_720.0000122.fl3step_0.lock: no extents
[root@atgrzsl3150 DOM_0]# ls -laFtr .AN_720.0000122.fl3step_0.lock
-rw-rw-r-- 1 user group 0 Oct 22 14:14 .AN_720.0000122.fl3step_0.lock
strace -T -tt /bin/rm .AN_720.0000122.fl3step_0.lock
11:41:11.621005 execve("/bin/rm", ["/bin/rm", ".AN_720.0000122.fl3step_0.lock"], [/* 31 vars */]) = 0 <0.000169>
11:41:11.621312 brk(NULL) = 0x6f5000 <0.000023>
11:41:11.621378 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f4d96017000 <0.000058>
…….
11:41:11.622485 newfstatat(AT_FDCWD, ".AN_720.0000122.fl3step_0.lock", {st_mode=S_IFREG|0664, st_size=0, ...}, AT_SYMLINK_NOFOLLOW) = 0 <0.000009>
11:41:11.622522 geteuid() = 0 <0.000009>
-> 11:41:11.622546 unlinkat(AT_FDCWD, ".AN_720.0000122.fl3step_0.lock", 0) = 0 <89.612833>
-> 11:42:41.235428 lseek(0, 0, SEEK_CUR) = -1 ESPIPE (Illegal seek) <0.000065>
11:42:41.235548 close(0) = 0 <0.000052>
11:42:41.235689 close(1) = 0 <0.000011>
11:42:41.235738 close(2) = 0 <0.000055>
11:42:41.235830 exit_group(0) = ?
11:42:41.235941 +++ exited with 0 +++
xfs_info /dev/mapper/vg_calc2-calc2
meta-data=/dev/mapper/vg_calc2-calc2 isize=512 agcount=50, agsize=268435448 blks
= sectsz=512 attr=2, projid32bit=1
= crc=1 finobt=0 spinodes=0
data = bsize=4096 blocks=13421711360, imaxpct=20
= sunit=8 swidth=40 blks
naming =version 2 bsize=4096 ascii-ci=0 ftype=1
log =internal bsize=4096 blocks=521728, version=2
= sectsz=512 sunit=8 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
Issue on:
xfsprogs-4.5.0-18.el7.x86_64
xfsdump-3.1.7-1.el7.x86_64
Red Hat Enterprise Linux Server release 7.4 (Maipo)
df -kh .
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg_calc2 50T 20T 31T 40% /calc2
Layers:
SSD based commercial Storage exports many small LUN’s -> LUN#s striped via LVM2 for speed, xfs with default opts on top of LVM
Currently no discard Option for mount and no fstrim manually called
Mount Options used
/dev/mapper/vg_calc2-calc2 /calc2 xfs noatime,delaylog,nobarrier,nodiratime,logbsize=256k,logbufs=8 0 0
thanks for any tip / hint / question
Micha
