Fwd: xfs remove / unlink extremely slow ?

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hello XFS Gurus,

Problem: /bin/rm extremely slow on a major xfs (SSD based) HPC Storage
slow == 90 seconds for unlink of an empty file without any extents
strace says: time completely used for unlink call

Question; Is there any issue resolution ?

Information re XFS Version and OS at end of this Post

Example of issue:


[root@atgrzsl3150 DOM_0]# xfs_bmap -a .AN_720.0000122.fl3step_0.lock 

.AN_720.0000122.fl3step_0.lock: no extents
[root@atgrzsl3150 DOM_0]# ls -laFtr .AN_720.0000122.fl3step_0.lock 

-rw-rw-r-- 1 user group 0 Oct 22 14:14 .AN_720.0000122.fl3step_0.lock


strace -T -tt /bin/rm .AN_720.0000122.fl3step_0.lock 


1:41:11.621005 execve("/bin/rm", ["/bin/rm", ".AN_720.0000122.fl3step_0.lock"], [/* 31 vars */]) = 0 <0.000169>11:41:11.621312 brk(NULL)               = 0x6f5000 <0.000023>11:41:11.621378 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f4d96017000 <0.000058>
…….
11:41:11.622485 newfstatat(AT_FDCWD, ".AN_720.0000122.fl3step_0.lock", {st_mode=S_IFREG|0664, st_size=0, ...}, AT_SYMLINK_NOFOLLOW) = 0 <0.000009>
11:41:11.622522 geteuid()               = 0 <0.000009>
-> 11:41:11.622546 unlinkat(AT_FDCWD, ".AN_720.0000122.fl3step_0.lock", 0) = 0 <89.612833>
-> 11:42:41.235428 lseek(0, 0, SEEK_CUR)   = -1 ESPIPE (Illegal seek) <0.000065>
11:42:41.235548 close(0)                = 0 <0.000052>
11:42:41.235689 close(1)                = 0 <0.000011>
11:42:41.235738 close(2)                = 0 <0.000055>
11:42:41.235830 exit_group(0)           = ?
11:42:41.235941 +++ exited with 0 +++

xfs_info /dev/mapper/vg_calc2-calc2
meta-data=/dev/mapper/vg_calc2-calc2 isize=512    agcount=50, agsize=268435448 blks
        =                       sectsz=512   attr=2, projid32bit=1
        =                       crc=1        finobt=0 spinodes=0
data     =                       bsize=4096   blocks=13421711360, imaxpct=20
        =                       sunit=8      swidth=40 blks
naming   =version 2              bsize=4096   ascii-ci=0 ftype=1
log      =internal               bsize=4096   blocks=521728, version=2
        =                       sectsz=512   sunit=8 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

Issue on:

xfsprogs-4.5.0-18.el7.x86_64
xfsdump-3.1.7-1.el7.x86_64
Red Hat Enterprise Linux Server release 7.4 (Maipo)
df -kh .
Filesystem                  Size  Used Avail Use% Mounted on
/dev/mapper/vg_calc2        50T   20T   31T  40% /calc2

Layers: 

SSD based commercial Storage exports many small LUN’s -> LUN#s striped via LVM2 for speed, xfs with default opts on top of LVM
Currently no discard Option for mount and no fstrim manually called

Mount Options used
/dev/mapper/vg_calc2-calc2 /calc2 xfs noatime,delaylog,nobarrier,nodiratime,logbsize=256k,logbufs=8 0 0 

thanks for any tip / hint / question
Micha




[Index of Archives]     [XFS Filesystem Development (older mail)]     [Linux Filesystem Development]     [Linux Audio Users]     [Yosemite Trails]     [Linux Kernel]     [Linux RAID]     [Linux SCSI]


  Powered by Linux