xfs and swift

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



I've recently found the performance our development swift system is degrading over time as the number of objects/files increases.  This is a relatively small system, each server has 3 400GB disks.  The system I'm currently looking at has about 70GB tied up in slabs alone, close to 55GB in xfs inodes and ili, and about 2GB free.  The kernel is 3.14.57-1-amd64-hlinux.

Here's the way the filesystems are mounted:

/dev/sdb1 on /srv/node/disk0 type xfs (rw,noatime,nodiratime,attr2,nobarrier,inode64,logbufs=8,logbsize=256k,sunit=512,swidth=1536,noquota)

I can do about 2000 1K file creates/sec when running 2 minute PUT tests at 100 threads.  If I repeat that tests for multiple hours, I see the number of IOPS steadily decreasing to about 770 and the very next run it drops to 260 and continues to fall from there.  This happens at about 12M files.

The directory structure is 2 tiered, with 1000 directories per tier so we can have about 1M of them, though they don't currently all exist.

I've written a collectl plugin that lets me watch many of the xfs stats in real-time and also have a test script that exercises the swift PUT code directly and so eliminates all the inter-node communications.  This script also allows me to write to the existing swift directories as well as redirect to an empty structure so mimics clean environment with no existing subdirectories.

I'm attaching some xfs stats during the run and hope they're readable.  These values are in operations/sec and each line is 1 second's worth of data.  The first set of numbers is on the clean directory and the second on the existing 12M file one.  At the bottom of these stats are also the xfs slab allocations as reported by collectl.  I can also watch these during a test and can see the number of inode and ilo objects steadily grow at about 1K/sec, which is curious since I'm only creating about 300.

If there is anything else I can provide just let me know.

I don't fully understand all the xfs stats but what does jump out at me is the XFS read/write ops have increased by a factor of about 5 when the system is slower.  Right now the collectl plugin is not something I've released, but if there is interest and someone would like to help me present the data in a more organized/meaningful manner just let me know.

if there are any tuning suggestions I'm more than happy to try them out.

-mark
>>> Fast <<<
#<--XFS Ops--><-----------XFS Logging----------><------Extents------><------DirOps-------><----Trans---><----Xstrat---><-------AttrOps-----><--------------INodes-------------->
# Write Reads  Writes WrtKBs NoRoom Force Sleep  ExtA BlkA ExtF ExtF  Look Cre8 Remv Gdnt  Sync Asyn Empt  Quick Split  Gets Sets Rmov List  Atpt  Hit Miss Recy  Dup Recl Chgd 
     53   599       4   1024      0     4     4     3   65   10   70   155    6   14  284     0   75    1      2     0   275    1    0   24     0  149    5    0    0    0    1
     92   836      16   4096      0    16    16    24  117    8   98   200   58   18  272     0  181    1      3     0   235   11    0   10     0  152   39    0    0   15    1
    370   732     295  75520      0   295   295   592  685    6   96  1599 1442  293  829     0 3527    1      9     0   244  290    0   13     0  153 1144    0    0    0    1
    383   837     284  72704      0   284   285   559  683   10  130  1532 1352  284  816     0 3343    0      4     0   236  276    0   10     0  155 1073    0    0    9    0
    341   734     289  73984      0   289   289   583  690    8   68  1574 1393  297  860     0 3472    3      6     0   291  288    0   30     0  143 1105    0    0    0    3
    342   812     291  74496      0   291   291   583  720    6   66  1574 1376  294  840     0 3439    2      2     0   261  289    0   19     0  144 1087    0    0    0    2
    427   415     301  77056      0   301   302   598  843   14  164  1613 1391  305  870     0 3531    1      5     0   279  292    0   26     0  163 1090    0    0    0    1
    401   832     302  77312      0   302   303   598  797   10  130  1604 1390  303  862     0 3522    1      4     0   244  295    0   13     0  148 1093    0    0   90    1
    349   384     275  70400      0   275   275   549  717   10  100  1480 1258  281  814     0 3224    1      4     0   251  270    0   15     0  146  985    0    0    0    1
     79   432       6   1536      0     6     6     9  102    6   96   158    3    3  250     0   47    0      9     0   248    0    0   14     0  156    2    0    0    0    0
     54   253       4   1024      0     4     4     2   64    4   64   157    2    2  274     0   23    0      2     0   284    0    0   26     0  156    1    0    0    0    0

>>> Slow <<<

#<--XFS Ops--><-----------XFS Logging----------><------Extents------><------DirOps-------><----Trans---><----Xstrat---><-------AttrOps-----><--------------INodes-------------->
# Write Reads  Writes WrtKBs NoRoom Force Sleep  ExtA BlkA ExtF ExtF  Look Cre8 Remv Gdnt  Sync Asyn Empt  Quick Split  Gets Sets Rmov List  Atpt  Hit Miss Recy  Dup Recl Chgd 
      0    61       0      0      0     0     0     0    0    0    0   132    0    0  218     0    0    0      0     0   213    0    0    0     0  126    6    0    0    0    0
     59   115      11   2816      0    11    11    16   78    4   65   160   33    9  230     0  104    0      2     0   210    7    0    0     0  128   28    0    0    0    0
   1384  1263     272  69632      0   272   272   423 1998   92 1443   875  872  227  576     0 2639    0     45     0   210  182    0    0     0  153  675    0    0    4    0
   1604  1503     294  75264      0   294   294   438 2201  106 1696   907  890  241  590     0 2772    0     53     0   210  188    0    0     0  151  681    0    0    9    0
   1638  2255     309  79104      0   309   307   460 2314  114 1734   946  934  260  632     0 2942    0     54     0   237  199    0    0     0  193  678    0    0    0    0
   1678  2298     337  86272      0   338   330   486 2326  128 1779  1031  987  291  712     0 3168    0     55     0   284  220    0    4     0  189  712    0    0    0    0
   1578  2423     333  85248      0   332   325   492 2268  118 1649  1041  991  289  714     0 3153    0     51     0   270  222    0    0     0  200  700    0    0    0    0
   1040  1861     239  61184      0   241   231   353 1496   82 1072   774  718  212  588     0 2272    0     33     0   264  164    0    0     0  153  524    0    0 1730    0
   1709  2361     336  86016      0   340   331   485 2401  128 1810  1029  969  291  706     0 3137    0     56     0   278  216    0    4     0  174  722    0    0 4751    0
   1616  2063     325  83200      0   326   321   485 2278  114 1707  1038  973  274  674     1 3067    0     53     0   240  214    0    3     0  173  731    0    0    0    0
   1599  1557     312  79872      0   312   313   482 2274  104 1664  1048  962  260  684     0 2980    0     52     0   224  208    0    7     0  219  681    0    0    0    0
   1114  1312     229  58624      0   229   230   356 1577   72 1152   817  709  192  552     0 2188    0     36     0   216  157    0    3     0  165  524    0    0 3570    0
   1066  1185     175  44800      0   176   176   249 1440   72 1153   585  466  141  440     0 1577    0     36     0   214  104    0    2     0  155  339    0    0 4497    0
     54   487       6   1536      0     6     6     2   64    4   64   137    2    2  238     0   24    0      2     0   216    0    0    3     0  135    2    0    0 2664    0
      0   590       0      0      0     0     0     0    0    0    0   136    0    0  224     0    0    0      0     0   210    0    0    0     0  136    0    0    0    0    0
     54   514       4   1024      0     4     4     2   64    4   64   142    2    2  248     0   24    0      2     0   214    0    0    2     0  140    2    0    0    0    0

>>> slabs <<<


stack@helion-cp1-swobj0001-mgmt:~$ sudo collectl -sY -i:1 -c1 --slabfilt xfs
waiting for 1 second sample...

# SLAB DETAIL
#                           <-----------Objects----------><---------Slab Allocation------><---Change-->
#Name                       InUse   Bytes   Alloc   Bytes   InUse   Bytes   Total   Bytes   Diff    Pct
xfs_btree_cur                1872  389376    1872  389376      48  393216      48  393216      0    0.0
xfs_da_state                 1584  772992    1584  772992      48  786432      48  786432      0    0.0
xfs_dqtrx                       0       0       0       0       0       0       0       0      0    0.0
xfs_efd_item                 4360 1744000    4760 1904000     119 1949696     119 1949696      0    0.0
xfs_icr                         0       0       0       0       0       0       0       0      0    0.0
xfs_ili                    48127K   6976M  48197K   6986M  909380   7104M  909380   7104M      0    0.0
xfs_inode                  48210K  47080M  48244K  47113M   1507K  47123M   1507K  47123M      0    0.0
_______________________________________________
xfs mailing list
xfs@xxxxxxxxxxx
http://oss.sgi.com/mailman/listinfo/xfs

[Index of Archives]     [Linux XFS Devel]     [Linux Filesystem Development]     [Filesystem Testing]     [Linux USB Devel]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]

  Powered by Linux