I've recently found the performance our development swift system is degrading over time as the number of objects/files increases. This is a relatively small system, each server has 3 400GB disks. The system I'm currently looking at has about 70GB tied up in slabs alone, close to 55GB in xfs inodes and ili, and about 2GB free. The kernel is 3.14.57-1-amd64-hlinux.
Here's the way the filesystems are mounted:
/dev/sdb1 on /srv/node/disk0 type xfs (rw,noatime,nodiratime,attr2,nobarrier,inode64,logbufs=8,logbsize=256k,sunit=512,swidth=1536,noquota)
I can do about 2000 1K file creates/sec when running 2 minute PUT tests at 100 threads. If I repeat that tests for multiple hours, I see the number of IOPS steadily decreasing to about 770 and the very next run it drops to 260 and continues to fall from there. This happens at about 12M files.
The directory structure is 2 tiered, with 1000 directories per tier so we can have about 1M of them, though they don't currently all exist.
I've written a collectl plugin that lets me watch many of the xfs stats in real-time and also have a test script that exercises the swift PUT code directly and so eliminates all the inter-node communications. This script also allows me to write to the existing swift directories as well as redirect to an empty structure so mimics clean environment with no existing subdirectories.
I'm attaching some xfs stats during the run and hope they're readable. These values are in operations/sec and each line is 1 second's worth of data. The first set of numbers is on the clean directory and the second on the existing 12M file one. At the bottom of these stats are also the xfs slab allocations as reported by collectl. I can also watch these during a test and can see the number of inode and ilo objects steadily grow at about 1K/sec, which is curious since I'm only creating about 300.
If there is anything else I can provide just let me know.
I don't fully understand all the xfs stats but what does jump out at me is the XFS read/write ops have increased by a factor of about 5 when the system is slower. Right now the collectl plugin is not something I've released, but if there is interest and someone would like to help me present the data in a more organized/meaningful manner just let me know.
if there are any tuning suggestions I'm more than happy to try them out.
-mark
>>> Fast <<<
#<--XFS Ops--><-----------XFS Logging----------><------Extents------><------DirOps-------><----Trans---><----Xstrat---><-------AttrOps-----><--------------INodes-------------->
# Write Reads Writes WrtKBs NoRoom Force Sleep ExtA BlkA ExtF ExtF Look Cre8 Remv Gdnt Sync Asyn Empt Quick Split Gets Sets Rmov List Atpt Hit Miss Recy Dup Recl Chgd
53 599 4 1024 0 4 4 3 65 10 70 155 6 14 284 0 75 1 2 0 275 1 0 24 0 149 5 0 0 0 1
92 836 16 4096 0 16 16 24 117 8 98 200 58 18 272 0 181 1 3 0 235 11 0 10 0 152 39 0 0 15 1
370 732 295 75520 0 295 295 592 685 6 96 1599 1442 293 829 0 3527 1 9 0 244 290 0 13 0 153 1144 0 0 0 1
383 837 284 72704 0 284 285 559 683 10 130 1532 1352 284 816 0 3343 0 4 0 236 276 0 10 0 155 1073 0 0 9 0
341 734 289 73984 0 289 289 583 690 8 68 1574 1393 297 860 0 3472 3 6 0 291 288 0 30 0 143 1105 0 0 0 3
342 812 291 74496 0 291 291 583 720 6 66 1574 1376 294 840 0 3439 2 2 0 261 289 0 19 0 144 1087 0 0 0 2
427 415 301 77056 0 301 302 598 843 14 164 1613 1391 305 870 0 3531 1 5 0 279 292 0 26 0 163 1090 0 0 0 1
401 832 302 77312 0 302 303 598 797 10 130 1604 1390 303 862 0 3522 1 4 0 244 295 0 13 0 148 1093 0 0 90 1
349 384 275 70400 0 275 275 549 717 10 100 1480 1258 281 814 0 3224 1 4 0 251 270 0 15 0 146 985 0 0 0 1
79 432 6 1536 0 6 6 9 102 6 96 158 3 3 250 0 47 0 9 0 248 0 0 14 0 156 2 0 0 0 0
54 253 4 1024 0 4 4 2 64 4 64 157 2 2 274 0 23 0 2 0 284 0 0 26 0 156 1 0 0 0 0
>>> Slow <<<
#<--XFS Ops--><-----------XFS Logging----------><------Extents------><------DirOps-------><----Trans---><----Xstrat---><-------AttrOps-----><--------------INodes-------------->
# Write Reads Writes WrtKBs NoRoom Force Sleep ExtA BlkA ExtF ExtF Look Cre8 Remv Gdnt Sync Asyn Empt Quick Split Gets Sets Rmov List Atpt Hit Miss Recy Dup Recl Chgd
0 61 0 0 0 0 0 0 0 0 0 132 0 0 218 0 0 0 0 0 213 0 0 0 0 126 6 0 0 0 0
59 115 11 2816 0 11 11 16 78 4 65 160 33 9 230 0 104 0 2 0 210 7 0 0 0 128 28 0 0 0 0
1384 1263 272 69632 0 272 272 423 1998 92 1443 875 872 227 576 0 2639 0 45 0 210 182 0 0 0 153 675 0 0 4 0
1604 1503 294 75264 0 294 294 438 2201 106 1696 907 890 241 590 0 2772 0 53 0 210 188 0 0 0 151 681 0 0 9 0
1638 2255 309 79104 0 309 307 460 2314 114 1734 946 934 260 632 0 2942 0 54 0 237 199 0 0 0 193 678 0 0 0 0
1678 2298 337 86272 0 338 330 486 2326 128 1779 1031 987 291 712 0 3168 0 55 0 284 220 0 4 0 189 712 0 0 0 0
1578 2423 333 85248 0 332 325 492 2268 118 1649 1041 991 289 714 0 3153 0 51 0 270 222 0 0 0 200 700 0 0 0 0
1040 1861 239 61184 0 241 231 353 1496 82 1072 774 718 212 588 0 2272 0 33 0 264 164 0 0 0 153 524 0 0 1730 0
1709 2361 336 86016 0 340 331 485 2401 128 1810 1029 969 291 706 0 3137 0 56 0 278 216 0 4 0 174 722 0 0 4751 0
1616 2063 325 83200 0 326 321 485 2278 114 1707 1038 973 274 674 1 3067 0 53 0 240 214 0 3 0 173 731 0 0 0 0
1599 1557 312 79872 0 312 313 482 2274 104 1664 1048 962 260 684 0 2980 0 52 0 224 208 0 7 0 219 681 0 0 0 0
1114 1312 229 58624 0 229 230 356 1577 72 1152 817 709 192 552 0 2188 0 36 0 216 157 0 3 0 165 524 0 0 3570 0
1066 1185 175 44800 0 176 176 249 1440 72 1153 585 466 141 440 0 1577 0 36 0 214 104 0 2 0 155 339 0 0 4497 0
54 487 6 1536 0 6 6 2 64 4 64 137 2 2 238 0 24 0 2 0 216 0 0 3 0 135 2 0 0 2664 0
0 590 0 0 0 0 0 0 0 0 0 136 0 0 224 0 0 0 0 0 210 0 0 0 0 136 0 0 0 0 0
54 514 4 1024 0 4 4 2 64 4 64 142 2 2 248 0 24 0 2 0 214 0 0 2 0 140 2 0 0 0 0
>>> slabs <<<
stack@helion-cp1-swobj0001-mgmt:~$ sudo collectl -sY -i:1 -c1 --slabfilt xfs
waiting for 1 second sample...
# SLAB DETAIL
# <-----------Objects----------><---------Slab Allocation------><---Change-->
#Name InUse Bytes Alloc Bytes InUse Bytes Total Bytes Diff Pct
xfs_btree_cur 1872 389376 1872 389376 48 393216 48 393216 0 0.0
xfs_da_state 1584 772992 1584 772992 48 786432 48 786432 0 0.0
xfs_dqtrx 0 0 0 0 0 0 0 0 0 0.0
xfs_efd_item 4360 1744000 4760 1904000 119 1949696 119 1949696 0 0.0
xfs_icr 0 0 0 0 0 0 0 0 0 0.0
xfs_ili 48127K 6976M 48197K 6986M 909380 7104M 909380 7104M 0 0.0
xfs_inode 48210K 47080M 48244K 47113M 1507K 47123M 1507K 47123M 0 0.0
_______________________________________________ xfs mailing list xfs@xxxxxxxxxxx http://oss.sgi.com/mailman/listinfo/xfs
