On Fri, Mar 28, 2014 at 9:16 AM, Karol Kozubal <Karol.Kozubal@xxxxxxxxx> wrote:
Hi Everyone,
I have been seeing some weird results with rbd bench-write when the number of threads in my tests goes above 32. Specifically with 64 and 128 threads respectively.
I am running an older dumpling release 0.67.4 and am using it to build a set test cases that I will run for testing future releases in an automated fashion.
I see a bug-fixes for bench-write v0.75 —> rbd: fix bench-write command (Hoamai Wang) … I am wondering if related? See sample test output below.
I remember it's fixed by this patch.
======= NORMAL TEST ==========[root@node-27 ceph-test]# rbd bench-write volumes/loadtest-rbd0 --io-size 1048576 --io-threads 16 --io-pattern randbench-write io_size 1048576 io_threads 16 bytes 1073741824 pattern randSEC OPS OPS/SEC BYTES/SEC1 376 375.65 266087649.432 652 325.16 261994438.303 908 302.63 250578341.714 1148 282.90 230236366.305 1340 267.68 205484358.74elapsed: 5 ops: 1446 ops/sec: 253.19 bytes/sec: 188009371.21
======= WEIRD TEST ==========[root@node-27 ceph-test]# rbd bench-write volumes/loadtest-rbd0 --io-size 1048576 --io-threads 128 --io-pattern randbench-write io_size 1048576 io_threads 128 bytes 1073741824 pattern randSEC OPS OPS/SEC BYTES/SEC1 384 337.12 16194736416915101696.002 512 237.67 8562842898446498816.003 768 210.92 5066194600812718080.004 1024 220.11 3965174267083614720.005 1152 216.40 3465237360609427968.006 1280 207.23 2986498010687531008.007 1664 226.84 2514715944203117568.008 1920 236.36 2270865716147703552.009 2048 227.05 2045053674517228032.0010 2560 243.67 1755846484859866880.0011 2688 244.13 1675406116883640832.0012 3072 244.41 1467639564426388224.0013 3200 241.14 1390071006629100032.0014 3456 237.17 1265939868813378048.0015 3712 239.56 1190468048400115200.0016 3968 241.58 1123095458508870016.0017 4096 238.15 1072546417833582464.0018 4352 240.01 1017316350597379712.0019 4608 237.13 949293044175138048.0020 4864 235.47 893029710006287104.0021 4992 233.67 863485082645484416.0022 5120 228.48 823195784066028544.0023 5376 227.38 780205656792357120.0024 5504 228.95 767317154779148672.0025 5760 227.14 727429806747233536.0026 5888 225.44 706276706322019200.0027 6144 225.54 677157401345555328.0028 6400 225.71 650568424673900544.0029 6656 222.86 617636389350426112.0030 6784 224.50 610458017766997504.0031 7040 225.55 591007193895404160.0032 7296 223.37 564746325168071616.0033 7424 223.78 556035835815750208.0034 7680 224.75 539834468179971648.0035 8064 226.05 517098854196574400.0036 8192 225.39 507543316869586176.0037 8448 228.30 498511095941450432.0038 8832 229.66 479673543746555520.0039 8960 228.84 471124583205229952.0040 9216 229.27 458905121166883328.0041 9472 229.61 447168385720164864.0042 9856 233.47 436964214525758848.0043 10112 232.68 424467809770207168.0044 10240 231.77 417514336196732224.0045 10496 231.86 407500779715553088.0046 10624 230.08 399498013584861376.0047 11008 233.35 391043279738133888.0048 11264 233.32 382100687976533056.0049 11520 233.22 373457549496762816.0050 11648 232.21 367741508793568768.0051 11904 232.51 360309335406545792.0052 12160 231.35 350953792015869248.0053 12416 231.16 343446731043054272.0054 12544 232.01 341185266978134592.0055 12928 233.29 332877335529639168.0056 13056 233.04 329262062622102400.0057 13312 232.87 322687293406453696.0058 13568 233.02 316810684805172736.0059 13824 231.78 309285929137423680.0060 13952 231.71 306353049821044608.0061 14080 230.75 302315372155774272.0062 14336 230.36 296416568090511744.0063 14720 232.43 291281463002263104.0064 14976 231.17 284743487487861024.0065 15104 231.47 282692275460864640.0066 15232 229.77 278265363060257824.0067 15488 230.50 274528806424432416.0068 15744 230.23 269747556006066848.0069 16000 230.65 265925761494344000.0070 16256 230.28 261311480178859008.0071 16384 229.61 0.0072 16512 229.14 14551.0973 16768 229.36 43028.5174 17024 228.25 70294.6575 17280 228.57 97091.7776 17408 228.98 110341.4477 17664 228.38 135571.8978 17920 227.80 159954.5579 18048 228.21 172363.5880 18304 228.74 196556.5581 18688 229.10 231384.7382 18816 228.64 242092.4583 19072 228.42 263729.2384 19200 228.02 273963.8385 19584 227.88 305036.3286 19712 227.68 314893.6287 19968 227.62 334688.2588 20096 228.24 345363.6289 20352 228.32 364666.4790 20608 227.56 382101.4391 20736 227.62 391352.7092 20992 227.73 409520.0993 21248 228.11 427772.1594 21504 228.48 445635.2995 21760 228.64 462738.0696 22016 227.74 477265.4997 22144 228.10 486046.5198 22528 228.85 511293.5799 22784 228.29 525327.89100 22912 228.30 532866.01101 23040 227.98 539528.40102 23296 227.71 553476.20103 23552 227.53 567271.58104 23808 227.74 581768.93105 23936 227.45 587865.58106 24192 227.55 601640.06107 24448 228.14 616455.75108 24704 228.22 629640.52109 24960 228.64 643536.23110 25216 228.66 656101.70111 25472 228.75 668580.33112 25728 229.32 682261.82113 25984 229.42 694358.13114 26240 228.87 704216.95115 26368 228.79 709654.51116 26624 229.04 721660.58117 26880 229.38 733729.20118 27136 229.95 746378.14119 27520 230.40 763762.04120 27648 229.77 766839.23121 27904 230.15 778362.07122 28160 229.91 787612.98123 28288 229.75 792024.74124 28544 229.80 801978.50125 28800 229.91 811977.44126 29056 229.46 819787.11127 29184 229.72 825395.11128 29568 230.42 841640.06129 29824 230.76 851908.57130 30080 231.30 862740.18131 30464 231.57 876756.34132 30592 231.51 880832.15133 30848 231.33 888534.77134 31104 231.82 898721.44135 31360 231.64 906209.71136 31616 231.54 913813.42137 31872 231.59 921916.64138 32000 231.60 925852.38139 32256 232.01 935230.01140 32512 231.59 941138.21141 32768 231.37 947690.01142 32896 230.96 949694.09143 33152 231.63 959761.73144 33536 232.14 972602.68145 33664 231.67 974178.25146 33920 232.13 983077.98147 34176 232.17 990133.00148 34560 232.46 1001547.46149 34816 232.93 1010208.08150 35072 233.06 1017327.45151 35200 232.85 1019642.47152 35328 232.16 1019839.87153 35712 233.02 1033120.77154 35968 233.39 1040997.63155 36352 234.01 1053022.73156 36608 234.44 1060982.19157 36864 234.14 1065613.31158 36992 234.04 1068111.43159 37248 234.13 1074343.60160 37632 234.50 1084676.37161 37760 233.84 1084438.81162 37888 233.34 1084926.16163 38144 233.51 1091263.99164 38400 233.66 1097440.66165 38656 233.47 1101974.43166 38784 233.45 1104537.64167 39040 233.77 1111333.08168 39296 233.83 1116894.88169 39552 233.43 1120133.47170 39680 233.05 1120858.39171 39936 232.64 1123929.79172 40192 233.44 1132809.48173 40448 233.08 1135945.76174 40576 233.05 1138268.49175 40960 232.92 1144869.25176 41088 233.28 1149000.83177 41344 233.01 1152388.03178 41472 232.78 1153586.56179 41728 232.54 1157001.98180 41856 232.47 1158918.42181 42112 232.37 1162962.38182 42368 232.28 1167006.58183 42496 232.15 1168577.80184 42752 231.80 1171157.45185 42880 231.70 1172828.11186 43136 231.90 1178149.03187 43392 232.03 1183078.91188 43776 232.34 1190992.75189 44032 232.65 1196705.15190 44288 232.28 1198914.76191 44416 232.41 1201598.08192 44800 232.64 1208811.68193 44928 232.77 1211488.61194 45184 232.57 1214376.02195 45568 233.31 1224095.81196 45824 233.02 1226367.45197 46080 233.48 1232618.10198 46208 233.12 1232609.63199 46464 233.03 1235848.01200 46592 232.94 1237227.72201 46976 233.44 1245374.59202 47104 232.97 1244687.87203 47488 233.39 1252306.11204 47616 233.28 1253463.36205 47872 233.36 1257405.22206 48128 233.61 1262235.18207 48512 234.16 1270416.92208 48768 233.53 1270374.51209 48896 233.53 1272039.15210 49152 233.61 1275801.88211 49280 233.50 1276859.63212 49664 233.80 1283466.15213 49792 233.43 1283039.34214 50048 233.85 1288557.19215 50304 233.87 1291861.57216 50688 234.02 1297398.66217 50816 234.15 1299731.79218 51072 233.93 1301601.03219 51328 234.21 1306210.23220 51584 234.12 1308755.54221 51840 234.16 1311985.82222 51968 233.96 1312371.07223 52224 234.17 1316470.53224 52480 233.83 1317542.09225 52736 234.04 1321620.65226 52864 233.63 1320710.56227 53120 233.70 1323987.67228 53376 233.82 1327482.51229 53632 233.20 1326754.01230 53760 233.33 1328879.16231 53888 233.25 1329830.94232 54144 233.14 1331949.25233 54272 232.86 1331730.40234 54528 232.33 1331356.25235 54656 232.34 1332786.38236 55040 232.46 1337457.12237 55168 232.60 1339574.94238 55424 232.31 1340523.79239 55680 232.85 1346196.08240 55936 232.41 1346223.91241 56064 232.52 1348124.20242 56320 232.49 1350522.36243 56576 232.64 1353905.61244 56832 232.63 1356337.96245 57088 232.33 1357032.25246 57216 232.21 1357520.85247 57472 232.15 1359641.48248 57600 232.14 1360752.36249 57984 232.53 1366660.06250 58240 232.73 1370185.18251 58496 233.04 1374336.35252 58752 233.07 1376847.82253 59008 232.49 1375762.81254 59136 232.50 1376941.91255 59264 232.29 1376840.52256 59520 232.31 1379202.73257 59776 232.47 1382448.26258 60032 232.51 1384890.91259 60288 232.14 1384892.07260 60416 232.30 1386957.80261 60800 232.77 1393008.77262 61056 232.50 1393526.48263 61184 232.26 1393145.13264 61312 232.13 1393475.70265 61568 232.30 1396589.50266 61824 232.31 1398739.33267 62080 232.34 1400996.22268 62336 232.07 1401443.18269 62464 232.09 1402597.33270 62720 232.21 1405360.99271 62976 231.87 1405326.22272 63232 231.72 1406409.55273 63360 231.90 1408496.78274 63744 232.22 1413399.95275 63872 231.91 1412463.80276 64128 232.26 1416557.75277 64384 232.14 1417748.57278 64640 232.46 1421623.16279 64896 232.13 1421518.14280 65152 232.45 1425360.40281 65408 232.62 1428310.41282 65664 232.79 1431207.24283 65920 232.58 1431754.11284 66176 232.55 1433421.40285 66432 232.68 1436038.13286 66560 232.61 1436512.32287 66688 232.34 1435705.22288 66944 232.37 1437696.75289 67328 232.61 1441815.77290 67584 232.79 1444732.09291 67712 232.44 1443431.35292 67968 232.70 1446762.19293 68224 232.56 1447622.23294 68480 232.54 1449199.34295 68608 232.42 1449291.26296 68864 232.56 1451845.42297 69248 232.78 1455739.56298 69376 232.79 1456670.08299 69632 232.53 1456686.83300 69760 232.31 1456136.78301 70016 232.34 1457916.74302 70144 232.13 1457409.65303 70528 232.54 1462408.53304 70784 232.40 1463161.31305 71040 232.44 1464985.93306 71296 232.65 1467871.69307 71552 232.62 1469269.22308 71808 232.90 1472629.94309 72064 233.10 1475385.15310 72448 233.08 1477563.06311 72576 233.16 1478881.83312 72832 232.85 1478403.16313 72960 232.98 1479991.79314 73088 232.63 1478482.65315 73344 232.63 1479984.29316 73600 232.35 1479697.69317 73728 232.24 1479709.90318 73856 232.08 1479412.80319 73984 231.84 1478642.34320 74240 231.73 1479367.59321 74496 232.02 1482676.27322 74880 232.32 1486716.36323 75008 232.22 1486829.92324 75264 232.02 1486918.64325 75520 232.12 1489011.05326 75776 232.37 1492011.50327 76032 232.37 1493368.04328 76288 232.47 1495428.48329 76672 232.74 1499215.10330 76800 232.56 1498683.95331 77056 232.54 1499924.76332 77312 232.79 1502890.16333 77568 232.76 1504040.09334 77696 232.62 1503768.78335 77824 232.24 1501990.12336 78080 231.89 1501049.17337 78336 232.06 1503408.18338 78464 231.80 1502422.81339 78592 231.80 1503028.10340 78848 231.85 1504650.47341 79104 231.59 1504241.20342 79232 231.52 1504438.14343 79488 231.68 1506694.18344 79744 231.73 1508281.37345 80000 231.49 1508003.78346 80128 231.53 1508893.63347 80384 231.52 1510031.74348 80640 231.28 1509726.62349 80768 231.39 1511056.77350 81152 231.54 1513857.71351 81408 231.91 1517435.76352 81664 231.60 1516647.39353 81792 231.68 1517751.00354 82176 231.89 1520914.77355 82304 231.81 1520967.71356 82560 231.80 1522046.58357 82944 231.99 1525075.56358 83200 231.92 1525754.40359 83328 231.79 1525498.33360 83584 231.85 1527001.08361 83840 232.15 1530127.04362 84096 232.12 1531077.05363 84352 232.18 1532575.38364 84608 232.36 1534887.05365 84864 232.37 1536073.81366 85120 232.36 1537092.48367 85376 232.44 1538708.79368 85632 232.53 1540445.74369 85888 232.65 1542305.23370 86144 232.76 1544089.41371 86528 232.73 1545525.12372 86656 232.82 1546626.30373 86912 232.70 1546893.71374 87168 232.79 1548561.23375 87296 232.74 1548744.55376 87680 232.75 1550387.44377 87808 232.64 1550185.86378 88064 232.65 1551266.43379 88320 232.86 1553748.27380 88576 233.01 1555741.46381 88832 232.88 1555867.15382 88960 232.81 1555947.50383 89344 233.05 1559023.40384 89600 232.90 1559050.92385 89728 232.72 1558350.53386 89856 232.63 1558254.33387 89984 232.49 1557783.54388 90240 232.47 1558647.56389 90496 232.55 1560153.72390 90880 232.86 1563686.26391 91136 232.83 1564422.64392 91264 232.72 1564174.41393 91648 232.97 1567338.02394 91776 232.65 1565605.57395 91904 232.66 1566170.23396 92160 232.34 1564930.59397 92288 232.40 1565862.30398 92544 232.46 1567192.84399 92800 232.39 1567658.73400 93056 232.30 1567942.90401 93184 232.08 1566942.43402 93312 232.04 1567138.96403 93568 232.11 1568523.49404 93952 232.18 1570361.79405 94080 232.24 1571205.67406 94336 232.21 1571890.85407 94592 232.14 1572312.30408 94720 232.05 1572147.37409 95104 232.34 1575417.27410 95360 232.35 1576350.24411 95488 232.31 1576524.82412 95872 232.50 1579150.69413 96128 232.69 1581324.36414 96384 232.60 1581567.19415 96512 232.53 1581506.87416 96768 232.47 1581943.21417 96896 232.33 1581403.29418 97152 232.32 1582183.55419 97408 232.18 1582096.02420 97664 232.45 1584750.44421 97920 232.27 1584363.51422 98048 232.04 1583250.75423 98304 232.23 1585359.50424 98560 231.96 1584342.63425 98688 232.19 1586309.25426 98944 232.21 1587298.48427 99200 232.27 1588471.40428 99456 232.29 1589460.85429 99712 232.42 1591152.29430 99968 232.33 1591335.10431 100224 232.46 1592999.68432 100480 232.38 1593238.81433 100736 232.48 1594714.86434 100864 232.35 1594202.89435 101248 232.63 1597290.61436 101504 232.61 1597942.45437 101760 232.69 1599292.67438 102016 232.80 1600822.95439 102144 232.67 1600319.75440 102656 233.13 1605019.55441 102912 233.31 1606966.57442 103040 233.10 1605952.67443 103424 233.19 1607675.49444 103552 233.04 1606986.95445 103808 232.86 1606523.93446 103936 232.81 1606546.01447 104064 232.75 1606504.49448 104448 232.88 1608484.82449 104704 232.97 1609861.09450 104832 232.95 1610077.05451 105088 232.89 1610388.55452 105216 232.77 1609932.87453 105600 232.90 1611912.68454 105856 232.84 1612232.95455 105984 232.92 1613102.05456 106368 233.04 1614987.14457 106496 233.02 1615198.01458 106752 232.97 1615590.18459 107008 233.07 1617005.53460 107264 232.87 1616268.53461 107392 232.82 1616303.92462 107648 232.88 1617402.90463 107904 232.85 1617883.26464 108032 232.72 1617333.58465 108288 232.83 1618787.43466 108672 233.06 1621382.39467 108800 232.86 1620305.05468 109056 232.81 1620629.10469 109312 232.89 1621875.98470 109568 233.06 1623768.22471 109952 233.15 1625328.41472 110080 232.95 1624327.75473 110208 232.99 1624904.65474 110464 233.03 1625823.90475 110848 233.08 1627192.38476 111104 233.27 1629166.24477 111360 233.37 1630526.06478 111488 233.21 1629687.05479 111744 232.87 1627987.09480 111872 232.97 1628995.16481 112128 232.91 1629224.59482 112384 233.00 1630468.93483 112512 232.72 1628799.10484 112768 232.95 1631101.24485 113152 233.06 1632782.71486 113408 233.02 1633128.59487 113536 232.86 1632326.75488 113664 232.87 1632673.28489 113920 232.90 1633518.93490 114176 232.90 1634145.00491 114432 232.90 1634723.85492 114688 232.87 1635139.00493 114816 232.79 1634881.11494 115072 232.81 1635644.69495 115328 232.97 1637378.69496 115584 233.00 1638179.84497 115840 233.05 1639100.83498 116224 233.32 1641904.97499 116480 233.34 1642648.93500 116864 233.45 1644320.13501 116992 233.40 1644236.15502 117248 233.22 1643570.02503 117376 233.11 1643086.47504 117504 232.98 1642418.03505 117632 232.89 1642128.87506 117888 232.57 1640423.36507 118016 232.58 1640813.37508 118272 232.47 1640615.53509 118528 232.71 1642819.80510 118656 232.49 1641564.43511 118784 232.44 1641533.63512 119168 232.71 1644245.76513 119424 232.71 1644825.81514 119680 232.77 1645827.30515 119936 232.75 1646213.24516 120192 232.73 1646636.42517 120320 232.71 1646797.46518 120704 232.87 1648754.45519 120832 232.78 1648345.89520 121216 232.97 1650501.10521 121472 233.09 1651908.39522 121728 233.11 1652618.27523 121984 233.08 1652918.89524 122240 233.15 1653991.51525 122496 233.27 1655363.47526 122752 233.36 1656502.07527 123008 233.35 1656970.85528 123264 233.25 1656834.60529 123520 233.19 1656894.55530 123648 233.15 1656871.22531 123904 233.17 1657536.76532 124160 233.25 1658617.57533 124416 233.36 1659907.07534 124672 233.42 1660901.59535 124928 233.12 1659239.30536 125056 233.12 1659512.93537 125312 233.12 1660054.31538 125440 233.04 1659728.61539 125824 233.17 1661406.86540 125952 233.21 1661963.68541 126208 233.26 1662775.61542 126592 233.46 1665010.68543 126848 233.48 1665609.17544 127104 233.58 1666843.58545 127232 233.43 1666025.38546 127616 233.61 1668034.11547 127872 233.70 1669174.40548 128128 233.67 1669472.13549 128384 233.49 1668618.19550 128512 233.57 1669461.17551 128768 233.58 1670052.55552 128896 233.42 1669099.47553 129152 233.48 1670003.92554 129408 233.50 1670646.41555 129536 233.35 1669850.64556 129920 233.54 1671886.88557 130176 233.46 1671776.35558 130304 233.29 1670813.93559 130560 233.24 1670940.66560 130688 233.28 1671468.77561 130944 233.29 1672003.56elapsed: 561 ops: 131072 ops/sec: 233.37 bytes/sec: 1911729.44
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Best Regards,
Wheat
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com