Hello! I'm currently in process of playing around with ceph, and before migrating my servers to use it i wanted to benchmark it to have some idea what expected performance it might achieve. I am using 2 400GB Intel S3700 ssds with crush rule created as follows: ➜ ~ ceph osd crush rule create-replicated ssd_mirror default osd ssd And later creating pool ssd_test setting both size and min_size to 2. I proceeded to benchmark it with rbd bench, and using default image object size of 4M the result is about what i have been expecting (10k iops per OSD). However, when creating image with 4K block size, IOPS go down the drain, despite theoretically being better suited for it (4k writes should be amplified to 4M ones in case of 4M object size, reducing iops, while 4k object size should be the one without write amplification). Here is what i used to create the images and benchmark them: ➜ ~ rbd create -p ssd_test --size 16G --object-size 4K big_4k ➜ ~ rbd create -p ssd_test --size 16G --object-size 4M big_4m ➜ ~ rbd bench --io-type write --io-size 4K --io-pattern rand --io-total 256M ssd_test/big_4m bench type write io_size 4096 io_threads 16 bytes 268435456 pattern random SEC OPS OPS/SEC BYTES/SEC 1 8624 8640.00 35389458.37 2 12928 6472.00 26509325.75 3 22672 7562.67 30976698.73 4 37344 9340.00 38256659.85 5 52000 10403.21 42611529.30 elapsed: 6 ops: 65536 ops/sec: 10886.38 bytes/sec: 44590630.43 ➜ ~ rbd bench --io-type write --io-size 4K --io-pattern rand --io-total 256M ssd_test/big_4k bench type write io_size 4096 io_threads 16 bytes 268435456 pattern random SEC OPS OPS/SEC BYTES/SEC 1 6736 6645.67 27220675.51 2 7312 3664.00 15007751.77 3 7920 2645.33 10835290.95 4 8560 2135.46 8746841.19 5 9168 1828.03 7487596.24 6 9760 605.77 2481232.06 7 10368 610.22 2499477.33 8 10960 605.58 2480447.50 9 11536 596.15 2441847.42 10 12160 599.84 2456944.34 11 12816 610.22 2499477.33 12 13488 624.00 2555905.32 13 14144 636.29 2606249.15 14 14816 654.43 2680544.09 15 15472 661.87 2711022.99 16 16128 664.53 2721901.89 17 16800 662.40 2713191.81 18 17472 668.27 2737248.00 19 18080 650.72 2665341.09 20 18544 614.40 2516583.70 21 19056 584.20 2392875.94 22 19376 513.97 2105207.79 23 19584 416.73 1706936.95 24 19840 343.21 1405804.16 25 19984 287.77 1178705.65 26 20128 214.06 876780.01 27 20368 199.04 815255.64 28 20896 265.80 1088726.66 29 21488 339.65 1391222.48 30 22080 419.87 1719795.76 31 22688 512.82 2100513.91 32 23280 582.40 2385511.64 33 23888 598.88 2453010.08 34 24480 600.32 2458916.20 35 25104 603.83 2473304.79 36 25744 611.69 2505480.88 37 26432 629.39 2577994.94 38 27072 636.80 2608334.15 39 27760 653.39 2676272.30 40 28464 673.08 2756924.50 41 29104 673.08 2756924.50 42 29776 668.27 2737216.44 43 30400 665.60 2726299.01 44 31088 667.74 2735051.18 45 31744 654.43 2680544.09 46 32400 657.10 2691471.89 47 33056 657.05 2691283.44 48 33680 655.48 2684829.53 49 34304 642.69 2632442.61 50 35024 656.53 2689128.69 51 35616 645.27 2643006.18 52 36240 635.78 2604167.48 53 36880 641.03 2625642.38 54 37568 652.28 2671732.80 55 38192 634.62 2599385.96 56 38832 642.69 2632442.61 57 39472 647.44 2651898.81 58 40144 651.24 2667468.26 59 40816 648.56 2656512.56 60 41424 647.96 2654025.43 61 42112 653.39 2676272.30 62 42752 655.48 2684829.53 63 43392 649.08 2658636.07 64 44064 650.12 2662893.29 65 44704 653.91 2678406.49 66 45376 654.37 2680302.91 67 46048 658.67 2697926.26 68 46704 663.46 2717539.87 69 47344 654.95 2682685.09 70 48000 659.73 2702246.39 71 48672 660.26 2704411.65 72 49344 659.73 2702246.39 73 50032 665.07 2724119.71 74 50688 672.03 2752618.79 75 51344 667.20 2732847.38 76 52000 661.89 2711116.75 77 52592 650.12 2662893.29 78 53248 642.69 2632442.61 79 53920 645.37 2643426.29 80 54576 649.52 2660425.81 81 55232 649.00 2658288.92 82 55872 655.48 2684829.52 83 56544 661.32 2708752.60 84 57184 653.32 2676010.99 85 57856 656.00 2686977.39 86 58512 657.05 2691283.44 87 59216 668.27 2737216.44 88 59904 670.39 2745923.21 89 60592 679.97 2785150.68 90 61264 680.51 2787375.24 91 61936 684.25 2802700.09 92 62608 680.58 2787648.31 93 63248 667.73 2735030.17 94 63856 654.90 2682454.03 95 64496 645.88 2645539.34 96 65184 648.05 2654392.43 elapsed: 105 ops: 65536 ops/sec: 621.88 bytes/sec: 2547213.95 What could be causing such decrease in random write performance in case of 4k object size image? Best regards, w1kl4s _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |