When you're using "rados put" you're only writing to a single object, which means a single file on a single OSD. "rados bench" is writing to multiple objects, and RGW is striping. You'll need to use similar strategies with librados. -Greg Software Engineer #42 @ http://inktank.com | http://ceph.com On Thu, May 8, 2014 at 9:39 AM, Lukac, Erik <Erik.Lukac at br.de> wrote: > Hi Guys > > I am using ceph in version 0.72 on RHEL 6.5 with four nodes (three of them are mons) with six osds each with 300GB. Every ceph server has four 1GB-Interfaces configured as follows: 1 GB Interface for maintenance (installing, ssh access and so on), 1 GB Interface for client access (filer network) and 1 GB interface for Ceph-Intraconnect. > Hardware is HP DL 380 G7. > > Somehow rados bench tells me I do have a bandwidth performance at about 110 MB/sec (on hardware-client) or 220 MB/sec (from a virtual server connected with 10Gbit on its host). On the same virtual host I can Upload via S3 Gateway (see on the bottom) with 60MB/sec. I'd prefer librados with a java-Client but sadly its performance is low: > * "rados -p pool put infile outfile" on VM: 24MB/sec > * "rados -p pool put infile outfile" on hardware: 28MB/sec > * librados java client on hardware: 34MB/sec > * s3gateway on VM: 60MB/sec > * rados bench with 110MB/sec > > So could anybody tell me how I could increase speed for librados? > > Thanks in advance > > Erik > > > Below I'll separate areas with hashes for better view > > > ##################################################### > osd tree looks like this > root at mir-daten-dev:~# ceph osd tree > # id weight type name up/down reweight > -1 6.48 root default > -6 3.24 room room0 > -2 1.62 host ceph-mir-1-backup > 0 0.27 osd.0 up 1 > 1 0.27 osd.1 up 1 > 2 0.27 osd.2 up 1 > 3 0.27 osd.3 up 1 > 4 0.27 osd.4 up 1 > 5 0.27 osd.5 up 1 > -4 1.62 host ceph-mir-3-backup > 12 0.27 osd.12 up 1 > 13 0.27 osd.13 up 1 > 14 0.27 osd.14 up 1 > 15 0.27 osd.15 up 1 > 16 0.27 osd.16 up 1 > 17 0.27 osd.17 up 1 > -7 3.24 room room1 > -3 1.62 host ceph-mir-2-backup > 6 0.27 osd.6 up 1 > 7 0.27 osd.7 up 1 > 8 0.27 osd.8 up 1 > 9 0.27 osd.9 up 1 > 10 0.27 osd.10 up 1 > 11 0.27 osd.11 up 1 > -5 1.62 host ceph-mir-4-backup > 18 0.27 osd.18 up 1 > 19 0.27 osd.19 up 1 > 20 0.27 osd.20 up 1 > 21 0.27 osd.21 up 1 > 22 0.27 osd.22 up 1 > 23 0.27 osd.23 up 1 > > > ##################################################### > ceph.conf looks like this > root at mir-daten-dev:~# cat /etc/ceph/ceph.conf > [global] > osd_pool_default_pgp_num = 512 > auth_service_required = cephx > filestore_xattr_use_omap = true > auth_client_required = cephx > osd_pool_default_pg_num = 512 > auth_cluster_required = cephx > mon_host = 10.65.161.3,10.65.161.4,10.65.161.7 > public_network = 10.65.161.0/24 > mon_initial_members = ceph-mir-1-backup, ceph-mir-2-backup, ceph-mir-3-backup > cluster_network = 10.65.158.0/24 > fsid = d3e1fe2e-ada7-44aa-b54d-4ca072e54c53 > > > ##################################################### > Here are my testing-results with rados bench > > root at mir-daten-dev:~# rados bench -p rbd 300 write > Maintaining 16 concurrent writes of 4194304 bytes for up to 300 seconds or 0 objects > Object prefix: benchmark_data_mir-daten-dev.mm.br.de_11651 > sec Cur ops started finished avg MB/s cur MB/s last lat avg lat > 0 0 0 0 0 0 - 0 > 1 16 37 21 83.9743 84 0.995239 0.601003 > 2 16 66 50 99.9729 116 0.996489 0.547691 > 3 16 92 76 101.307 104 0.434289 0.56835 > 4 16 120 104 103.974 112 0.469363 0.56067 > 5 16 146 130 103.975 104 0.359053 0.568956 > 6 16 177 161 107.307 124 0.47566 0.567078 > 7 16 206 190 108.544 116 0.540694 0.570833 > 8 16 233 217 108.473 108 0.54371 0.565849 > 9 16 261 245 108.863 112 0.715507 0.56315 > 10 16 286 270 107.974 100 0.413028 0.564032 > 11 16 316 300 109.064 120 0.314785 0.566659 > 12 16 344 328 109.307 112 0.885827 0.564408 > 13 16 374 358 110.127 120 0.543841 0.565009 > 14 16 401 385 109.973 108 0.242369 0.563333 > 15 16 428 412 109.838 108 0.367646 0.570209 > 16 16 457 441 110.222 116 0.378297 0.568373 > 17 16 485 469 110.325 112 0.40088 0.572784 > 18 16 511 495 109.972 104 0.394922 0.566104 > 19 16 541 525 110.498 120 1.26352 0.568944 > 2014-05-07 15:17:34.551363min lat: 0.210907 max lat: 1.85656 avg lat: 0.569114 > sec Cur ops started finished avg MB/s cur MB/s last lat avg lat > 20 16 569 553 110.572 112 0.696153 0.569114 > 21 16 597 581 110.638 112 0.364201 0.565985 > 22 16 625 609 110.699 112 0.478669 0.569997 > 23 16 653 637 110.754 112 1.03317 0.570433 > 24 16 684 668 111.305 124 0.808845 0.569482 > 25 16 708 692 110.691 96 0.335705 0.569535 > 26 16 736 720 110.741 112 0.331024 0.567988 > 27 16 766 750 111.083 120 0.255695 0.568244 > 28 16 793 777 110.972 108 0.298041 0.566006 > 29 16 821 805 111.006 112 0.510306 0.570566 > 30 16 849 833 111.038 112 0.505633 0.569873 > 31 16 880 864 111.455 124 0.586974 0.569956 > 32 16 906 890 111.222 104 0.407643 0.570373 > 33 16 933 917 111.123 108 0.893645 0.570125 > 34 16 959 943 110.913 104 0.400113 0.569121 > 35 15 989 974 111.286 124 0.355817 0.568408 > 36 16 1015 999 110.972 100 1.11119 0.569377 > 37 16 1049 1033 111.648 136 0.421384 0.570536 > 38 16 1073 1057 111.236 96 0.293437 0.569655 > 39 16 1104 1088 111.562 124 0.919575 0.56928 > 2014-05-07 15:17:54.556129min lat: 0.210907 max lat: 1.8669 avg lat: 0.570308 > sec Cur ops started finished avg MB/s cur MB/s last lat avg lat > 40 16 1131 1115 111.472 108 0.511175 0.570308 > 41 16 1160 1144 111.582 116 0.261233 0.569832 > 42 16 1186 1170 111.401 104 0.689742 0.570022 > 43 16 1213 1197 111.321 108 0.483744 0.569546 > 44 16 1243 1227 111.518 120 0.268109 0.569638 > 45 16 1270 1254 111.439 108 0.357269 0.568316 > 46 16 1299 1283 111.538 116 0.412614 0.569021 > 47 16 1326 1310 111.462 108 0.974249 0.570068 > 48 16 1355 1339 111.556 116 0.288035 0.568963 > 49 16 1383 1367 111.564 112 0.767493 0.570603 > 50 16 1412 1396 111.652 116 0.301228 0.570284 > 51 16 1439 1423 111.58 108 0.852262 0.571029 > 52 16 1468 1452 111.665 116 0.714697 0.569751 > 53 16 1494 1478 111.52 104 0.46731 0.57031 > 54 16 1523 1507 111.602 116 0.919255 0.570477 > 55 16 1551 1535 111.609 112 0.37924 0.569673 > 56 16 1580 1564 111.687 116 0.256389 0.569739 > 57 16 1605 1589 111.482 100 0.414508 0.570286 > 58 16 1636 1620 111.697 124 0.430781 0.570266 > 59 16 1661 1645 111.498 100 0.329036 0.570594 > 2014-05-07 15:18:14.560864min lat: 0.208912 max lat: 1.8669 avg lat: 0.57041 > sec Cur ops started finished avg MB/s cur MB/s last lat avg lat > 60 16 1692 1676 111.706 124 0.435502 0.57041 > 61 16 1720 1704 111.71 112 0.311227 0.569743 > 62 16 1749 1733 111.779 116 0.248325 0.569927 > 63 16 1774 1758 111.592 100 0.383215 0.57023 > 64 16 1803 1787 111.66 116 0.850225 0.570017 > 65 16 1831 1815 111.665 112 0.457226 0.570214 > 66 16 1863 1847 111.912 128 0.823255 0.569323 > 67 16 1888 1872 111.734 100 0.300361 0.570023 > 68 16 1913 1897 111.561 100 0.579203 0.569528 > 69 16 1941 1925 111.567 112 0.830359 0.570383 > 70 16 1971 1955 111.687 120 0.368781 0.569875 > 71 16 1999 1983 111.691 112 1.07621 0.569866 > 72 16 2030 2014 111.862 124 0.541884 0.570009 > 73 16 2057 2041 111.808 108 0.497861 0.570163 > 74 16 2086 2070 111.865 116 0.326893 0.569289 > 75 16 2113 2097 111.813 108 0.343165 0.569029 > 76 16 2141 2125 111.815 112 0.557292 0.569121 > 77 16 2168 2152 111.765 108 0.819133 0.569281 > 78 16 2193 2177 111.614 100 0.535671 0.569944 > 79 16 2225 2209 111.821 128 0.831723 0.570187 > 2014-05-07 15:18:34.565588min lat: 0.178012 max lat: 2.16826 avg lat: 0.570225 > sec Cur ops started finished avg MB/s cur MB/s last lat avg lat > 80 16 2252 2236 111.773 108 0.37631 0.570225 > 81 16 2281 2265 111.825 116 0.546754 0.569991 > 82 16 2308 2292 111.778 108 0.709762 0.569898 > 83 16 2338 2322 111.876 120 0.369926 0.570534 > 84 16 2365 2349 111.83 108 0.673388 0.570518 > 85 16 2394 2378 111.879 116 0.350684 0.569976 > 86 16 2420 2404 111.787 104 0.322865 0.569561 > 87 16 2449 2433 111.835 116 0.688097 0.570266 > 88 16 2478 2462 111.882 116 0.492835 0.569798 > 89 16 2506 2490 111.883 112 0.34937 0.570089 > 90 16 2532 2516 111.795 104 0.412616 0.570305 > 91 16 2559 2543 111.754 108 0.399555 0.569667 > 92 16 2588 2572 111.799 116 0.272254 0.570271 > 93 16 2617 2601 111.844 116 0.384923 0.569714 > 94 16 2645 2629 111.846 112 0.432468 0.56963 > 95 16 2675 2659 111.931 120 0.737037 0.570326 > 96 16 2701 2685 111.848 104 0.439707 0.569896 > 97 16 2731 2715 111.932 120 0.702376 0.569979 > 98 16 2757 2741 111.851 104 1.01248 0.570174 > 99 16 2784 2768 111.812 108 0.63679 0.569945 > 2014-05-07 15:18:54.570009min lat: 0.174255 max lat: 2.16826 avg lat: 0.569939 > sec Cur ops started finished avg MB/s cur MB/s last lat avg lat > 100 16 2814 2798 111.893 120 0.776998 0.569939 > 101 16 2844 2828 111.973 120 0.344383 0.569556 > 102 16 2869 2853 111.856 100 0.425127 0.570074 > 103 16 2897 2881 111.857 112 0.736055 0.56995 > 104 16 2926 2910 111.896 116 0.364164 0.569939 > 105 16 2955 2939 111.935 116 0.450451 0.570208 > 106 16 2983 2967 111.935 112 0.33402 0.570272 > 107 16 3011 2995 111.936 112 0.338428 0.56986 > 108 16 3039 3023 111.936 112 0.709239 0.570008 > 109 16 3066 3050 111.9 108 0.558163 0.57014 > 110 16 3093 3077 111.864 108 0.600908 0.570271 > 111 16 3121 3105 111.865 112 0.669814 0.570037 > 112 16 3151 3135 111.937 120 0.348272 0.569947 > 113 16 3178 3162 111.902 108 0.665669 0.569337 > 114 16 3209 3193 112.008 124 1.57979 0.56983 > 115 16 3234 3218 111.903 100 1.35558 0.57012 > 116 16 3263 3247 111.938 116 0.386747 0.570292 > 117 16 3292 3276 111.973 116 0.299954 0.570136 > 118 16 3322 3306 112.041 120 0.509079 0.569812 > 119 16 3348 3332 111.973 104 0.358387 0.569476 > 2014-05-07 15:19:14.575343min lat: 0.174255 max lat: 2.16826 avg lat: 0.569941 > sec Cur ops started finished avg MB/s cur MB/s last lat avg lat > 120 16 3375 3359 111.94 108 0.445028 0.569941 > 121 16 3404 3388 111.973 116 0.454418 0.570076 > 122 16 3430 3414 111.907 104 0.401206 0.570018 > 123 15 3461 3446 112.038 128 0.573862 0.570225 > 124 16 3488 3472 111.973 104 0.511579 0.570008 > 125 16 3516 3500 111.973 112 0.660253 0.569958 > 126 16 3543 3527 111.941 108 0.519482 0.569806 > 127 16 3570 3554 111.91 108 0.432645 0.570315 > 128 16 3597 3581 111.879 108 0.569502 0.570453 > 129 16 3626 3610 111.911 116 0.855213 0.570156 > 130 16 3655 3639 111.942 116 0.525246 0.570005 > 131 16 3687 3671 112.064 128 0.664841 0.56997 > 132 16 3711 3695 111.942 96 0.415433 0.570042 > 133 16 3734 3718 111.792 92 0.490554 0.570393 > 134 16 3768 3752 111.972 136 0.27225 0.570162 > 135 16 3797 3781 112.002 116 0.379587 0.570286 > 136 16 3824 3808 111.972 108 0.502209 0.570024 > 137 16 3852 3836 111.972 112 0.798701 0.57013 > 138 16 3881 3865 112.001 116 0.421902 0.570169 > 139 16 3907 3891 111.944 104 0.503618 0.570294 > 2014-05-07 15:19:34.580882min lat: 0.174255 max lat: 2.16826 avg lat: 0.570275 > sec Cur ops started finished avg MB/s cur MB/s last lat avg lat > 140 16 3935 3919 111.944 112 0.693392 0.570275 > 141 16 3965 3949 112.001 120 0.966384 0.569917 > 142 16 3990 3974 111.916 100 0.513661 0.570226 > 143 16 4021 4005 112 124 0.570069 0.570123 > 144 16 4050 4034 112.028 116 0.899172 0.570064 > 145 16 4075 4059 111.945 100 0.328666 0.57015 > 146 16 4106 4090 112.027 124 0.363944 0.570249 > 147 16 4135 4119 112.054 116 0.883335 0.570093 > 148 16 4162 4146 112.026 108 0.410028 0.570276 > 149 16 4191 4175 112.053 116 0.327328 0.569752 > 150 16 4218 4202 112.025 108 0.390957 0.569733 > 151 16 4245 4229 111.999 108 0.798766 0.570268 > 152 16 4270 4254 111.92 100 0.588784 0.570268 > 153 16 4302 4286 112.024 128 0.710391 0.570196 > 154 16 4330 4314 112.024 112 0.417519 0.569602 > 155 16 4357 4341 111.998 108 0.349489 0.569915 > 156 16 4387 4371 112.049 120 0.506385 0.570289 > 157 16 4411 4395 111.947 96 0.504476 0.570322 > 158 16 4440 4424 111.972 116 0.473411 0.570088 > 159 16 4468 4452 111.972 112 0.432837 0.570101 > 2014-05-07 15:19:54.586163min lat: 0.174255 max lat: 2.16826 avg lat: 0.570156 > sec Cur ops started finished avg MB/s cur MB/s last lat avg lat > 160 16 4499 4483 112.047 124 0.352313 0.570156 > 161 16 4525 4509 111.997 104 0.535317 0.570074 > 162 16 4555 4539 112.046 120 0.404709 0.569884 > 163 16 4582 4566 112.021 108 0.348859 0.570201 > 164 16 4611 4595 112.045 116 0.497089 0.570198 > 165 16 4637 4621 111.996 104 0.40071 0.570064 > 166 16 4667 4651 112.044 120 0.655012 0.570288 > 167 16 4693 4677 111.996 104 0.588973 0.570166 > 168 16 4720 4704 111.972 108 0.387201 0.570245 > 169 16 4749 4733 111.996 116 0.398496 0.570141 > 170 16 4778 4762 112.019 116 0.343578 0.570124 > 171 16 4807 4791 112.042 116 0.389564 0.570082 > 172 16 4836 4820 112.065 116 0.466953 0.570179 > 173 16 4861 4845 111.995 100 0.613515 0.570412 > 174 16 4890 4874 112.018 116 0.549259 0.570359 > 175 16 4916 4900 111.972 104 0.643929 0.570289 > 176 16 4948 4932 112.063 128 0.414175 0.570148 > 177 16 4973 4957 111.995 100 0.441505 0.570318 > 178 16 5003 4987 112.04 120 1.25549 0.569983 > 179 16 5030 5014 112.017 108 0.336975 0.569644 > 2014-05-07 15:20:14.590982min lat: 0.174255 max lat: 2.16826 avg lat: 0.570173 > sec Cur ops started finished avg MB/s cur MB/s last lat avg lat > 180 16 5054 5038 111.928 96 1.51731 0.570173 > 181 16 5088 5072 112.061 136 0.42772 0.570161 > 182 16 5116 5100 112.06 112 0.294168 0.570291 > 183 16 5144 5128 112.06 112 0.537038 0.570064 > 184 16 5171 5155 112.037 108 0.752229 0.570163 > 185 16 5200 5184 112.059 116 0.318506 0.569913 > 186 16 5226 5210 112.015 104 1.47843 0.569888 > 187 16 5254 5238 112.015 112 0.752977 0.569937 > 188 16 5283 5267 112.036 116 0.404206 0.570304 > 189 16 5311 5295 112.036 112 0.418082 0.570128 > 190 16 5341 5325 112.078 120 0.883179 0.570244 > 191 16 5366 5350 112.014 100 0.350256 0.569957 > 192 16 5394 5378 112.014 112 0.875397 0.569928 > 193 16 5423 5407 112.035 116 0.491794 0.570264 > 194 16 5452 5436 112.055 116 0.506443 0.570207 > 195 16 5481 5465 112.075 116 0.568327 0.570232 > 196 16 5509 5493 112.074 112 0.302135 0.570117 > 197 16 5537 5521 112.074 112 0.616855 0.570028 > 198 16 5563 5547 112.033 104 0.295098 0.569937 > 199 16 5591 5575 112.033 112 0.39924 0.570102 > 2014-05-07 15:20:34.595454min lat: 0.174255 max lat: 2.2057 avg lat: 0.570139 > sec Cur ops started finished avg MB/s cur MB/s last lat avg lat > 200 16 5622 5606 112.092 124 0.674583 0.570139 > 201 16 5647 5631 112.032 100 0.547469 0.570146 > 202 16 5675 5659 112.032 112 0.527545 0.570154 > 203 16 5703 5687 112.032 112 0.962126 0.570288 > 204 16 5731 5715 112.031 112 0.472406 0.570214 > 205 16 5760 5744 112.05 116 0.422501 0.570429 > 206 16 5787 5771 112.031 108 0.618225 0.570166 > 207 16 5815 5799 112.03 112 0.494916 0.570187 > 208 16 5845 5829 112.069 120 0.467093 0.570061 > 209 16 5873 5857 112.068 112 0.659965 0.570255 > 210 16 5899 5883 112.03 104 0.380473 0.570237 > 211 16 5928 5912 112.048 116 0.667309 0.570291 > 212 16 5956 5940 112.048 112 0.408311 0.570264 > 213 16 5985 5969 112.066 116 0.423455 0.570138 > 214 16 6015 5999 112.103 120 1.07327 0.570201 > 215 16 6038 6022 112.01 92 0.351275 0.569812 > 216 16 6067 6051 112.028 116 0.552788 0.570362 > 217 16 6098 6082 112.083 124 0.495721 0.570101 > 218 16 6128 6112 112.119 120 0.296382 0.570358 > 219 16 6153 6137 112.064 100 0.637389 0.570387 > 2014-05-07 15:20:54.600149min lat: 0.174255 max lat: 2.2057 avg lat: 0.570293 > sec Cur ops started finished avg MB/s cur MB/s last lat avg lat > 220 16 6181 6165 112.063 112 0.528891 0.570293 > 221 16 6209 6193 112.063 112 0.427072 0.569998 > 222 16 6237 6221 112.063 112 0.366226 0.570107 > 223 16 6266 6250 112.08 116 0.655426 0.570216 > 224 16 6296 6280 112.115 120 0.288003 0.570228 > 225 16 6321 6305 112.061 100 0.385101 0.570111 > 226 16 6350 6334 112.079 116 0.317248 0.569956 > 227 16 6376 6360 112.043 104 0.602346 0.570181 > 228 16 6405 6389 112.06 116 0.555873 0.570293 > 229 16 6434 6418 112.077 116 0.42114 0.570214 > 230 16 6464 6448 112.112 120 1.0661 0.570312 > 231 16 6488 6472 112.042 96 0.613699 0.570142 > 232 16 6518 6502 112.076 120 0.390894 0.569782 > 233 16 6548 6532 112.11 120 0.93171 0.57002 > 234 16 6576 6560 112.109 112 0.401531 0.570329 > 235 16 6603 6587 112.092 108 0.212764 0.570149 > 236 16 6630 6614 112.074 108 0.548616 0.570081 > 237 16 6658 6642 112.074 112 0.952604 0.570287 > 238 16 6686 6670 112.073 112 0.517625 0.570102 > 239 16 6715 6699 112.09 116 0.510559 0.570204 > 2014-05-07 15:21:14.605333min lat: 0.174255 max lat: 2.2057 avg lat: 0.570303 > sec Cur ops started finished avg MB/s cur MB/s last lat avg lat > 240 16 6741 6725 112.056 104 0.696723 0.570303 > 241 16 6772 6756 112.105 124 0.458639 0.570237 > 242 16 6799 6783 112.088 108 0.519677 0.570185 > 243 16 6828 6812 112.104 116 0.487797 0.570044 > 244 16 6855 6839 112.087 108 1.01748 0.5702 > 245 16 6883 6867 112.087 112 1.16187 0.570241 > 246 16 6908 6892 112.037 100 0.251722 0.570216 > 247 16 6940 6924 112.102 128 0.860928 0.570233 > 248 16 6969 6953 112.117 116 0.727263 0.570257 > 249 16 6993 6977 112.053 96 0.425839 0.569995 > 250 16 7024 7008 112.1 124 0.306001 0.57019 > 251 16 7051 7035 112.084 108 0.482107 0.570278 > 252 16 7078 7062 112.067 108 0.873547 0.570228 > 253 16 7107 7091 112.083 116 0.9643 0.57023 > 254 16 7137 7121 112.114 120 0.69332 0.570155 > 255 16 7165 7149 112.113 112 0.272354 0.570191 > 256 16 7189 7173 112.05 96 0.607597 0.57029 > 257 16 7219 7203 112.081 120 0.585617 0.570187 > 258 16 7247 7231 112.081 112 0.493043 0.570103 > 259 16 7276 7260 112.096 116 0.917735 0.570052 > 2014-05-07 15:21:34.611151min lat: 0.174255 max lat: 2.2057 avg lat: 0.570264 > sec Cur ops started finished avg MB/s cur MB/s last lat avg lat > 260 16 7304 7288 112.095 112 0.612558 0.570264 > 261 16 7330 7314 112.064 104 0.350458 0.570288 > 262 16 7361 7345 112.109 124 0.49463 0.570312 > 263 16 7388 7372 112.094 108 0.396892 0.570132 > 264 16 7416 7400 112.093 112 0.384927 0.570231 > 265 16 7444 7428 112.093 112 0.907725 0.570205 > 266 16 7474 7458 112.122 120 0.652105 0.570234 > 267 16 7499 7483 112.077 100 0.778864 0.570244 > 268 16 7528 7512 112.091 116 0.345874 0.57017 > 269 16 7557 7541 112.106 116 0.556877 0.570219 > 270 16 7585 7569 112.105 112 0.307575 0.570093 > 271 16 7610 7594 112.061 100 0.579548 0.570186 > 272 16 7642 7626 112.119 128 0.458491 0.57018 > 273 16 7670 7654 112.118 112 0.567194 0.570201 > 274 16 7698 7682 112.118 112 0.344892 0.570066 > 275 16 7726 7710 112.117 112 0.550079 0.570216 > 276 16 7752 7736 112.088 104 0.446476 0.570156 > 277 16 7784 7768 112.145 128 0.310376 0.570067 > 278 16 7809 7793 112.101 100 0.478405 0.570063 > 279 16 7840 7824 112.144 124 0.69451 0.57029 > 2014-05-07 15:21:54.616358min lat: 0.174255 max lat: 2.2057 avg lat: 0.570128 > sec Cur ops started finished avg MB/s cur MB/s last lat avg lat > 280 16 7865 7849 112.1 100 0.638483 0.570128 > 281 16 7892 7876 112.086 108 0.551777 0.570344 > 282 16 7919 7903 112.071 108 0.217284 0.570022 > 283 16 7949 7933 112.099 120 0.547334 0.570056 > 284 16 7977 7961 112.099 112 0.334655 0.570195 > 285 16 8002 7986 112.056 100 0.527645 0.570253 > 286 16 8034 8018 112.112 128 0.494884 0.570122 > 287 16 8064 8048 112.139 120 0.703692 0.570229 > 288 16 8093 8077 112.152 116 1.26224 0.570019 > 289 16 8118 8102 112.11 100 0.354712 0.570032 > 290 16 8146 8130 112.11 112 0.38737 0.570173 > 291 16 8171 8155 112.068 100 0.433604 0.570262 > 292 16 8202 8186 112.109 124 0.477104 0.570029 > 293 16 8229 8213 112.095 108 0.370964 0.570162 > 294 16 8259 8243 112.121 120 0.542573 0.570031 > 295 16 8284 8268 112.08 100 1.72887 0.570075 > 296 16 8313 8297 112.093 116 0.479744 0.570247 > 297 16 8339 8323 112.066 104 0.665292 0.570267 > 298 16 8372 8356 112.133 132 0.466169 0.570263 > 299 16 8398 8382 112.106 104 0.339438 0.570362 > 2014-05-07 15:22:14.621574min lat: 0.174255 max lat: 2.76322 avg lat: 0.570285 > sec Cur ops started finished avg MB/s cur MB/s last lat avg lat > 300 16 8426 8410 112.105 112 0.432868 0.570285 > Total time run: 300.511227 > Total writes made: 8426 > Write size: 4194304 > Bandwidth (MB/sec): 112.156 > > Stddev Bandwidth: 10.5095 > Max bandwidth (MB/sec): 136 > Min bandwidth (MB/sec): 0 > Average Latency: 0.570522 > Stddev Latency: 0.261288 > Max latency: 2.76322 > Min latency: 0.174255 > > > > > [devel at transcoder-devel-2-backup julian]$ rados bench -p rbd 300 write > Maintaining 16 concurrent writes of 4194304 bytes for up to 300 seconds or 0 objects > Object prefix: benchmark_data_transcoder-devel-2-backup.mm._24669 > sec Cur ops started finished avg MB/s cur MB/s last lat avg lat > 0 0 0 0 0 0 - 0 > 1 16 57 41 163.951 164 0.196733 0.304261 > 2 16 109 93 185.953 208 0.673813 0.300496 > 3 16 160 144 191.957 204 0.153186 0.310876 > 4 16 212 196 195.959 208 0.185877 0.307243 > 5 16 272 256 204.759 240 0.185487 0.304008 > 6 16 328 312 207.959 224 0.331776 0.300013 > 7 16 391 375 214.244 252 0.18373 0.294665 > 8 16 441 425 212.458 200 0.263788 0.294404 > 9 16 494 478 212.403 212 0.313927 0.294699 > 10 16 548 532 212.759 216 0.420021 0.295057 > 11 16 603 587 213.414 220 0.206472 0.293776 > 12 16 660 644 214.626 228 0.166892 0.294026 > 13 16 719 703 216.267 236 0.274922 0.29229 > 14 16 772 756 215.96 212 0.131731 0.292452 > 15 16 828 812 216.494 224 0.368353 0.292708 > 16 16 880 864 215.96 208 0.258035 0.292958 > 17 16 942 926 217.843 248 0.261063 0.291515 > 18 16 1001 985 218.85 236 0.313773 0.289734 > 19 16 1055 1039 218.698 216 0.370068 0.289579 > 2014-05-07 15:36:05.930782min lat: 0.118212 max lat: 0.982659 avg lat: 0.288833 > sec Cur ops started finished avg MB/s cur MB/s last lat avg lat > 20 16 1111 1095 218.961 224 0.382812 0.288833 > 21 16 1165 1149 218.818 216 0.155925 0.290081 > 22 16 1220 1204 218.87 220 0.287617 0.290663 > 23 16 1274 1258 218.744 216 0.188214 0.289873 > 24 16 1330 1314 218.962 224 0.222969 0.290436 > 25 16 1385 1369 219.002 220 0.321177 0.289558 > 26 16 1440 1424 219.039 220 0.260352 0.290391 > 27 16 1495 1479 219.074 220 0.301911 0.290416 > 28 16 1546 1530 218.534 204 0.453075 0.290408 > 29 16 1603 1587 218.86 228 0.263811 0.291121 > 30 16 1665 1649 219.83 248 0.157802 0.290306 > 31 16 1720 1704 219.834 220 0.197737 0.289517 > 32 16 1776 1760 219.963 224 0.350793 0.289711 > 33 16 1838 1822 220.812 248 0.213898 0.288446 > 34 16 1885 1869 219.845 188 0.29558 0.288869 > 35 16 1942 1926 220.077 228 0.180696 0.289057 > 36 16 1997 1981 220.074 220 0.147671 0.289322 > 37 16 2058 2042 220.719 244 0.485571 0.289081 > 38 16 2111 2095 220.489 212 0.171761 0.288655 > 39 16 2170 2154 220.886 236 0.483077 0.28843 > 2014-05-07 15:36:25.933888min lat: 0.118212 max lat: 1.31226 avg lat: 0.287819 > sec Cur ops started finished avg MB/s cur MB/s last lat avg lat > 40 16 2231 2215 221.463 244 0.139965 0.287819 > 41 16 2287 2271 221.524 224 0.748409 0.287424 > 42 16 2342 2326 221.486 220 0.128901 0.287633 > 43 16 2396 2380 221.358 216 0.162042 0.287813 > 44 16 2454 2438 221.599 232 0.228326 0.288024 > 45 16 2509 2493 221.563 220 0.20862 0.28763 > 46 16 2561 2545 221.267 208 0.162174 0.287787 > 47 16 2623 2607 221.835 248 0.220641 0.287375 > 48 16 2681 2665 222.046 232 0.333684 0.287381 > 49 16 2740 2724 222.33 236 0.267276 0.287094 > 50 16 2794 2778 222.203 216 0.245417 0.28705 > 51 16 2853 2837 222.473 236 0.201729 0.286772 > 52 16 2909 2893 222.501 224 0.353249 0.286742 > 53 16 2965 2949 222.528 224 0.202386 0.286661 > 54 16 3019 3003 222.406 216 0.132111 0.286654 > 55 16 3079 3063 222.726 240 0.540595 0.28653 > 56 16 3129 3113 222.319 200 0.16454 0.286663 > 57 16 3187 3171 222.488 232 0.307091 0.286818 > 58 16 3244 3228 222.582 228 0.302001 0.28697 > 59 16 3301 3285 222.672 228 0.231865 0.286736 > 2014-05-07 15:36:45.937815min lat: 0.11603 max lat: 1.31226 avg lat: 0.286587 > sec Cur ops started finished avg MB/s cur MB/s last lat avg lat > 60 16 3354 3338 222.494 212 0.17932 0.286587 > 61 15 3410 3395 222.583 228 0.377044 0.28645 > 62 16 3466 3450 222.541 220 0.503329 0.286602 > 63 16 3521 3505 222.499 220 0.239012 0.286859 > 64 16 3579 3563 222.647 232 0.273032 0.286604 > 65 16 3630 3614 222.359 204 0.295754 0.286864 > 66 16 3684 3668 222.262 216 0.267726 0.286993 > 67 15 3738 3723 222.228 220 0.240881 0.287273 > 68 16 3800 3784 222.547 244 0.17743 0.286966 > 69 15 3850 3835 222.278 204 0.300037 0.287251 > 70 16 3902 3886 222.016 204 0.263521 0.287257 > 71 16 3962 3946 222.268 240 0.165698 0.287258 > 72 16 4017 4001 222.236 220 0.240939 0.287216 > 73 16 4069 4053 222.04 208 0.285829 0.287303 > 74 16 4128 4112 222.228 236 0.196076 0.287468 > 75 16 4179 4163 221.985 204 0.307563 0.287798 > 76 16 4234 4218 221.958 220 0.16673 0.287598 > 77 16 4287 4271 221.828 212 0.442863 0.287649 > 78 16 4343 4327 221.855 224 0.160556 0.2879 > 79 16 4397 4381 221.78 216 0.302359 0.287955 > 2014-05-07 15:37:05.942570min lat: 0.101119 max lat: 1.31226 avg lat: 0.287963 > sec Cur ops started finished avg MB/s cur MB/s last lat avg lat > 80 16 4451 4435 221.707 216 0.244891 0.287963 > 81 16 4504 4488 221.587 212 0.406555 0.288143 > 82 16 4559 4543 221.567 220 0.367044 0.288375 > 83 16 4609 4593 221.306 200 0.238351 0.288399 > 84 16 4670 4654 221.576 244 0.254535 0.288308 > 85 16 4723 4707 221.462 212 0.429971 0.288324 > 86 16 4779 4763 221.491 224 0.335297 0.288531 > 87 16 4831 4815 221.336 208 0.231926 0.288593 > 88 16 4888 4872 221.411 228 0.173742 0.288654 > 89 16 4947 4931 221.574 236 0.455374 0.288325 > 90 16 5002 4986 221.556 220 0.201723 0.288224 > 91 16 5064 5048 221.846 248 0.248725 0.288172 > 92 16 5117 5101 221.738 212 0.522876 0.287972 > 93 16 5177 5161 221.934 240 0.358288 0.287923 > 94 16 5230 5214 221.828 212 0.225764 0.288023 > 95 16 5288 5272 221.934 232 0.158011 0.287802 > 96 16 5339 5323 221.747 204 0.228831 0.288154 > 97 16 5393 5377 221.687 216 0.266464 0.288197 > 98 16 5456 5440 221.996 252 0.210746 0.287881 > 99 16 5508 5492 221.854 208 0.354967 0.287937 > 2014-05-07 15:37:25.947699min lat: 0.101119 max lat: 1.31226 avg lat: 0.28801 > sec Cur ops started finished avg MB/s cur MB/s last lat avg lat > 100 16 5559 5543 221.675 204 0.140023 0.28801 > 101 16 5617 5601 221.775 232 0.21516 0.288175 > 102 16 5674 5658 221.836 228 0.680157 0.288088 > 103 16 5724 5708 221.624 200 0.392328 0.288043 > 104 16 5780 5764 221.646 224 0.365259 0.288279 > 105 16 5832 5816 221.516 208 0.554432 0.288295 > 106 16 5891 5875 221.652 236 0.337652 0.288325 > 107 16 5945 5929 221.598 216 0.140403 0.288345 > 108 16 5998 5982 221.509 212 0.17101 0.288433 > 109 16 6052 6036 221.458 216 0.325362 0.288317 > 110 16 6113 6097 221.662 244 0.431961 0.288293 > 111 16 6165 6149 221.539 208 0.213641 0.288337 > 112 16 6223 6207 221.631 232 0.290333 0.288413 > 113 16 6281 6265 221.723 232 0.127446 0.288206 > 114 16 6330 6314 221.497 196 0.291686 0.288445 > 115 16 6381 6365 221.344 204 0.439944 0.28866 > 116 16 6435 6419 221.298 216 0.189623 0.288739 > 117 16 6494 6478 221.423 236 0.150869 0.288596 > 118 15 6546 6531 221.342 212 0.349119 0.288871 > 119 16 6600 6584 221.263 212 0.164204 0.288836 > 2014-05-07 15:37:45.952978min lat: 0.101119 max lat: 1.31226 avg lat: 0.288824 > sec Cur ops started finished avg MB/s cur MB/s last lat avg lat > 120 16 6653 6637 221.186 212 0.280823 0.288824 > 121 16 6703 6687 221.01 200 0.439371 0.289099 > 122 16 6757 6741 220.969 216 0.300013 0.28925 > 123 16 6819 6803 221.188 248 0.320116 0.289027 > 124 16 6875 6859 221.21 224 0.379419 0.289027 > 125 16 6930 6914 221.2 220 0.257334 0.288887 > 126 16 6982 6966 221.095 208 0.222303 0.289058 > 127 16 7039 7023 221.149 228 0.237355 0.289058 > 128 16 7093 7077 221.109 216 0.294762 0.289136 > 129 16 7146 7130 221.037 212 0.226828 0.289207 > 130 16 7205 7189 221.152 236 0.223497 0.289055 > 131 16 7261 7245 221.173 224 0.15263 0.289045 > 132 16 7316 7300 221.164 220 0.415826 0.288999 > 133 16 7374 7358 221.245 232 0.299776 0.288904 > 134 16 7428 7412 221.206 216 0.233308 0.288926 > 135 16 7480 7464 221.108 208 0.258876 0.28901 > 136 16 7542 7526 221.305 248 0.555874 0.288949 > 137 16 7597 7581 221.295 220 0.181116 0.288823 > 138 16 7655 7639 221.372 232 0.589999 0.288784 > 139 16 7706 7690 221.247 204 0.41453 0.288974 > 2014-05-07 15:38:05.957693min lat: 0.101119 max lat: 1.31226 avg lat: 0.289045 > sec Cur ops started finished avg MB/s cur MB/s last lat avg lat > 140 16 7757 7741 221.123 204 0.207242 0.289045 > 141 16 7808 7792 221.001 204 0.542268 0.289207 > 142 16 7863 7847 220.994 220 0.2161 0.289343 > 143 16 7917 7901 220.959 216 0.438382 0.289303 > 144 15 7967 7952 220.84 204 0.184216 0.289409 > 145 16 8026 8010 220.917 232 0.198802 0.28942 > 146 16 8082 8066 220.938 224 0.377304 0.289521 > 147 16 8134 8118 220.849 208 0.207329 0.289623 > 148 16 8185 8169 220.735 204 0.446269 0.289627 > 149 16 8234 8218 220.569 196 0.391211 0.289634 > 150 16 8296 8280 220.751 248 0.286601 0.289606 > 151 16 8353 8337 220.799 228 0.203956 0.28951 > 152 16 8405 8389 220.715 208 0.359358 0.289726 > 153 16 8455 8439 220.579 200 0.468256 0.289742 > 154 16 8508 8492 220.523 212 0.348861 0.28984 > 155 16 8566 8550 220.597 232 0.188032 0.289738 > 156 16 8623 8607 220.644 228 0.324188 0.289767 > 157 16 8676 8660 220.588 212 0.307222 0.289818 > 158 16 8736 8720 220.711 240 0.192185 0.289669 > 159 16 8788 8772 220.63 208 0.251223 0.28971 > 2014-05-07 15:38:25.962604min lat: 0.101119 max lat: 1.31226 avg lat: 0.28963 > sec Cur ops started finished avg MB/s cur MB/s last lat avg lat > 160 16 8849 8833 220.776 244 0.321942 0.28963 > 161 16 8910 8894 220.92 244 0.291673 0.289396 > 162 16 8964 8948 220.889 216 0.335248 0.289452 > 163 16 9027 9011 221.08 252 0.174739 0.289321 > 164 16 9078 9062 220.975 204 0.4813 0.289338 > 165 16 9135 9119 221.018 228 0.391787 0.289297 > 166 16 9191 9175 221.035 224 0.173787 0.289256 > 167 15 9239 9224 220.885 196 0.150146 0.289419 > 168 16 9297 9281 220.927 228 0.502478 0.28929 > 169 16 9354 9338 220.968 228 0.166848 0.289367 > 170 16 9410 9394 220.986 224 0.181487 0.28934 > 171 16 9466 9450 221.003 224 0.135469 0.289335 > 172 16 9517 9501 220.904 204 0.318316 0.289544 > 173 16 9561 9545 220.644 176 0.6612 0.289637 > 174 16 9615 9599 220.617 216 0.307173 0.289769 > 175 16 9672 9656 220.659 228 0.205008 0.289731 > 176 15 9725 9710 220.632 216 0.281084 0.289828 > 177 16 9776 9760 220.515 200 0.200917 0.28984 > 178 16 9827 9811 220.422 204 0.54947 0.290039 > 179 16 9883 9867 220.442 224 0.191815 0.290031 > 2014-05-07 15:38:45.967797min lat: 0.101119 max lat: 1.31226 avg lat: 0.290102 > sec Cur ops started finished avg MB/s cur MB/s last lat avg lat > 180 16 9936 9920 220.395 212 0.332011 0.290102 > 181 16 9998 9982 220.547 248 0.13831 0.289878 > 182 16 10053 10037 220.544 220 0.150082 0.289824 > 183 16 10110 10094 220.584 228 0.187842 0.28975 > 184 16 10173 10157 220.754 252 0.166696 0.289616 > 185 16 10227 10211 220.728 216 0.469861 0.289713 > 186 15 10280 10265 220.703 216 0.574895 0.289726 > 187 16 10335 10319 220.677 216 0.255842 0.28972 > 188 16 10392 10376 220.716 228 0.41683 0.289739 > 189 16 10444 10428 220.648 208 0.181431 0.289713 > 190 16 10495 10479 220.56 204 0.209041 0.289805 > 191 16 10555 10539 220.662 240 0.280371 0.289747 > 192 16 10612 10596 220.7 228 0.312355 0.289669 > 193 16 10669 10653 220.737 228 0.32539 0.289663 > 194 16 10730 10714 220.856 244 0.261148 0.289648 > 195 16 10783 10767 220.811 212 0.2097 0.289549 > 196 16 10841 10825 220.867 232 0.316249 0.289543 > 197 16 10896 10880 220.863 220 0.128579 0.289509 > 198 16 10952 10936 220.878 224 0.168387 0.289526 > 199 16 11006 10990 220.854 216 0.331512 0.28956 > 2014-05-07 15:39:05.973340min lat: 0.101119 max lat: 1.31226 avg lat: 0.289592 > sec Cur ops started finished avg MB/s cur MB/s last lat avg lat > 200 16 11055 11039 220.729 196 0.470352 0.289592 > 201 16 11114 11098 220.805 236 0.400764 0.289588 > 202 16 11172 11156 220.86 232 0.126321 0.28951 > 203 16 11225 11209 220.816 212 0.396037 0.289653 > 204 16 11273 11257 220.674 192 0.446068 0.289803 > 205 16 11333 11317 220.768 240 0.250535 0.289648 > 206 16 11387 11371 220.745 216 0.261559 0.289689 > 207 16 11439 11423 220.683 208 0.241687 0.289775 > 208 16 11493 11477 220.66 216 0.182643 0.289805 > 209 16 11545 11529 220.6 208 0.467472 0.289864 > 210 16 11599 11583 220.577 216 0.179334 0.289928 > 211 16 11652 11636 220.537 212 0.191147 0.289982 > 212 16 11712 11696 220.628 240 0.486861 0.289892 > 213 16 11756 11740 220.418 176 0.251059 0.290009 > 214 15 11813 11798 220.472 232 0.220856 0.290104 > 215 16 11867 11851 220.432 212 0.280293 0.289999 > 216 16 11918 11902 220.356 204 0.256108 0.290196 > 217 16 11968 11952 220.262 200 0.235377 0.290241 > 218 16 12032 12016 220.426 256 0.207769 0.290198 > 219 16 12079 12063 220.277 188 0.217877 0.290414 > 2014-05-07 15:39:25.978563min lat: 0.101119 max lat: 1.31226 avg lat: 0.290498 > sec Cur ops started finished avg MB/s cur MB/s last lat avg lat > 220 16 12127 12111 220.149 192 0.256724 0.290498 > 221 16 12181 12165 220.13 216 0.359885 0.290496 > 222 16 12238 12222 220.165 228 0.249357 0.290477 > 223 16 12300 12284 220.289 248 0.199543 0.2904 > 224 16 12361 12345 220.395 244 0.630404 0.290275 > 225 16 12407 12391 220.233 184 0.180899 0.290311 > 226 16 12463 12447 220.249 224 0.154449 0.290278 > 227 16 12522 12506 220.319 236 0.246978 0.29031 > 228 16 12580 12564 220.37 232 0.121791 0.290182 > 229 16 12641 12625 220.472 244 0.275696 0.290169 > 230 16 12694 12678 220.435 212 0.872334 0.290054 > 231 15 12749 12734 220.45 224 0.34631 0.290183 > 232 16 12802 12786 220.397 208 0.320038 0.290167 > 233 16 12862 12846 220.48 240 0.149949 0.290078 > 234 16 12917 12901 220.478 220 0.235769 0.290046 > 235 16 12972 12956 220.476 220 0.371443 0.29007 > 236 16 13033 13017 220.575 244 0.222503 0.289984 > 237 16 13092 13076 220.64 236 0.240298 0.289902 > 238 16 13146 13130 220.62 216 0.249867 0.289867 > 239 16 13203 13187 220.651 228 0.308651 0.289906 > 2014-05-07 15:39:45.983532min lat: 0.101119 max lat: 1.31226 avg lat: 0.289966 > sec Cur ops started finished avg MB/s cur MB/s last lat avg lat > 240 16 13250 13234 220.515 188 0.276054 0.289966 > 241 16 13316 13300 220.695 264 0.162547 0.289883 > 242 15 13369 13354 220.675 216 0.31763 0.289797 > 243 16 13428 13412 220.722 232 0.123805 0.28971 > 244 15 13479 13464 220.669 208 0.230601 0.289809 > 245 16 13534 13518 220.65 216 0.115343 0.289813 > 246 16 13585 13569 220.582 204 0.168631 0.289851 > 247 16 13645 13629 220.661 240 0.211015 0.2898 > 248 16 13705 13689 220.738 240 0.187368 0.289749 > 249 16 13762 13746 220.767 228 0.18544 0.289761 > 250 16 13816 13800 220.748 216 0.234778 0.289711 > 251 16 13867 13851 220.681 204 0.299891 0.289799 > 252 16 13922 13906 220.678 220 0.178209 0.289802 > 253 16 13982 13966 220.754 240 0.15363 0.289719 > 254 16 14041 14025 220.814 236 0.319503 0.289642 > 255 16 14096 14080 220.811 220 0.15513 0.289653 > 256 16 14151 14135 220.807 220 0.191936 0.289682 > 257 16 14211 14195 220.882 240 0.42164 0.289601 > 258 16 14261 14245 220.801 200 0.429204 0.289667 > 259 16 14319 14303 220.844 232 0.152678 0.289632 > 2014-05-07 15:40:05.988372min lat: 0.101119 max lat: 1.31226 avg lat: 0.289618 > sec Cur ops started finished avg MB/s cur MB/s last lat avg lat > 260 16 14375 14359 220.856 224 0.31703 0.289618 > 261 16 14427 14411 220.806 208 0.205809 0.289642 > 262 16 14490 14474 220.925 252 0.32792 0.289559 > 263 16 14538 14522 220.815 192 0.242579 0.289644 > 264 16 14591 14575 220.781 212 0.324884 0.289696 > 265 16 14647 14631 220.793 224 0.293476 0.289649 > 266 16 14705 14689 220.835 232 0.184659 0.289608 > 267 16 14761 14745 220.847 224 0.347211 0.289595 > 268 16 14820 14804 220.903 236 0.417986 0.289545 > 269 16 14879 14863 220.959 236 0.157183 0.289504 > 270 16 14935 14919 220.97 224 0.203481 0.289445 > 271 16 14998 14982 221.084 252 0.281984 0.289334 > 272 16 15058 15042 221.154 240 0.19693 0.289247 > 273 16 15110 15094 221.105 208 0.322749 0.289256 > 274 16 15173 15157 221.218 252 0.276563 0.289142 > 275 16 15225 15209 221.17 208 0.529459 0.289165 > 276 16 15285 15269 221.238 240 0.513058 0.289092 > 277 16 15336 15320 221.175 204 0.261624 0.289186 > 278 16 15397 15381 221.257 244 0.216828 0.289069 > 279 16 15454 15438 221.281 228 0.239265 0.289058 > 2014-05-07 15:40:25.993548min lat: 0.101119 max lat: 1.31226 avg lat: 0.289203 > sec Cur ops started finished avg MB/s cur MB/s last lat avg lat > 280 16 15505 15489 221.219 204 0.278719 0.289203 > 281 16 15562 15546 221.243 228 0.423709 0.289144 > 282 16 15617 15601 221.238 220 0.117976 0.289073 > 283 16 15674 15658 221.262 228 0.215652 0.289111 > 284 16 15727 15711 221.229 212 0.240889 0.289105 > 285 16 15786 15770 221.281 236 0.235276 0.289058 > 286 16 15844 15828 221.318 232 0.2063 0.28903 > 287 16 15900 15884 221.327 224 0.150426 0.289063 > 288 16 15955 15939 221.322 220 0.163179 0.289015 > 289 16 16015 15999 221.387 240 0.22401 0.288967 > 290 16 16070 16054 221.382 220 0.419973 0.288929 > 291 16 16124 16108 221.363 216 0.270925 0.288894 > 292 16 16173 16157 221.276 196 0.181312 0.288978 > 293 16 16227 16211 221.258 216 0.364086 0.289128 > 294 16 16283 16267 221.267 224 0.136789 0.289044 > 295 16 16339 16323 221.276 224 0.178604 0.289074 > 296 16 16393 16377 221.258 216 0.374312 0.289017 > 297 16 16450 16434 221.281 228 0.320074 0.289071 > 298 16 16513 16497 221.384 252 0.226936 0.288992 > 299 16 16566 16550 221.352 212 0.182827 0.288901 > 2014-05-07 15:40:45.998624min lat: 0.101119 max lat: 1.31808 avg lat: 0.288874 > sec Cur ops started finished avg MB/s cur MB/s last lat avg lat > 300 16 16620 16604 221.334 216 0.330701 0.288874 > Total time run: 300.298736 > Total writes made: 16620 > Write size: 4194304 > Bandwidth (MB/sec): 221.380 > > Stddev Bandwidth: 20.1369 > Max bandwidth (MB/sec): 264 > Min bandwidth (MB/sec): 0 > Average Latency: 0.289054 > Stddev Latency: 0.125063 > Max latency: 1.31808 > Min latency: 0.101119 > > > ##################################################### > Here are testing results for librados ( I suppose c++?) with "rados -p pool put infile outfile": > In both cases write performance is below 30MB/sec. > > root at mir-daten-dev:~# ll /tmp/testradosput_X.mp4 > -rw-r--r-- 1 root root 1368187617 May 8 09:51 /tmp/testradosput_X.mp4 > [devel at transcoder-devel-2-backup julian]$ ll test_X.mp4 > -rw-r--r-- 1 devel nfsnobody 1368187617 2. Mai 12:22 test_X.mp4 > > [devel at transcoder-devel-2-backup julian]$ time rados -p mirdatatest put video_X.mp4 test_X.mp4 > > real 0m54.720s > user 0m0.647s > sys 0m1.954s > [devel at transcoder-devel-2-backup julian]$ time rados -p mirdatatest put video_X.mp4 test_X.mp4 > > real 0m52.118s > user 0m0.559s > sys 0m1.709s > > root at mir-daten-dev:~# time rados -p mirdatatest put testradosput_X.mp4 /tmp/testradosput_X.mp4 > > real 0m44.860s > user 0m0.844s > sys 0m1.402s > root at mir-daten-dev:~# time rados -p mirdatatest put testradosput_X.mp4 /tmp/testradosput_X.mp4 > > real 0m44.668s > user 0m0.791s > sys 0m1.495s > > > ##################################################### > Here are the results for librados clients written in Java (using OpenJDK 1.7) > > [mir at mir-daten-dev ~]$ time java CephClient WRITE ff161ab4-f700-471e-913d-3ec8ddfe8687_X.mp4 "mirdatatest/test_from_daten_dev_with_java_X.mp4" > Connected to the cluster. > start writing > wrote 33554432 > wrote 1368187617 > wrote file ff161ab4-f700-471e-913d-3ec8ddfe8687_X.mp4 of size 1368187617 in 38458 milliseconds. speed is 35 MB/s > > real 0m38.544s > user 0m1.790s > sys 0m1.407s > > Doing two write commands at once doesnt have an effect on speed: > [mir at mir-daten-dev ~]$ time java CephClient WRITE ff161ab4-f700-471e-913d-3ec8ddfe8687_X.mp4 "mirdatatest/test_from_daten_dev_with_java_X.mp4" & time java CephClient WRITE ff161ab4-f700-471e-913d-3ec8ddfe8687_X.mp4 "mirdatatest/test_farom_daten_dev_with_java_X.mp4"& > > wrote file ff161ab4-f700-471e-913d-3ec8ddfe8687_X.mp4 of size 1368187617 in 39200 milliseconds. speed is 34 MB/s > > real 0m39.291s > user 0m2.121s > sys 0m1.560s > > wrote file ff161ab4-f700-471e-913d-3ec8ddfe8687_X.mp4 of size 1368187617 in 41972 milliseconds. speed is 32 MB/s > > real 0m42.063s > user 0m1.784s > sys 0m1.415s > > ##################################################### > Here are the results for S3-Gateway for ceph with another java client: > > [devel at transcoder-devel-2-backup julian]$ time java -cp "libs/*:." CephS3Client PUT test_X.mp4 "s3test/transfer-test-3.mp4" > OpenJDK 64-Bit Server VM warning: Insufficient space for shared memory file: > /tmp/hsperfdata_devel/27950 > Try using the -Djava.io.tmpdir= option to select an alternate temp location. > > transter in percent 0.8011252158555385 > transter in percent 9.799114561055115 > transter in percent 18.29774783000393 > transter in percent 27.948372814428126 > transter in percent 35.83089409001719 > transter in percent 47.13346561446039 > transter in percent 57.863035022630235 > transter in percent 62.27161870300716 > transter in percent 70.26670465765515 > transter in percent 77.31636705786674 > transter in percent 87.52682169612123 > transter in percent 99.54070619248996 > transter in percent 100.0 > transter in percent 100.0 > transter in percent 100.0 > transter in percent 100.0 > transter in percent 100.0 > transter in percent 100.0 > > real 0m21.642s > user 0m26.108s > sys 0m18.269s > > [devel at transcoder-devel-2-backup julian]$ ls -la test_X.mp4 > -rw-r--r-- 1 devel nfsnobody 1368187617 2. Mai 12:22 test_X.mp4 > -------------------------------------------------------------------------------------------------- > Bayerischer Rundfunk; Rundfunkplatz 1; 80335 M?nchen > Telefon: +49 89 590001; E-Mail: info at BR.de; Website: http://www.BR.de > _______________________________________________ > ceph-users mailing list > ceph-users at lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |