|
Hi, Could you take a look on my problem. It’s about high latency on my OSDs on HP G8 servers (ceph01, ceph02 and ceph03). When I run a rados bench for 60 sec, the results are surprising : after a few seconds, there is no traffic, then it’s resume, etc. Finally, the maximum latency is high and VM’s disks freeze lot. #rados bench -p pool-test-g8 60 write Maintaining 16 concurrent writes of 4194304 bytes for up to 60 seconds or 0 objects Object prefix: benchmark_data_ceph02_56745 sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 0 0 0 0 0 0 - 0 1 16 82 66 263.959 264 0.0549584 0.171148 2 16 134 118 235.97 208 0.344873 0.232103 3 16 189 173 230.639 220 0.015583 0.24581 4 16 248 232 231.973 236 0.0704699 0.252504 5 16 306 290 231.974 232 0.0229872 0.258343 6 16 371 355 236.64 260 0.27183 0.255469 7 16 419 403 230.26 192 0.0503492 0.263304 8 16 460 444 221.975 164 0.0157241 0.261779 9 16 506 490 217.754 184 0.199418 0.271501 10 16 518 502 200.778
48 0.0472324 0.269049 11 16 518 502 182.526
0 - 0.269049 12 16 556 540 179.981
76 0.100336 0.301616 13 16 607 591 181.827 204 0.173912 0.346105 14 16 655 639 182.552 192 0.0484904 0.339879 15 16 683 667 177.848 112 0.0504184 0.349929 16 16 746 730 182.481 252 0.276635 0.347231 17 16 807 791 186.098 244 0.391491 0.339275 18 16 845 829 184.203 152 0.188608 0.342021 19 16 850 834 175.561 20 0.960175 0.342717 2015-05-28 17:09:48.397376min lat: 0.013532 max lat: 6.28387 avg lat: 0.346987 sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 20 16 859 843 168.582 36 0.0182246 0.346987 21 16 863 847 161.316 16 3.18544 0.355051 22 16 897 881 160.165 136 0.0811037 0.371209 23 16 901 885 153.897 16 0.0482124 0.370793 24 16 943 927 154.484 168 0.63064 0.397204 25 15 997 982 157.104 220 0.0933448 0.392701 26 16 1058 1042 160.291 240 0.166463 0.385943 27 16 1088 1072 158.798 120 1.63882 0.388568 28 16 1125 1109 158.412 148 0.0511479 0.38419 29 16 1155 1139 157.087 120 0.162266 0.385898 30 16 1163 1147 152.917 32 0.0682181 0.383571 31 16 1190 1174 151.468 108 0.0489185 0.386665 32 16 1196 1180 147.485 24 2.95263 0.390657 33 16 1213 1197 145.076 68 0.0467788 0.389299 34 16 1265 1249 146.926 208 0.0153085 0.420687 35 16 1332 1316 150.384 268 0.0157061 0.42259 36 16 1374 1358 150.873 168 0.251626 0.417373 37 16 1402 1386 149.822 112 0.0475302 0.413886 38 16 1444 1428 150.3 168 0.0507577 0.421055 39 16 1500 1484 152.189 224 0.0489163 0.416872 2015-05-28 17:10:08.399434min lat: 0.013532 max lat: 9.26596 avg lat: 0.415296 sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 40 16 1530 1514 151.384 120 0.951713 0.415296 41 16 1551 1535 149.741 84 0.0686787 0.416571 42 16 1606 1590 151.413 220 0.0826855 0.41684 43 16 1656 1640 152.542 200 0.0706539 0.409974 44 16 1663 1647 149.712
28 0.046672 0.408476 45 16 1685 1669 148.34
88 0.0989566 0.424918 46 16 1707 1691 147.028
88 0.0490569 0.421116 47 16 1707 1691 143.9
0 - 0.421116 48 16 1707 1691 140.902
0 - 0.421116 49 16 1720 1704 139.088
17.3333 0.0480335 0.428997 50 16 1752 1736 138.866 128 0.053219 0.4416 51 16 1786 1770 138.809 136 0.602946 0.440357 52 16 1810 1794 137.986 96 0.0472518 0.438376 53 16 1831 1815 136.967 84 0.0148999 0.446801 54 16 1831 1815 134.43
0 - 0.446801 55 16 1853 1837 133.586 44 0.0499486 0.455561 56 16 1898 1882 134.415 180 0.0566593 0.461019 57 16 1932 1916 134.442 136 0.0162902 0.454385 58 16 1948 1932 133.227 64 0.62188 0.464403 59 16 1966 1950 132.19 72 0.563613 0.472147 2015-05-28 17:10:28.401525min lat: 0.013532 max lat: 12.4828 avg lat: 0.472084 sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 60 16 1983 1967 131.12 68 0.030789 0.472084 61 16 1984 1968 129.036 4 0.0519125 0.471871 62 16 1984 1968 126.955
0 - 0.471871 63 16 1984 1968 124.939
0 - 0.471871 64 14 1984 1970 123.112
2.66667 4.20878 0.476035 Total time run: 64.823355 Total writes made: 1984 Write size: 4194304 Bandwidth (MB/sec): 122.425 Stddev Bandwidth: 85.3816 Max bandwidth (MB/sec): 268 Min bandwidth (MB/sec): 0 Average Latency: 0.520956 Stddev Latency: 1.17678 Max latency:
12.4828 Min latency: 0.013532 I have installed a new ceph06 box which has best latencies but hardware is different (RAID card, disks, …). All OSDs are formatted in XFS but are mounted differently : -
On ceph box with high latency (strip size of raid logical disk is 256KB) : /dev/sdd1 on /var/lib/ceph/osd/ceph-4 type xfs (rw,noatime,attr2,inode64,logbsize=256k,sunit=512,swidth=512,noquota) -
On ceph box with low latency (strip size of raid logical disk is 128KB) : /dev/sdc1 on /var/lib/ceph/osd/ceph-9 type xfs (rw,noatime,attr2,inode64,noquota) Thread on Proxmox Forum :
http://forum.proxmox.com/threads/22206-Ceph-High-apply-latency-on-OSD-causes-poor-performance-on-VM If you have any idea ... Thanks. Franck ALLOUIS De : Franck Allouis
Hi, Since we have installing our new Ceph Cluster, we have frequently high apply latency on OSDs (near 200 ms to 1500 ms), while commit latency is continuously at 0 ms ! In Ceph documentation, when you run the command "ceph osd perf", the fs_commit_latency is generally higher than fs_apply_latency. For us it's the opposite. The phenomenon has increased since we changed the Ceph version (migrate Giant 0.87.1 to Hammer 0.94.1) The consequence is that our Windows VMs are very slow. Does anyone could tell us if our configuration is good or not, and in what direction investigate ? # ceph osd perf osd fs_commit_latency(ms) fs_apply_latency(ms) 0 0 62 1 0 193 2 0 88 3 0 269 4 0 1055 5 0 322 6 0 272 7 0 116 8 0 653 9 0 4 10 0 1 11 0 7 12 0 4 Different informations in our configuration : - Proxmox 3.4-6 - kernel : 3.10.0-10-pve - CEPH : - Hammer 0.94.1 - 3 hosts with 3 OSDs of 4 TB (9 OSDs) + 1 SSD of 500 GB per host for journals - 1 host with 4 OSDs of 300 GB (4 OSDs) + 1 SSD of 500 GB for journals - OSD Tree : # ceph osd tree ID WEIGHT TYPE NAME UP/DOWN REWEIGHT PRIMARY-AFFINITY -1 33.83995 root default -6 22.91995 room salle-dr -2 10.92000 host ceph01 0 3.64000 osd.0 up 1.00000 1.00000 2 3.64000 osd.2 up 1.00000 1.00000 1 3.64000 osd.1 up 1.00000 1.00000 -3 10.92000 host ceph02 3 3.64000 osd.3 up 1.00000 1.00000 4 3.64000 osd.4 up 1.00000 1.00000 5 3.64000 osd.5 up 1.00000 1.00000 -5 1.07996 host ceph06 9 0.26999 osd.9 up 1.00000 1.00000 10 0.26999 osd.10 up 1.00000 1.00000 11 0.26999 osd.11 up 1.00000 1.00000 12 0.26999 osd.12 up 1.00000 1.00000 -7 10.92000 room salle-log -4 10.92000 host ceph03 6 3.64000 osd.6 up 1.00000 1.00000 7 3.64000 osd.7 up 1.00000 1.00000 8 3.64000 osd.8 up 1.00000 1.00000 - ceph.conf : [global] auth client required = cephx auth cluster required = cephx auth service required = cephx auth supported = cephx cluster network = 10.10.1.0/24 filestore xattr use omap = true fsid = 2dbbec32-a464-4bc5-bb2b-983695d1d0c6 keyring = /etc/pve/priv/$cluster.$name.keyring mon osd adjust heartbeat grace = true mon osd down out subtree limit = host osd disk threads = 24 osd heartbeat grace = 10 osd journal size = 5120 osd max backfills = 1 osd op threads = 24 osd pool default min size = 1 osd recovery max active = 1 public network = 192.168.80.0/24 [osd] keyring = /var/lib/ceph/osd/ceph-$id/keyring [mon.0] host = ceph01 mon addr = 192.168.80.41:6789 [mon.1] host = ceph02 mon addr = 192.168.80.42:6789 [mon.2] host = ceph03 mon addr = 192.168.80.43:6789 Thanks. Best regards Franck ALLOUIS
NOTICE: This e-mail (including any attachments) may contain information that is private, confidential or legally privileged information or material and is intended solely for the use of the addressee(s). If you receive this e-mail in error, please delete it from your system without copying it and immediately notify the sender(s) by reply e-mail. Any unauthorized use or disclosure of this message is strictly prohibited. STEF does not guarantee the integrity of this transmission and may therefore never be liable if the message is altered or falsified nor for any virus, interception or damage to your system.
AVIS : Ce message (y compris toutes pièces jointes) peut contenir des informations privées, confidentielles et est pour l'usage du(es) seul(s) destinataire(s). Si vous avez reçu ce message par erreur, merci d'en avertir l'expéditeur par retour d'email immédiatement et de procéder à la destruction de l'ensemble des éléments reçus, dont vous ne devez garder aucune copie. Toute diffusion, utilisation ou copie de ce message ou des renseignements qu'il contient par une personne autre que le(les) destinataire(s) désigné(s) est interdite. STEF ne garantit pas l'intégrité de cette transmission et ne saurait être tenu responsable du message, de son contenu, de toute modification ou falsification, d’une interception ou de dégâts à votre système. |
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
