å 2010-11-30äç 11:18 +0800ïGregory Farnumåéï > On Mon, Nov 29, 2010 at 6:55 PM, Jeff Wu <cpwu@xxxxxxxxxxxxx> wrote: > >> Could you run "ceph osd tell * bench", then run "ceph -w", and report > >> the results? (That'll just run local benchmarking on the OSD to report > >> the approximate write speed it's capable of.) > > > > I run six times for the command: "$ sudo ceph osd tell 0/1 bench " > > use "$ sudo ceph -w" to get the following results: > > > > osd0 : [INF] bench: wrote 1024 MB in blocks of 4096 KB in 12.906598 sec > > at 49775 KB/sec > > osd1 : [INF] bench: wrote 1024 MB in blocks of 4096 KB in 21.023294 sec > > at 49384 KB/sec > > osd0 : [INF] bench: wrote 1024 MB in blocks of 4096 KB in 12.834682 sec > > at 51535 KB/sec > > osd1 : [INF] bench: wrote 1024 MB in blocks of 4096 KB in 20.792697 sec > > at 37547 KB/sec > > osd0 : [INF] bench: wrote 1024 MB in blocks of 4096 KB in 13.058412 sec > > at 77191 KB/sec > > osd1 : [INF] bench: wrote 1024 MB in blocks of 4096 KB in 21.113612 sec > > at 47369 KB/sec > Okay, those are a bit slow but reasonable. Based on these I'd expect > you to generally manage about 40-50MB/s (since everything is > replicated; in a 2-disk configuration it'll just be the speed of your > slowest disk), assuming a properly configured system. Is "40-50MB/s" the speed that it run bench at local btrfs disk ? not the speed that run bench from client to osd server ? with this speed ,run bench from client to osd server ,will which get about 20~25MB/s( 40~50MB /2 )speed ? > > >> You can also run "rados -p data bench 60 write", and then "rados -p > >> data bench 0 seq" to get a simpler (better understood) performance > >> test. > > > > I run twice for the command: "rados -p data bench 60 write" , > > Get the results: > > $ sudo rados -p data bench 60 write > > .......................... > > .......................... > > > > Total time run: 76.182225 > > Total writes made: 121 > > Write size: 4194304 > > Bandwidth (MB/sec): 6.219 > > > > Average Latency: 13.3068 > > Max latency: 23.9986 > > Min latency: 7.01847 > WOAH. That's a lot of latency. Rather more than I'd expect to get just > from seek times in a non-journaled environment. What's the round-trip > time to ping the OSDs from your client? Are your disks okay? > Hi,i get the RTT and the disks are okay . 1. RTT: client ping OSD0: rtt min/avg/max/mdev = 0.103/0.124/0.143/0.016 ms client ping OSD1: rtt min/avg/max/mdev = 0.112/0.116/0.122/0.005 ms 2. disks 1) mon data save in sda4, which formatted with mkfs.btrfs $lsscsi [0:0:0:0] disk ATA WDC WD3200AAKS-7 02.0 /dev/sda $fdisk -l ............. /dev/sda4 17264 38913 173897568+ 83 Linux 2) OSD0 data save in sda5,which formatted with mkfs.btrfs ~$ lsscsi [0:0:0:0] disk ATA ST3320418AS CC45 /dev/sda $ fdisk -l ....................... /dev/sda5 18480 38913 164131649 83 Linux 3) OSD1 data save in sda5,which formatted with mkfs.btrfs $ lsscsi [0:0:0:0] disk ATA ST3160812AS 3.AD /dev/sda $ fdisk -l /dev/sda5 11185 19452 66405470+ 83 Linux more detail logs attached(see below) Thank you jeff.wu > > but run "$ sudo rados -p data bench 0 seq" > > fail to get the results, Maybe it's a bug,ceph version 0.23. > Oh right, sorry. We put in a quick fix to make the write benchmark > scale across multiple client writers and forgot to adjust the read > benchmark. *oops* thank you ! > -- > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in > the body of a message to majordomo@xxxxxxxxxxxxxxx > More majordomo info at http://vger.kernel.org/majordomo-info.html logs: ================================================================= transoft@ubuntu-mon0:/usr/local/etc/ceph$ ping 172.16.10.65 PING 172.16.10.65 (172.16.10.65) 56(84) bytes of data. 64 bytes from 172.16.10.65: icmp_req=1 ttl=64 time=0.127 ms 64 bytes from 172.16.10.65: icmp_req=2 ttl=64 time=0.121 ms 64 bytes from 172.16.10.65: icmp_req=3 ttl=64 time=0.127 ms 64 bytes from 172.16.10.65: icmp_req=4 ttl=64 time=0.124 ms 64 bytes from 172.16.10.65: icmp_req=5 ttl=64 time=0.128 ms 64 bytes from 172.16.10.65: icmp_req=6 ttl=64 time=0.128 ms 64 bytes from 172.16.10.65: icmp_req=7 ttl=64 time=0.127 ms 64 bytes from 172.16.10.65: icmp_req=8 ttl=64 time=0.126 ms 64 bytes from 172.16.10.65: icmp_req=9 ttl=64 time=0.126 ms 64 bytes from 172.16.10.65: icmp_req=10 ttl=64 time=0.128 ms 64 bytes from 172.16.10.65: icmp_req=11 ttl=64 time=0.120 ms 64 bytes from 172.16.10.65: icmp_req=12 ttl=64 time=0.127 ms 64 bytes from 172.16.10.65: icmp_req=13 ttl=64 time=0.120 ms 64 bytes from 172.16.10.65: icmp_req=14 ttl=64 time=0.125 ms 64 bytes from 172.16.10.65: icmp_req=15 ttl=64 time=0.131 ms 64 bytes from 172.16.10.65: icmp_req=16 ttl=64 time=0.125 ms 64 bytes from 172.16.10.65: icmp_req=17 ttl=64 time=0.124 ms 64 bytes from 172.16.10.65: icmp_req=18 ttl=64 time=0.103 ms 64 bytes from 172.16.10.65: icmp_req=19 ttl=64 time=0.124 ms 64 bytes from 172.16.10.65: icmp_req=20 ttl=64 time=0.125 ms 64 bytes from 172.16.10.65: icmp_req=21 ttl=64 time=0.143 ms 64 bytes from 172.16.10.65: icmp_req=22 ttl=64 time=0.127 ms 64 bytes from 172.16.10.65: icmp_req=23 ttl=64 time=0.117 ms ^C --- 172.16.10.65 ping statistics --- 23 packets transmitted, 23 received, 0% packet loss, time 21999ms rtt min/avg/max/mdev = 0.103/0.124/0.143/0.016 ms transoft@ubuntu-mon0:/usr/local/etc/ceph$ ping 172.16.10.42 PING 172.16.10.42 (172.16.10.42) 56(84) bytes of data. 64 bytes from 172.16.10.42: icmp_req=1 ttl=64 time=0.121 ms 64 bytes from 172.16.10.42: icmp_req=2 ttl=64 time=0.116 ms 64 bytes from 172.16.10.42: icmp_req=3 ttl=64 time=0.122 ms 64 bytes from 172.16.10.42: icmp_req=5 ttl=64 time=0.117 ms 64 bytes from 172.16.10.42: icmp_req=6 ttl=64 time=0.112 ms 64 bytes from 172.16.10.42: icmp_req=7 ttl=64 time=0.117 ms 64 bytes from 172.16.10.42: icmp_req=8 ttl=64 time=0.112 ms 64 bytes from 172.16.10.42: icmp_req=9 ttl=64 time=0.112 ms 64 bytes from 172.16.10.42: icmp_req=10 ttl=64 time=0.112 ms 64 bytes from 172.16.10.42: icmp_req=11 ttl=64 time=0.117 ms 64 bytes from 172.16.10.42: icmp_req=12 ttl=64 time=0.114 ms 64 bytes from 172.16.10.42: icmp_req=13 ttl=64 time=0.121 ms ^C --- 172.16.10.42 ping statistics --- 13 packets transmitted, 12 received, 7% packet loss, time 11999ms rtt min/avg/max/mdev = 0.112/0.116/0.122/0.005 ms transoft@ubuntu-mon0:/usr/local/etc/ceph$ transoft@ubuntu-mon0:~$ lsscsi [0:0:0:0] disk ATA WDC WD3200AAKS-7 02.0 /dev/sda transoft@ubuntu-mon0:~$ sudo fdisk -l [sudo] password for transoft: Disk /dev/sda: 320.1 GB, 320072933376 bytes 255 heads, 63 sectors/track, 38913 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x00006279 Device Boot Start End Blocks Id System /dev/sda1 * 1 4864 39061504 83 Linux /dev/sda2 4864 17264 99607553 5 Extended /dev/sda3 38914 38914 0+ 83 Linux /dev/sda4 17264 38913 173897568+ 83 Linux /dev/sda5 4864 17021 97654784 83 Linux /dev/sda6 17021 17264 1951744 82 Linux swap / Solaris Partition table entries are not in disk order transoft@ubuntu-mon0:~$ transoft@ubuntu-osd0:~$ lsscsi [0:0:0:0] disk ATA ST3320418AS CC45 /dev/sda transoft@ubuntu-osd0:~$ sudo fdisk -l [sudo] password for transoft: Disk /dev/sda: 320.1 GB, 320072933376 bytes 255 heads, 63 sectors/track, 38913 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x50000000 Device Boot Start End Blocks Id System /dev/sda1 * 1 6079 48827392 83 Linux /dev/sda2 6079 18237 97655808 83 Linux /dev/sda3 18237 18480 1952768 82 Linux swap / Solaris /dev/sda4 18480 38913 164131680+ 5 Extended /dev/sda5 18480 38913 164131649 83 Linux transoft@ubuntu-osd0:~$ transoft@ubuntu-osd1:~$ lsscsi [0:0:0:0] disk ATA ST3160812AS 3.AD /dev/sda transoft@ubuntu-osd1:~$ sudo fdisk -l [sudo] password for transoft: Disk /dev/sda: 160.0 GB, 160000000000 bytes 255 heads, 63 sectors/track, 19452 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0xc45cc45c Device Boot Start End Blocks Id System /dev/sda1 * 1 3648 29295616 83 Linux /dev/sda2 3648 10942 58593280 83 Linux /dev/sda3 10942 11185 1952768 82 Linux swap / Solaris /dev/sda4 11185 19452 66405502 5 Extended /dev/sda5 11185 19452 66405470+ 83 Linux transoft@ubuntu-osd1:~$ -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
