Hello, scrub settings will only apply to new scrubs, not running ones, as you found out. On Mon, 11 Jul 2016 15:37:49 +0300 K K wrote: > > I have tested windows instance Crystal Disk Mark. Result is: > Again, when running a test like this, check with atop/iostat how your OSDs/HDDs are doing > Sequential Read : 43.049 MB/s > Sequential Write : 45.181 MB/s > Random Read 512KB : 78.660 MB/s > Random Write 512KB : 39.292 MB/s > Random Read 4KB (QD=1) : 3.511 MB/s [ 857.3 IOPS] > Random Write 4KB (QD=1) : 1.380 MB/s [ 337.0 IOPS] > Random Read 4KB (QD=32) : 32.220 MB/s [ 7866.1 IOPS] > Random Write 4KB (QD=32) : 12.564 MB/s [ 3067.4 IOPS] > Test : 4000 MB [D: 97.5% (15699.7/16103.1 GB)] (x3) > These numbers aren't all that bad, with your network and w/o SSD journals the 4KB ones are pretty much on par. You may get better read performance by permanently enabling read-ahead, as per: http://docs.ceph.com/docs/hammer/rbd/rbd-config-ref/ Windows may have native settings to do that, but I know zilch about that. Christian > >Понедельник, 11 июля 2016, 12:38 +05:00 от Christian Balzer <chibi@xxxxxxx>: > > > > > >Hello, > > > >On Mon, 11 Jul 2016 09:54:59 +0300 K K wrote: > > > >> > >> > I hope the fastest of these MONs (CPU and storage) has the lowest IP > >> > number and thus is the leader. > >> no, the lowest IP has slowest CPU. But zabbix didn't show any load at all mons. > > > >In your use case and configuration no surprise, but again, the lowest IP > >will be leader by default and thus the busiest. > > > >> > Also what Ceph, OS, kernel version? > >> > >> ubuntu 16.04 kernel 4.4.0-22 > >> > >Check the ML archives, I remember people having performance issues with the > >4.4 kernels. > > > >Still don't know your Ceph version, is it the latest Jewel? > > > >> > Two GbE ports, given the "frontend" up there with the MON description I > >> > assume that's 1 port per client (front) and cluster (back) network? > >> yes, one GbE for ceph client, one GbE for back network. > >OK, so (from a single GbE client) 100MB/s at most. > > > >> > Is there any other client on than that Windows VM on your Ceph cluster? > >> Yes, another one instance but without load. > >OK. > > > >> > Is Ceph understanding this now? > >> > Other than that, the queue options aren't likely to do much good with pure > >> >HDD OSDs. > >> > >> I can't find those parameter in running config: > >> ceph --admin-daemon /var/run/ceph/ceph-mon.block01.asok config show|grep "filestore_queue" > > > >These are OSD parameters, you need to query an OSD daemon. > > > >> "filestore_queue_max_ops": "3000", > >> "filestore_queue_max_bytes": "1048576000", > >> "filestore_queue_max_delay_multiple": "0", > >> "filestore_queue_high_delay_multiple": "0", > >> "filestore_queue_low_threshhold": "0.3", > >> "filestore_queue_high_threshhold": "0.9", > >> > That should be 512, 1024 really with one RBD pool. > >> > >> Yes, I know. Today for test I added scbench pool with 128 pg > >> There are output status and osd tree: > >> ceph status > >> cluster 830beb43-9898-4fa9-98c1-ee04c1cdf69c > >> health HEALTH_OK > >> monmap e6: 3 mons at {block01=10.30.9.21:6789/0,object01=10.30.9.129:6789/0,object02=10.30.9.130:6789/0} > >> election epoch 238, quorum 0,1,2 block01,object01,object02 > >> osdmap e6887: 18 osds: 18 up, 18 in > >> pgmap v9738812: 1280 pgs, 3 pools, 17440 GB data, 4346 kobjects > >> 35049 GB used, 15218 GB / 50267 GB avail > >> 1275 active+clean > >> 3 active+clean+scrubbing+deep > >> 2 active+clean+scrubbing > >> > >Check the ML archives and restrict scrubs to off-peak hours as well as > >tune things to keep their impact low. > > > >Scrubbing is a major performance killer, especially on non-SSD journal > >OSDs and with older Ceph versions and/or non-tuned parameters: > >--- > >osd_scrub_end_hour = 6 > >osd_scrub_load_threshold = 2.5 > >osd_scrub_sleep = 0.1 > >--- > > > >> client io 5030 kB/s rd, 1699 B/s wr, 19 op/s rd, 0 op/s wr > >> > >> ID WEIGHT TYPE NAME UP/DOWN REWEIGHT PRIMARY-AFFINITY > >> -1 54.00000 root default > >> -2 27.00000 host cn802 > >> 0 3.00000 osd.0 up 1.00000 1.00000 > >> 2 3.00000 osd.2 up 1.00000 1.00000 > >> 4 3.00000 osd.4 up 1.00000 1.00000 > >> 6 3.00000 osd.6 up 0.89995 1.00000 > >> 8 3.00000 osd.8 up 1.00000 1.00000 > >> 10 3.00000 osd.10 up 1.00000 1.00000 > >> 12 3.00000 osd.12 up 0.89999 1.00000 > >> 16 3.00000 osd.16 up 1.00000 1.00000 > >> 18 3.00000 osd.18 up 0.90002 1.00000 > >> -3 27.00000 host cn803 > >> 1 3.00000 osd.1 up 1.00000 1.00000 > >> 3 3.00000 osd.3 up 0.95316 1.00000 > >> 5 3.00000 osd.5 up 1.00000 1.00000 > >> 7 3.00000 osd.7 up 1.00000 1.00000 > >> 9 3.00000 osd.9 up 1.00000 1.00000 > >> 11 3.00000 osd.11 up 0.95001 1.00000 > >> 13 3.00000 osd.13 up 1.00000 1.00000 > >> 17 3.00000 osd.17 up 0.84999 1.00000 > >> 19 3.00000 osd.19 up 1.00000 1.00000 > >> > Wrong way to test this, test it from a monitor node, another client node > >> > (like your openstack nodes). > >> > In your 2 node cluster half of the reads or writes will be local, very > >> > much skewing your results. > >> I have been tested from copmute node also and have same result. 80-100Mb/sec > >> > >That's about as good as it gets (not 148MB/s, though!). > >But rados bench is not the same as real client I/O. > > > >> > Very high max latency, telling us that your cluster ran out of steam at > >> some point. > >> > >> I copying data from my windows instance right now. > > > >Re-do any testing when you've stopped all scrubbing. > > > >> > I'd de-frag anyway, just to rule that out. > >> > >> > >> >When doing your tests or normal (busy) operations from the client VM, run > >> > atop on your storage nodes and observe your OSD HDDs. > >> > Do they get busy, around 100%? > >> > >> Yes, high IO load (600-800 io). But this is very strange on SATA HDD. All HDD have own OSD daemon and presented in OS as hardware RAID0(each block node have hardware RAID). Example: > > > >Your RAID controller and its HW cache are likely to help with that speed, > >also all of these are reads, most likely the scrubs above, not a single > >write to be seen. > > > >> avg-cpu: %user %nice %system %iowait %steal %idle > >> 1.44 0.00 3.56 17.56 0.00 77.44 > >> Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util > >> sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 > >> sdc 0.00 0.00 649.00 0.00 82912.00 0.00 255.51 8.30 12.74 12.74 0.00 1.26 81.60 > >> sdd 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 > >> sde 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 > >> sdf 0.00 0.00 761.00 0.00 94308.00 0.00 247.85 8.66 11.26 11.26 0.00 1.18 90.00 > >> sdg 0.00 0.00 761.00 0.00 97408.00 0.00 256.00 7.80 10.22 10.22 0.00 1.01 76.80 > >> sdh 0.00 0.00 801.00 0.00 102344.00 0.00 255.54 8.05 10.05 10.05 0.00 0.96 76.80 > >> sdi 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 > >> sdj 0.00 0.00 537.00 0.00 68736.00 0.00 256.00 5.54 10.26 10.26 0.00 0.98 52.80 > >> > >> > >> > Check with iperf or NPtcp that your network to the clients from the > >> > storage nodes is fully functional. > >> The network have been tested by iperf. 950-970Mbit among all nodes in clustes (openstack and ceph) > > > >Didn't think it was that, one thing off the list to check. > > > >Christian > > > >Понедельник, 11 июля 2016, 10:58 +05:00 от Christian Balzer > >< chibi@xxxxxxx >: > >> > > >> > > >> >Hello, > >> > > >> >On Mon, 11 Jul 2016 07:35:02 +0300 K K wrote: > >> > > >> >> > >> >> Hello, guys > >> >> > >> >> I to face a task poor performance into windows 2k12r2 instance running > >> >> on rbd (openstack cluster). RBD disk have a size 17Tb. My ceph cluster > >> >> consist from: > >> >> - 3 monitors nodes (Celeron G530/6Gb RAM, DualCore E6500/2Gb RAM, > >> >> Core2Duo E7500/2Gb RAM). Each node have 1Gbit network to frontend subnet > >> >> od Ceph cluster > >> > > >> >I hope the fastest of these MONs (CPU and storage) has the lowest IP > >> >number and thus is the leader. > >> > > >> >Also what Ceph, OS, kernel version? > >> > > >> >> - 2 block nodes (Xeon E5620/32Gb RAM/2*1Gbit NIC). Each node have > >> >> 2*500Gb HDD for operation system and 9*3Tb SATA HDD (WD SE). Total 18 > >> >> OSD daemons on 2 nodes. > >> > > >> >Two GbE ports, given the "frontend" up there with the MON description I > >> >assume that's 1 port per client (front) and cluster (back) network? > >> > > >> >>Journals placed on same HDD as a rados data. I > >> >> know that better using for those purpose separate SSD disk. > >> >Indeed... > >> > > >> >>When I test > >> >> new windows instance performance was good (read/write something about > >> >> 100Mb/sec). But after I copied 16Tb data to windows instance read > >> >> performance has down to 10Mb/sec. Type of data on VM - image and video. > >> >> > >> >100MB/s would be absolute perfect with the setup you have, assuming no > >> >contention (other clients). > >> > > >> >Is there any other client on than that Windows VM on your Ceph cluster? > >> > > >> >> ceph.conf on client side: > >> >> [global] > >> >> auth cluster required = cephx > >> >> auth service required = cephx > >> >> auth client required = cephx > >> >> filestore xattr use omap = true > >> >> filestore max sync interval = 10 > >> >> filestore queue max ops = 3000 > >> >> filestore queue commiting max bytes = 1048576000 > >> >> filestore queue commiting max ops = 5000 > >> >> filestore queue max bytes = 1048576000 > >> >> filestore queue committing max ops = 4096 > >> >> filestore queue committing max bytes = 16 MiB > >> > ^^^ > >> >Is Ceph understanding this now? > >> >Other than that, the queue options aren't likely to do much good with pure > >> >HDD OSDs. > >> > > >> >> filestore op threads = 20 > >> >> filestore flusher = false > >> >> filestore journal parallel = false > >> >> filestore journal writeahead = true > >> >> journal dio = true > >> >> journal aio = true > >> >> journal force aio = true > >> >> journal block align = true > >> >> journal max write bytes = 1048576000 > >> >> journal_discard = true > >> >> osd pool default size = 2 # Write an object n times. > >> >> osd pool default min size = 1 > >> >> osd pool default pg num = 333 > >> >> osd pool default pgp num = 333 > >> >That should be 512, 1024 really with one RBD pool. > >> > http://ceph.com/pgcalc/ > >> > > >> >> osd crush chooseleaf type = 1 > >> >> > >> >> [client] > >> >> rbd cache = true > >> >> rbd cache size = 67108864 > >> >> rbd cache max dirty = 50331648 > >> >> rbd cache target dirty = 33554432 > >> >> rbd cache max dirty age = 2 > >> >> rbd cache writethrough until flush = true > >> >> > >> >> > >> >> rados bench show from block node show: > >> >Wrong way to test this, test it from a monitor node, another client node > >> >(like your openstack nodes). > >> >In your 2 node cluster half of the reads or writes will be local, very > >> >much skewing your results. > >> > > >> >> rados bench -p scbench 120 write --no-cleanup > >> > > >> >Default tests with 4MB "blocks", what are the writes or reads from you > >> >client VM like? > >> > > >> >> Total time run: 120.399337 > >> >> Total writes made: 3538 > >> >> Write size: 4194304 > >> >> Object size: 4194304 > >> >> Bandwidth (MB/sec): 117.542 > >> >> Stddev Bandwidth: 9.31244 > >> >> Max bandwidth (MB/sec): 148 > >> > ^^^ > >> >That wouldn't be possible from an external client. > >> > > >> >> Min bandwidth (MB/sec): 92 > >> >> Average IOPS: 29 > >> >> Stddev IOPS: 2 > >> >> Max IOPS: 37 > >> >> Min IOPS: 23 > >> >> Average Latency(s): 0.544365 > >> >> Stddev Latency(s): 0.35825 > >> >> Max latency(s): 5.42548 > >> >Very high max latency, telling us that your cluster ran out of steam at > >> >some point. > >> > > >> >> Min latency(s): 0.101533 > >> >> > >> >> rados bench -p scbench 120 seq > >> >> Total time run: 120.880920 > >> >> Total reads made: 1932 > >> >> Read size: 4194304 > >> >> Object size: 4194304 > >> >> Bandwidth (MB/sec): 63.9307 > >> >> Average IOPS 15 > >> >> Stddev IOPS: 3 > >> >> Max IOPS: 25 > >> >> Min IOPS: 5 > >> >> Average Latency(s): 0.999095 > >> >> Max latency(s): 8.50774 > >> >> Min latency(s): 0.0391591 > >> >> > >> >> rados bench -p scbench 120 rand > >> >> Total time run: 121.059005 > >> >> Total reads made: 1920 > >> >> Read size: 4194304 > >> >> Object size: 4194304 > >> >> Bandwidth (MB/sec): 63.4401 > >> >> Average IOPS: 15 > >> >> Stddev IOPS: 4 > >> >> Max IOPS: 26 > >> >> Min IOPS: 1 > >> >> Average Latency(s): 1.00785 > >> >> Max latency(s): 6.48138 > >> >> Min latency(s): 0.038925 > >> >> > >> >> On XFS partitions fragmentation no more than 1% > >> >I'd de-frag anyway, just to rule that out. > >> > > >> >When doing your tests or normal (busy) operations from the client VM, run > >> >atop on your storage nodes and observe your OSD HDDs. > >> >Do they get busy, around 100%? > >> > > >> >Check with iperf or NPtcp that your network to the clients from the > >> >storage nodes is fully functional. > >> > > >> >Christian > >> >-- > >> >Christian Balzer Network/Systems Engineer > >> > chibi@xxxxxxx Global OnLine Japan/Rakuten Communications > >> > http://www.gol.com/ > >> > > > > > >-- > >Christian Balzer Network/Systems Engineer > >chibi@xxxxxxx Global OnLine Japan/Rakuten Communications > >http://www.gol.com/ > -- Christian Balzer Network/Systems Engineer chibi@xxxxxxx Global OnLine Japan/Rakuten Communications http://www.gol.com/ _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |