-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 Resending sans attachment... A picture is worth a thousand words: http://robert.leblancnet.us/files/s3610-load-test-20160224.png The red lines are the m600s IO time (dotted) and IOPs (solid) and our baseline s3610s in green and our test set of s3610s in blue. We used weighting to manipulate how many PGs each SSD took. The m600s are 1TB while the s3610s are 800GBs and we only have the m600s about half filled. So we weighted the s3610s individually until they were about ~40GBs within the m600s. We did the same weighting to achieve similar percentage usage and 80% usage. This graph is stepping from 50% to 70% and finally very close to 80%. We have two production clusters currently, third one will be built in the next month all about the same size. 16 nodes, 3 - 1TB m600 drives and 9 - 4TB HGST HDDs, single E5-2640v2 and 64 GB RAM dual 40 Gigabit Ethernet ports, direct attached SATA. These clusters normally service 12K IOPs with bursts up to 22K IOPs all RBD. I've seen a peak of 64K IOPs from client traffic. -----BEGIN PGP SIGNATURE----- Version: Mailvelope v1.3.6 Comment: https://www.mailvelope.com wsFcBAEBCAAQBQJW0OgCCRDmVDuy+mK58QAAp+4P/0HJ+UU3gaAdRXyELCg5 mLifFliWYDFuabP+K5aI6mBn4qlF/1BAe6d9K8Zrcz+nZvXP+BcSEd1puUAW GIy+5O3xJkDUM5O9lAN+jIqw0X7ple2xni3Q5/fKwgGpD1TuEjGEnZlFfRJC 8HWfw6rnL+J7WEirhhXrk+NmOvLJRaozROuzKmKcbBVS2oVtrhOPA7eiNrUz NhN/YbvArGrQFneBO39Tp3YPn8cJ2nVgwv6eru9nnrvkEUD9nwJXlgyNf/NC IjX+LnKET0q0ouCFbjJGaUm4+tvNWWtXypYpcdC78RF+XMdsYHMKAikQ0aG7 7UbYlvf+DhFPqskXhpaB1+lEj+qyhYNwvaxt5QtYsuPK7zDfbV23ed/aiw7c 58q3ROMmIZGsVyBh3fR7EAvKcp3W8KQr9JUq3K3vLcWplNZsuvg4QZIx0ia2 YfGzBsJKugxMVGbmqnXCAcjUyEI/haoovIdMOVBWw8Uv8R9m2IpoNXgqsqi1 xJjIJ5pmiwMZliq2YLwcUy/6e3uPpPRYhgRkkHr167DDB0A5ijI7Y8Q5GX28 AeraQSHLBtOtyrXBcFCtZv2YVbl2juwwC2lNXHJZBd0b/iUDnrBA358U0crm +TqyYR7LoZiUjUMI0HZzjeyVIsST201R6uQ1Tv9b6DFAOxDMPWD7ViJLcSIO yAiI =vXUO -----END PGP SIGNATURE----- ---------------- Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1 On Fri, Feb 26, 2016 at 4:05 PM, Shinobu Kinjo <skinjo@xxxxxxxxxx> wrote: > Hello, > >> We started having high wait times on the M600s so we got 6 S3610s, 6 M500dcs, and 6 500 GB M600s (they have the SLC to MLC conversion that we thought might work better). > > Is it working better as you were expecting? > >> We have graphite gathering stats on the admin sockets for Ceph and the standard system stats. > > Very cool! > >> We weighted the drives so they had the same byte usage and let them run for a week or so, then made them the same percentage of used space, let them run a couple of weeks, then set them to 80% full and let them run a couple of weeks. > > Almost exactly same *byte* usage? I'm pretty interesting to how you realized that. > >> We compared IOPS and IO time of the drives to get our comparison. > > What is your feeling about the comparison? > >> This was done on live production clusters and not synthetic benchmarks. > > How large is your production the Ceph cluster? > > Rgds, > Shinobu > >> >> Hello, >> >> On Wed, 24 Feb 2016 22:56:15 -0700 Robert LeBlanc wrote: >> >> > We are moving to the Intel S3610, from our testing it is a good balance >> > between price, performance and longevity. But as with all things, do your >> > testing ahead of time. This will be our third model of SSDs for our >> > cluster. The S3500s didn't have enough life and performance tapers off >> > add it gets full. The Micron M600s looked good with the Sebastian journal >> > tests, but once in use for a while go downhill pretty bad. We also tested >> > Micron M500dc drives and they were on par with the S3610s and are more >> > expensive and are closer to EoL. The S3700s didn't have quite the same >> > performance as the S3610s, but they will last forever and are very stable >> > in terms of performance and have the best power loss protection. >> > >> That's interesting, how did you come to that conclusion and how did test >> it? >> Also which models did you compare? >> >> >> > Short answer is test them for yourself to make sure they will work. You >> > are pretty safe with the Intel S3xxx drives. The Micron M500dc is also >> > pretty safe based on my experience. It had also been mentioned that >> > someone has had good experience with a Samsung DC Pro (has to have both >> > DC and Pro in the name), but we weren't able to get any quick enough to >> > test so I can't vouch for them. >> > >> I have some Samsung DC Pro EVOs in production (non-Ceph, see that >> non-barrier thread). >> They do have issues with LSI occasionally, haven't gotten around to make >> that FS non-barrier to see if it fixes things. >> >> The EVOs are also similar to the Intel DC S3500s, meaning that they are >> not really suitable for Ceph due to their endurance. >> >> Never tested the "real" DC Pro ones, but they are likely to be OK. >> >> Christian >> >> > Sent from a mobile device, please excuse any typos. >> > On Feb 24, 2016 6:37 PM, "Shinobu Kinjo" <skinjo@xxxxxxxxxx> wrote: >> > >> > > Hello, >> > > >> > > There has been a bunch of discussion about using SSD. >> > > Does anyone have any list of SSDs describing which SSD is highly >> > > recommended, which SSD is not. >> > > >> > > Rgds, >> > > Shinobu >> > > _______________________________________________ >> > > ceph-users mailing list >> > > ceph-users@xxxxxxxxxxxxxx >> > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> > > >> >> >> -- >> Christian Balzer Network/Systems Engineer >> chibi@xxxxxxx Global OnLine Japan/Rakuten Communications >> http://www.gol.com/ >> _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |