> -----Original Message----- > From: ceph-users [mailto:ceph-users-bounces@xxxxxxxxxxxxxx] On Behalf Of > Dominik Hannen > Sent: 29 April 2015 00:30 > To: Nick Fisk > Cc: ceph-users@xxxxxxxxxxxxxx > Subject: Re: Cost- and Powerefficient OSD-Nodes > > > It's all about the total latency per operation. Most IO sizes over > > 10GB don't make much difference to the Round Trip Time. But > > comparatively even 128KB IO's over 1GB take quite a while. For example > > ping a host with a payload of 64k over 1GB and 10GB networks and look > > at the difference in times. Now double this for Ceph (Client->Prim > > OSD->Sec OSD) > > > > When you are using SSD journals you normally end up with write latency > > of 3-4ms over 10GB, 1GB networking will probably increase this by > > another 2-4ms. IOPs=1000/latency > > > > I guess it all really depends on how important performance is > > I recon we are talking about single-threaded IOPs? It looks like 10ms latency > is in the worst-case region.. 100 IOPs will do fine. > > At least in my understanding heavily multi-threaded load should be able to > get higher IOPs regardless of latency? Yes as the queue depth increases so will total IOPs, but I found it quite hard to get above 40-50MB/s unless doing large block sizes > > Some presentation material suggested that the adverse effects of higher > latency, due to 1Gbit, begin above IO sizes of 2k, maybe there is room to > tune IOPs hungry applications/vms accordingly. > > > Just had a look and the Seagate Surveillance disks spin at 7200RPM > > (missed that you put that there), whereas the WD ones that I am > > familiar with spin at 5400rpm, so not as bad as I thought. > > > > So probably ok to use, but I don't see many people using them for > > Ceph/ generic NAS so can't be sure there's no hidden gotchas. > > I am not sure how trustworthy newegg-reviews are, but somehow I get > some doubts about them now. > I guess it does not matter that much, at least if not more than a disk a month > is failing? The 3-year warranty gives some hope.. > > Are there some cost-efficient HDDs that someone can suggest? (Most likely > 3TB drives, that seems to be the sweet-spot at the moment.) I'm using WD Red Pro (non pro's are slower), reasonable cost and perform pretty much the same as the enterprise line drives > > > Sorry nothing in detail, I did actually build a ceph cluster on the > > same 8 core CPU as you have listed. I didn't have any performance > > problems but I do remember with SSD journals when doing high queue > > depth writes I could get the CPU quite high. It's like what I said > > before about the 1vs10Gb networking, how important is performance, If > > using this CPU gives you an extra 1ms of latency per OSD, is that > acceptable? > > > > Agree 12cores (guessing 2.5Ghz each) will be an overkill for just 12 > > OSDs. I have a very similar spec and see exactly the same as you, but > > will change the nodes to 1CPU each when I expand and use the spare > > CPU's for the new nodes. > > > > I'm using this:- > > > > http://www.supermicro.nl/products/system/4U/F617/SYS-F617H6- > FTPTL_.cfm > > > > Mainly because of rack density, which I know doesn't apply to you. But > > the fact they share PSU's/Rails/Chassis helps reduce power a bit and > > drives down cost > > > > I can get 14 disks in each and they have 10GB on board. The SAS > > controller is flashable to JBOD mode. > > > > Maybe one of the other Twin solutions might be suitable? > > I did consider that exact model (It was mentioned on the list some time ago) I > could get about the same effective storage-capacity with it, but 10G- > Networking is just too expensive on the Switch-side. > > Also those nodes and 10G-Switches consume a lot more power. > > By my estimates and numbers I found, the Avoton-Nodes should run at > about 55W each. The Switches (EX3300) according to tech-specs would need > 76W at max each. > Have you worked out how many watts per disk that is though? 55W/3Disks = 18.3W per disk My Chassis at the moment 170w/12disks = 13.3W per disk > ___ > Dominik > _______________________________________________ > ceph-users mailing list > ceph-users@xxxxxxxxxxxxxx > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |