> It's all about the total latency per operation. Most IO sizes over 10GB > don't make much difference to the Round Trip Time. But comparatively even > 128KB IO's over 1GB take quite a while. For example ping a host with a > payload of 64k over 1GB and 10GB networks and look at the difference in > times. Now double this for Ceph (Client->Prim OSD->Sec OSD) > > When you are using SSD journals you normally end up with write latency of > 3-4ms over 10GB, 1GB networking will probably increase this by another > 2-4ms. IOPs=1000/latency > > I guess it all really depends on how important performance is I recon we are talking about single-threaded IOPs? It looks like 10ms latency is in the worst-case region.. 100 IOPs will do fine. At least in my understanding heavily multi-threaded load should be able to get higher IOPs regardless of latency? Some presentation material suggested that the adverse effects of higher latency, due to 1Gbit, begin above IO sizes of 2k, maybe there is room to tune IOPs hungry applications/vms accordingly. > Just had a look and the Seagate Surveillance disks spin at 7200RPM (missed > that you put that there), whereas the WD ones that I am familiar with spin > at 5400rpm, so not as bad as I thought. > > So probably ok to use, but I don't see many people using them for Ceph/ > generic NAS so can't be sure there's no hidden gotchas. I am not sure how trustworthy newegg-reviews are, but somehow I get some doubts about them now. I guess it does not matter that much, at least if not more than a disk a month is failing? The 3-year warranty gives some hope.. Are there some cost-efficient HDDs that someone can suggest? (Most likely 3TB drives, that seems to be the sweet-spot at the moment.) > Sorry nothing in detail, I did actually build a ceph cluster on the same 8 > core CPU as you have listed. I didn't have any performance problems but I do > remember with SSD journals when doing high queue depth writes I could get > the CPU quite high. It's like what I said before about the 1vs10Gb > networking, how important is performance, If using this CPU gives you an > extra 1ms of latency per OSD, is that acceptable? > > Agree 12cores (guessing 2.5Ghz each) will be an overkill for just 12 OSDs. I > have a very similar spec and see exactly the same as you, but will change > the nodes to 1CPU each when I expand and use the spare CPU's for the new > nodes. > > I'm using this:- > > http://www.supermicro.nl/products/system/4U/F617/SYS-F617H6-FTPTL_.cfm > > Mainly because of rack density, which I know doesn't apply to you. But the > fact they share PSU's/Rails/Chassis helps reduce power a bit and drives down > cost > > I can get 14 disks in each and they have 10GB on board. The SAS controller > is flashable to JBOD mode. > > Maybe one of the other Twin solutions might be suitable? I did consider that exact model (It was mentioned on the list some time ago) I could get about the same effective storage-capacity with it, but 10G-Networking is just too expensive on the Switch-side. Also those nodes and 10G-Switches consume a lot more power. By my estimates and numbers I found, the Avoton-Nodes should run at about 55W each. The Switches (EX3300) according to tech-specs would need 76W at max each. ___ Dominik _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |