Hello, On Tue, 21 Apr 2015 08:33:21 +0200 Götz Reinicke - IT Koordinator wrote: > Hi Christian, > Am 13.04.15 um 12:54 schrieb Christian Balzer: > > > > Hello, > > > > On Mon, 13 Apr 2015 11:03:24 +0200 Götz Reinicke - IT Koordinator > > wrote: > > > >> Dear ceph users, > >> > >> we are planing a ceph storage cluster from scratch. Might be up to 1 > >> PB within the next 3 years, multiple buildings, new network > >> infrastructure for the cluster etc. > > <...> > > > So at your storage node density of 12 HDDs (16 HDD chassis are not > > space efficient), 40GbE is overkill with a single link/network, > > insanely so with 2 networks. > > What would you think, if we go with 20 OSDs, may be 22 OSd (24 HDD > chassis with 2 or 4 SSD (OS / Journal)) > Density is nice, as in cost-effective. But with Ceph, smaller is better, both in terms of performance and failure domains. If you can start with a large enough number of nodes (at least 10) and remember that you're probably looking for at least 2GHz per OSD with SSD journals, go for it. But you'll need NVMe SSDs to satisfy 11 OSD HDDs, never mind the rather large failure domain. So a 1:5 SSD journal to HDD OSD ratio would be better. > From the "Questions about an example of ceph infrastructure" topic ! got > the calculation of > > HDDs x expected spped per disk = Max Performance => > > 20 x 70 MB/s = 1400 MB/s ie 1.4 GB/s => 14 Gb/s redundant => 28 Gb/s > > I like the suggestion from Robert LeBlanc, using two 40Gb ports with > VLANs. > > Currently we have to extend our LAN anyway, all 10Gb ports are in use. > > Upgrading 10Gb ports costs more than buying new 10Gb hardware in our > case. (Good old Cisco 6500 vs. modern 4500x challenge :) ) > > Furthermore we will see a lot of more traffic and requirements regarding > speed within next year and than an other rise within the next 5 years. > (e.g. 4K/8K video realtime playback for some workstations. 4K is about 9 > Gb/s ! per Workstation.) > > Long story short, we have/will/should/and can start with 40Gb. The > question is how big :) > If you can afford it, sure. ^o^ Christian > I'd say, after some more internal discussions to, redundant switches are > mandatory in our case, and the 40Gb-VLANs a good balance regarding > redundancy, cost and performance. > > Thumbs up or down, your vote :D. Seriously what you think? > > Thanks for your feedback and best regards . Götz > -- Christian Balzer Network/Systems Engineer chibi@xxxxxxx Global OnLine Japan/Fusion Communications http://www.gol.com/ _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |