On Wed, 08 Oct 2014 00:45:06 +0000 Scott Laird wrote: > IIRC, one thing to look out for is that there are two ways to do IP over > Infiniband. You can either do IP over Infiniband directly (IPoIB), or > encapsulate Ethernet in Infiniband (EoIB), and then do IP over the fake > Ethernet network. > > IPoIB is more common, but I'd assume that IB<->Ethernet bridges really > only bridge EoIB. > Most of them do indeed, alas the 4036E supposedly has a FPGA based IPoIB to Ethernet gateway. Probably another reason to build your own, aside from the price tag. ^o^ Christian > On Tue Oct 07 2014 at 5:34:57 PM Christian Balzer <chibi@xxxxxxx> wrote: > > > On Tue, 07 Oct 2014 20:40:31 +0000 Scott Laird wrote: > > > > > I've done this two ways in the past. Either I'll give each machine > > > an Infiniband network link and a 1000baseT link and use the > > > Infiniband one as the private network for Ceph, or I'll throw an > > > Infiniband card into a PC and run something like Vyatta/VyOS on it > > > and make it a router, so IP traffic can get out of the IB network. > > > Of course, those have both been for test labs. YMMV. > > > > > > > That. > > > > Of course in a production environment you would want something with 2 > > routers in a failover configuration. > > And there are switches/gateways that combine IB and Ethernet, but they > > tend to be not so cheap. ^^ > > > > More below. > > > > > On Tue Oct 07 2014 at 11:05:23 AM Massimiliano Cuttini > > > <max@xxxxxxxxxxxxx> wrote: > > > > > > > Hi Christian, > > > > > > > > When you say "10 gig infiniband", do you mean QDRx4 Infiniband > > > > (usually flogged as 40Gb/s even though it is 32Gb/s, but who's > > > > counting), which tends to be the same basic hardware as the 10Gb/s > > > > Ethernet offerings from Mellanox? > > > > > > > > A brand new 18 port switch of that caliber will only cost about > > > > 180$ per port, too. > > > > > > > > > > > > > > > > I investigate about infiniband but i didn't found affordable > > > > prices at all. > > > > Then you're doing it wrong or comparing apples to oranges (you of > > course need to compare IB switches to similar 10GbE ones). > > And the prices of HCA (aka network cards in the servers) and cabling. > > > > > > Moreover how do you connect your *l**egacy node servers* to your > > > > *brand new storages* if you have Infiniband only on storages & > > > > switches? Is there any mixed switch that allow you both to connect > > > > with Infiniband and Ethernet? > > > > > > > > If there is, please send specs because i cannot find just by > > > > google it. > > > > > > The moment you type in "infiniband et" google will already predict > > amongst other pertinent things "infiniband ethernet gateway" and > > "infiniband ethernet bridge". > > But even "infiniband ethernet switch" has a link telling you pretty > > much what was said here now at the 6th position: > > http://www.tomshardware.com/forum/44997-42-connect- > > infiniband-switch-ethernet > > > > Christian > > > > Thanks, > > > > Max > > > > > > > > _______________________________________________ > > > > ceph-users mailing list > > > > ceph-users@xxxxxxxxxxxxxx > > > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > > > > > > > > -- > > Christian Balzer Network/Systems Engineer > > chibi@xxxxxxx Global OnLine Japan/Fusion Communications > > http://www.gol.com/ > > _______________________________________________ > > ceph-users mailing list > > ceph-users@xxxxxxxxxxxxxx > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > -- Christian Balzer Network/Systems Engineer chibi@xxxxxxx Global OnLine Japan/Fusion Communications http://www.gol.com/ _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |