Hermann, I think there was a discussion on recommended switches not too long ago. You should be able to find it in the mailing list archives. I think the latency of the network is usually very minor compared to ceph's dependency on cpu and disk latency, so for a simple cluster I wouldn't worry about it too much. I have found fs.com's dac cables to get stuck a lot, so I don't use them anymore. I usually buy dell or mellanox cables. Regarding network cards I've found the intel cards to be not that great due to bugs with lacp bonds, embedded lldp getting in the way and other issues. So I'm using mellanox cards instead, but broadcom should also work. hope it helps! best regards, Max On Wed, May 19, 2021 at 1:48 PM <ceph-users-request@xxxxxxx> wrote: > ---------- Forwarded message ---------- > From: Hermann Himmelbauer <hermann@xxxxxxx> > To: ceph-users@xxxxxxxx > Cc: > Bcc: > Date: Wed, 19 May 2021 11:22:26 +0200 > Subject: Suitable 10G Switches for ceph storage - any > recommendations? > Dear Ceph users, > I am currently constructing a small hyperconverged Proxmox cluster with > ceph as storage. So far I always had 3 nodes, which I directly linked > together via 2 bonded 10G network interfaces for the Ceph storage, so I > never needed any switching devices. > > This new cluster has more nodes, so I am considering using a 10G switch > for the storage network. As I have no experience with such a setup, I > wonder if there are any specific issues that I should think of > (latency...)? > > As the whole cluster should be not too expensive, I am currently > thinking of the following solution: > > 2* CRS317-1G-16s+RM switches: > https://mikrotik.com/product/crs317_1g_16s_rm#fndtn-testresults > > SFP+ Cables like these: > https://www.fs.com/de/products/48883.html > > Some network interface for each node with two SFP+ ports, e.g.: > > https://ark.intel.com/content/www/de/de/ark/products/39776/intel-ethernet-converged-network-adapter-x520-da2.html > > Connect each port with each switch and configure master/slave > configuration so that the switches are redundant. > > What do you think of this setup - or is there any information / > recommendation for an optimized setup of a 10G storage network? > > Best Regards, > Hermann > > -- > hermann@xxxxxxx > PGP/GPG: 299893C7 (on keyservers) > > _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |