Thanks for the reply. When I said the compute nodes mounted the cephfs volume, I am referring to a real linux cluster of physical machines,. Openstack VM/ compute nodes are not involved in my setup. We are transitioning from an older linux cluster using nfs from the head node/san to the new cluster using cephfs. All physical systems mounting the shared volume. Storing home directories and data. http://oi63.tinypic.com/2ljp72v.jpg The linux cluster is in a NAT private network, where the only systems attached to the corporate network are the ceph servers and our main linux head node. They are dual connected. Your saying I cant have ceph volumes mounted and the traffic to the osds coming in on more than one interface? It is limited to one interface? Sent from Mail<https://go.microsoft.com/fwlink/?LinkId=550986> for Windows 10 From: Wido den Hollander<mailto:wido@xxxxxxxx> Sent: Thursday, September 8, 2016 8:23 AM To: ceph-users@xxxxxxxxxxxxxx<mailto:ceph-users@xxxxxxxxxxxxxx>; Jim Kilborn<mailto:jim@xxxxxxxxxxx> Subject: Re: FW: Multiple public networks and ceph-mon daemons listening > Op 8 september 2016 om 15:02 schreef Jim Kilborn <jim@xxxxxxxxxxx>: > > > Hello all… > > I am setting up a ceph cluster (jewel) on a private network. The compute nodes are all running centos 7 and mounting the cephfs volume using the kernel driver. The ceph storage nodes are dual connected to the private network, as well as our corporate network, as some users need to mount the volume to their workstations (also centos 7) from the corporate network. > Compute via CephFS? I would highly recommend using RBD for Block Devices, don't use CephFS in between. > The private network is infiniband, so I have that set as the cluster network , and have both networks listed in the private networks in the ceph.conf. > > However, the mon daemons only listen on the private network, and if I want to mount the volume from the corporate network, it has to mount via the private network address of the ceph storage nodes, which means that the cluster head node (linux) has to route that traffic. > > I would like to know if there is a way to have the monitors listen on both their interfaces, like the osd/mds daemons do, so I could use the appropriate address in the fstab of the clients, depending on which network they are on. > No, not really. The monitors have one IP and that's the only IP they can work with. Your setup isn't really going to work either. Although the OSDs seem to listen on */[::], they don't The OSDMap contains the IPs of the OSDs where clients will connect to. That can't be dual-homed. You would have to come up with a routed network in this case so people can reach the IPs of all Ceph nodes. Wido > Alternatively, I could have one of the mon daemons added with its private network address, as all ceph storage nodes are dual connected, but I would lose some fault tolerance I think (if that monitor goes down) > > Just thought there must be a better way. I have 3 monitor nodes (dual functioning as osd nodes). There are all brand new dell 730xd with 12GB ram and dual xeons. I also have a ssd cache in front of a erasure coded pool. > > Any suggestions? > > Thanks for taking the time… > > _______________________________________________ > ceph-users mailing list > ceph-users@xxxxxxxxxxxxxx > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |