Hi,
In the case of three ceph hosts, you could also consider this setup:
https://pve.proxmox.com/wiki/Full_Mesh_Network_for_Ceph_Server
This only requires that you have two 10G nics on each machine. Plus an
extra 1G for 'regular' non-ceph traffic.
That way at least your ceph comms would be 10G, since 1G is surely going
to be a bottleneck.
We are running the above setup. No problems. Only issue is: adding a
fourth node will be relatively intrusive.
MJ
On 9/20/19 8:23 PM, Salsa wrote:
Replying inline.
--
Salsa
Sent with ProtonMail <https://protonmail.com> Secure Email.
‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Friday, September 20, 2019 1:34 PM, Martin Verges
<martin.verges@xxxxxxxx> wrote:
Hello Salsa,
I have tested Ceph using VMs but never got to put it to use and
had a lot of trouble to get it to install.
if you want to get rid of all the troubles from installing to day2day
operations, you could consider using https://croit.io/croit-virtual-demo
Amazing! Where were you 3 months ago? Only problem is that I think we
have no moer budget for this so I can't get approval for software license.
- Use 2 HDDs for SO using RAID 1 (I've left 3.5TB unallocated in
case I can use it later for storage)
- Install CentOS 7.7
Is ok, but won't be necessary if you choose croit as we boot from the
network and don't install a operating system.
No budget for software license
- Use 2 vLANs, one for ceph internal usage and another for
external access. Since they've 4 network adapters, I'll try to
bond them in pairs to speed up network (1Gb).
If there is no internal policy that forces you to do seperate
networks, you can use a simple 1 vlan setup and bond 4*1GbE. Otherwise
it's ok.
The service is critical and we are afraid that the network might be
congested and QoS for the end user degrades.
- I'll try to use ceph-ansible for installation. I failed to use
it on lab, but it seems more recommended.
- Install Ceph Nautilus
Ultra easy with croit, maybe look at our videos on youtube -
https://www.youtube.com/playlist?list=PL1g9zo59diHDSJgkZcMRUq6xROzt_YKox
Thanks! I'll be watching them.
- Each server will host OSD, MON, MGR and MDS.
ok, but you should use ssd for metadata.
No budget and no option to get those now.
- One VM for ceph-admin: This wil be used to run ceph-ansible and
maybe to host some ceph services later
perfect for croit ;)
- I'll have to serve samba, iscsi and probably NFS too. Not sure
how or on which servers.
Just put it on the servers as well, with croit it is just a click away
and everything is included in our interface.
If not using croit, you can still install it on the same systems and
configure it by hand/script.
Great! Thanks for the help and congratulations on that demo. It is the
best I've used and the easiest ceph setup I've found. As feedback, the
last part of the demo tutorial is not 100% compatible with the master
branch from github. The RBD pool creation has a different interface than
the one presented in your tutorial (Or I made some mistake along the
way). Also, my cluster is showing error in my placement groups after RB
pool creation, but I'll try to find out what happened.
Thanks again!
--
Martin Verges
Managing director
Mobile: +49 174 9335695
E-Mail: martin.verges@xxxxxxxx <mailto:martin.verges@xxxxxxxx>
Chat: https://t.me/MartinVerges
croit GmbH, Freseniusstr. 31h, 81247 Munich
CEO: Martin Verges - VAT-ID: DE310638492
Com. register: Amtsgericht Munich HRB 231263
Web: https://croit.io
YouTube: https://goo.gl/PGE1Bx
Am Fr., 20. Sept. 2019 um 18:14 Uhr schrieb Salsa
<salsa@xxxxxxxxxxxxxx <mailto:salsa@xxxxxxxxxxxxxx>>:
I have tested Ceph using VMs but never got to put it to use and
had a lot of trouble to get it to install.
Now I've been asked to do a production setup using 3 servers (Dell
R740) with 12 4TB each.
My plan is this:
- Use 2 HDDs for SO using RAID 1 (I've left 3.5TB unallocated in
case I can use it later for storage)
- Install CentOS 7.7
- Use 2 vLANs, one for ceph internal usage and another for
external access. Since they've 4 network adapters, I'll try to
bond them in pairs to speed up network (1Gb).
- I'll try to use ceph-ansible for installation. I failed to use
it on lab, but it seems more recommended.
- Install Ceph Nautilus
- Each server will host OSD, MON, MGR and MDS.
- One VM for ceph-admin: This wil be used to run ceph-ansible and
maybe to host some ceph services later
- I'll have to serve samba, iscsi and probably NFS too. Not sure
how or on which servers.
Am I missing anything? Am I doing anything "wrong"?
I searched for some actual guidance on setup but I couldn't find
anything complete, like a good tutorial or reference based on
possible use-cases.
So, is there any suggestions you could share or links and
references I should take a look?
Thanks;
--
Salsa
Sent with ProtonMail <https://protonmail.com> Secure Email.
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx <mailto:ceph-users@xxxxxxxxxxxxxx>
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
