Here is the procedure I wrote for our internal use (it is still a work in progress) and may help you:
Creating the First Monitor
Once you Ceph installed, DNS and networking configured and have a ceph.conf file built, you are ready to bootstrap the first monitor. The UUID is the same from the ceph.conf file generated earlier, cluster-name is the name of the Ceph cluster, usually just 'ceph', hostname is the short name of the host and must match `hostname -s`.
1. SSH into the monitor host
2. Create the monitor keyring
Adding Additional Monitors
Adding additional monitors will make the cluster less susceptible to outages due to hardware running the monitors going offline. As monitors are added, the load on each monitor can both increase and decrease, so it is import to understand when a new set of monitors should be added (to understand why two monitors should be added, please see Monitors). Having more monitors reduces the number of clients that each monitor has to service. However, each time a CRUSH map changes (an OSD is marked out, a new OSD is added, etc), a majority of monitors have to agree on the CRUSH changes in the PAXOS algorithm and the CRUSH map has to be updated on each Monitor. If the cluster is experiencing many CRUSH map changes it can add additional load on the Monitors.
Monitors perform many file sync operations and are sensitive to latencies with these operations. The large number of file sync operations can be very disruptive to to OSD processes especially if residing on the same traditional rotational disk. It is best to have a Monitor on its own dedicated hardware. If dedicated hardware is not an option, then locate the Monitor store on an SSD that is not primarily used for an OSD, an SSD journal would be OK.
The process of adding a new monitor is detailed in (http://ceph.com/docs/master/rados/operations/add-or-rm-mons/#adding-a-monitor-manual) and is outlined as follows:
Creating the First Monitor
Once you Ceph installed, DNS and networking configured and have a ceph.conf file built, you are ready to bootstrap the first monitor. The UUID is the same from the ceph.conf file generated earlier, cluster-name is the name of the Ceph cluster, usually just 'ceph', hostname is the short name of the host and must match `hostname -s`.
1. SSH into the monitor host
2. Create the monitor keyring
ceph-authtool --create-keyring /tmp/ceph.mon.keyring --gen-key -n mon. --cap mon 'allow *'3. Create the admin keyring
ceph-authtool --create-keyring /etc/ceph/ceph.client.admin.keyring --gen-key -n client.admin --set-uid=0 --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow'4. Add the admin key to the monitor keyring so the Admin user can manage the cluster
ceph-authtool /tmp/ceph.mon.keyring --import-keyring /etc/ceph/ceph.client.admin.keyring5. Create the initial monitor map
monmaptool --create --add {hostname} {ip-address} --fsid {uuid} /tmp/monmap6. Create the directory the monitor will store persistent data
sudo mkdir /var/lib/ceph/mon/{cluster-name}-{hostname}7. Create the file structure for the monitor
ceph-mon --mkfs -i {hostname} --monmap /tmp/monmap --keyring /tmp/ceph.mon.keyring8. Let the monitor know that everything is ready to go
sudo touch /var/lib/ceph/mon/{cluster-name}-{hostname}/done9. Start the Monitor service
sudo start ceph-mon id=node110. Set the Monitor service to start at boot
Adding Additional Monitors
Adding additional monitors will make the cluster less susceptible to outages due to hardware running the monitors going offline. As monitors are added, the load on each monitor can both increase and decrease, so it is import to understand when a new set of monitors should be added (to understand why two monitors should be added, please see Monitors). Having more monitors reduces the number of clients that each monitor has to service. However, each time a CRUSH map changes (an OSD is marked out, a new OSD is added, etc), a majority of monitors have to agree on the CRUSH changes in the PAXOS algorithm and the CRUSH map has to be updated on each Monitor. If the cluster is experiencing many CRUSH map changes it can add additional load on the Monitors.
Monitors perform many file sync operations and are sensitive to latencies with these operations. The large number of file sync operations can be very disruptive to to OSD processes especially if residing on the same traditional rotational disk. It is best to have a Monitor on its own dedicated hardware. If dedicated hardware is not an option, then locate the Monitor store on an SSD that is not primarily used for an OSD, an SSD journal would be OK.
The process of adding a new monitor is detailed in (http://ceph.com/docs/master/rados/operations/add-or-rm-mons/#adding-a-monitor-manual) and is outlined as follows:
- Copy the Monitor key and Monitor map from a running Monitor to the new monitor.
- Create a monitor directory on the new monitor.
- Add the new monitor to the Monitor map.
- Start the new monitor.
On Thu, Mar 12, 2015 at 9:58 AM, Jesus Chavez (jeschave) <jeschave@xxxxxxxxx> wrote:
Hi Robert yes I did disable completely actually with chkconfig off for not take the service up when booting, I have 2 networks 1 with internet for yum purposes and the network for the public network so before any configuration I specified on ceph.conf that public network but I am not sure if it is the cause for something...The thing is that I am not sure about which steps should run in the new monitor host and which one should I run in the initial monitor, it seems like step 3 and 4 that is generate keyring and mapping should be done in initial monitor server and also step 5 that is mkfs becuase if I try to run those steps in the new monitor host didnt work cause cant find the keys :(
Jesus Chavez
SYSTEMS ENGINEER-C.SALES
jeschave@xxxxxxxxx
Phone: +52 55 5267 3146
Mobile: +51 1 5538883255
CCIE - 44433If I remember right, the mon key has to be the same between all the mon hosts. I don't think I added an admin key to my second mon, it got all the other keys once it joined the mon closure. I do remember the join taking a while. Have you checked the firewall to make sure traffic is allowed? I don't remember if you said you checked it.
Robert LeBlanc
Sent from a mobile device please excuse any typos.
On Mar 11, 2015 8:08 PM, "Jesus Chavez (jeschave)" <jeschave@xxxxxxxxx> wrote:
Thanks Steffen I have followed everything not sure what is going on, the mon keyring and client admin are individual? Per mon host? Or do I need to copy from the first initial mon node?
Thanks again!
Jesus Chavez
SYSTEMS ENGINEER-C.SALES
jeschave@xxxxxxxxx
Phone: +52 55 5267 3146
Mobile: +51 1 5538883255
CCIE - 44433
On 12/03/2015, at 00.55, Jesus Chavez (jeschave) <jeschave@xxxxxxxxx> wrote:
can anybody tell me a good blog link that explain how to add monitor? I have tried manually and also with ceph-deploy without success =(Dunno if these might help U:
http://ceph.com/docs/master/rados/operations/add-or-rm-mons/#adding-a-monitor-manual
http://cephnotes.ksperis.com/blog/2013/08/29/mon-failed-to-start
/Steffen
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
