Re: Backup monmap, osdmap, and crushmap

[Craig Lewis <clewis@xxxxxxxxxxxxxxxxxx>]

You saved me a bunch of time; I was planning to test my backup and restore later today.   Thanks!


It occurred to me that the backups won't be as useful as I thought.  I'd need to make sure that the PGs hadn't moved around after the backup was made.  If they had, I'd spend a lot of time tracking down the new locations and manually rsyncing data.  Not a big deal on a small cluster, but it get harder as the cluster gets larger.  Now that I look at the dumps, it looks like PG locations are one of the things missing. 


Binary backups of the MON directories are fine.  All of the problems I've seen on the list occurred during cluster upgrades, so I'll make the backup part of my upgrade procedure instead of a cron.


If I wanted to restore a backup, what would be required?  Looking at http://eu.ceph.com/docs/v0.47.1/ops/manage/grow/mon/#removing-a-monitor-from-an-unhealthy-or-down-cluster, I don't see a monmap directory inside /var/lib/ceph/mon/ceph-#/ anymore.  I assume it went away during the switch to LevelDB, so I think I'll need to dump a copy when I make the binary backup. 

I'm assuming there is some node specific data in each MON's store.  If not, could I just stop all monitors, rsync the backup to all monitors, and start them all up? 

I'm not using cephx, but I assume the keyrings would complicate things.  It would probably be easiest to make binary backups on all of the monitors (with some time delay, so only one is offline at a time), then start them up in newest-to-oldest-backup order.  Or I could use LVM snapshots to make simultaneous backups on all monitors.


Thanks for the info.

Craig Lewis
Senior Systems Engineer
Office +1.714.602.1309
Email clewis@xxxxxxxxxxxxxxxxxx

Central Desktop. Work together in ways you never thought possible.
Connect with us   Website  |  Twitter  |  Facebook  |  LinkedIn  |  Blog

On 8/12/13 14:39 , Joao Eduardo Luis wrote:

On 08/08/13 15:21, Craig Lewis wrote:

I've seen a couple posts here about broken clusters that had to repair
by modifing the monmap, osdmap, or the crush rules.

The old school sysadmin in me says it would be a good idea to make
backups of these 3 databases.  So far though, it seems like everybody
was able to repair their clusters by dumping the current map and
modifying it.

I'll probably do it, just to assuage my paranoia, but I was wondering
what you guys thought.

Well, this could get you *some* infos, but you wouldn't be able to reconstruct a monitor this way.  There's just way too many maps that you'd need to reconstruct the monitor.


The not-so-best approach would be to grab all map epochs, from 1 to the map's current epoch.  We don't currently have a way to expose to the user what is the first available map epoch in the store (the need for it never came up), so for now you'd have to start at 1 and increment it until you'd find an existing version (we trim old versions, so that could be at 1, 10k, or a few hundred thousands, depending on how many maps you have).  With all that information, you could somehow reconstruct a monitor with some effort -- and even so, we currently only expose an interface to obtain maps for some services such as the mon, osd, pg and mds;  we have a bunch of other versions kept in the monitor that are not currently exposed to the user.


This is something we definitely want to improve on, but as of this moment the best approach to backup monitors reliably would be to stop the monitor, copy the store, and restart the monitor.  Assuming you have 3+ monitors, stopping just one of them wouldn't affect the quorum or cluster availability.  And assuming you're backing up a monitor that is in the quorum, then backing it up is as good as backing any other monitor.


Hope this helps.


  -Joao

I'm thinking of cronning this on the MON servers:
#!/usr/bin/env bash

# Number of days to keep backups
cleanup_age="10"

# Fetch the current timestamp, to use in the backup filenames
date=$(date +"%Y-%m-%dT%H:%M:%S")

# Dump the current maps
cd /var/lib/ceph/backups/
ceph mon getmap -o ./monmap.${date}
ceph osd getmap -o ./osdmap.${date}
ceph osd getcrushmap -o ./crushmap.${date}

# Delete old maps
find . -type f -regextype posix-extended -regex
'\./(mon|osd|crush)map\..*' -mtime +${cleanup_age} -print0 | xargs -0 rm




_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com