On 08/08/13 15:21, Craig Lewis wrote:
I've seen a couple posts here about broken clusters that had to repair
by modifing the monmap, osdmap, or the crush rules.
The old school sysadmin in me says it would be a good idea to make
backups of these 3 databases. So far though, it seems like everybody
was able to repair their clusters by dumping the current map and
modifying it.
I'll probably do it, just to assuage my paranoia, but I was wondering
what you guys thought.
Well, this could get you *some* infos, but you wouldn't be able to
reconstruct a monitor this way. There's just way too many maps that
you'd need to reconstruct the monitor.
The not-so-best approach would be to grab all map epochs, from 1 to the
map's current epoch. We don't currently have a way to expose to the
user what is the first available map epoch in the store (the need for it
never came up), so for now you'd have to start at 1 and increment it
until you'd find an existing version (we trim old versions, so that
could be at 1, 10k, or a few hundred thousands, depending on how many
maps you have). With all that information, you could somehow
reconstruct a monitor with some effort -- and even so, we currently only
expose an interface to obtain maps for some services such as the mon,
osd, pg and mds; we have a bunch of other versions kept in the monitor
that are not currently exposed to the user.
This is something we definitely want to improve on, but as of this
moment the best approach to backup monitors reliably would be to stop
the monitor, copy the store, and restart the monitor. Assuming you have
3+ monitors, stopping just one of them wouldn't affect the quorum or
cluster availability. And assuming you're backing up a monitor that is
in the quorum, then backing it up is as good as backing any other monitor.
Hope this helps.
-Joao
I'm thinking of cronning this on the MON servers:
#!/usr/bin/env bash
# Number of days to keep backups
cleanup_age="10"
# Fetch the current timestamp, to use in the backup filenames
date=$(date +"%Y-%m-%dT%H:%M:%S")
# Dump the current maps
cd /var/lib/ceph/backups/
ceph mon getmap -o ./monmap.${date}
ceph osd getmap -o ./osdmap.${date}
ceph osd getcrushmap -o ./crushmap.${date}
# Delete old maps
find . -type f -regextype posix-extended -regex
'\./(mon|osd|crush)map\..*' -mtime +${cleanup_age} -print0 | xargs -0 rm
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
--
Joao Eduardo Luis
Software Engineer | http://inktank.com | http://ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
