On 4/12/16 9:02 AM, Gregory Farnum wrote:
On Tue, Apr 12, 2016 at 4:41 AM, Eric Hall <eric.hall@xxxxxxxxxxxxxx> wrote:On 4/12/16 12:01 AM, Gregory Farnum wrote:Exactly what values are you reading that's giving you those values? The "real" OSDMap epoch is going to be at least 38630...if you're very lucky it will be exactly 38630. But since it reset itself to 1 in the monitor's store, I doubt you'll be lucky.
It's been my week...
I'm getting this from ceph-kvstore-tool list.I meant the keys that it was outputting...I forgot we actually had one called "osdmap".
From ceph-kvstore-tool /path/monN/store.db list |grep osd: mon1: osdmap:1 osdmap:38072 [...] osdmap:38630 osdmap:first_committed osdmap:full_38072 [...] osdmap:full_38456 osdmap:last_committed mon2: osdmap:1 osdmap:38072 [...] osdmap:38630 osdmap:first_committed osdmap:full_38072 [...] osdmap:full_38630 osdmap:full_latest osdmap:last_committed mon3: osdmap:1 osdmap:38072 [...] osdmap:38630 osdmap:first_committed osdmap:full_38072 [...] osdmap:full_38630 osdmap:full_latest osdmap:last_committed
So in order to get your cluster back up, you need to find the largest osdmap version in your cluster. You can do that, very tediously, by looking at the OSDMap stores. Or you may have debug logs indicating it more easily on the monitors.I don't see info like this in any logs. How/where do I inspect this?If you had debugging logs up high enough, it would tell you things like each map commit. And every time the monitor subsystems (like the OSD Monitor) print out any debugging info they include what epoch/version they are on, so it's in the log output prefix.
I doubt I have debug high enough... example lines from mon3 log:2016-04-11 02:59:27.534149 7fef19a86700 0 mon.mon3@2(peon) e1 handle_command mon_command({"prefix": "status"} v 0) v1 2016-04-11 02:59:34.556487 7fef19a86700 1 mon.mon3@2(peon).log v32366957 check_sub sending message to client.6567304 172.16.250.1:0/3381977473 with 1 entries (version 32366957)
Where is the OSDMap store if not in store.db? Thank you, -- Eric
Attachment:
smime.p7s
Description: S/MIME Cryptographic Signature
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
