So we recently had a power outage and I seem to have lost 2 of 3 of my monitors. I have since copied /var/lib/ceph/mon/ceph-$(hostname){,.BAK} and then created a new cephfs and finally generated a new filesystem via
''' sudo ceph-mon -i {mon-id} --mkfs --monmap {tmp}/{map-filename} --keyring {tmp}/{key-filename} '''
''' sudo ceph-mon -i {mon-id} --mkfs --monmap {tmp}/{map-filename} --keyring {tmp}/{key-filename} '''
After this I copied the monmap from the working monitor to the other two. via::
''' ceph-mon -i {mon-id} --inject-monmap {map-path} '''
''' ceph-mon -i {mon-id} --inject-monmap {map-path} '''
At this point I was left with a working monitor map (afaik) but ceph cli commands return ::
'''
root@kh11-8:/var/run/ceph# ceph -s
2016-08-10 14:13:58.563241 7fdd719b3700 0 librados: client.admin authentication error (1) Operation not permitted
Error connecting to cluster: PermissionError
'''
Now after waiting a little while it looks like the quorum kicked out the only working monitor::
'''
{
"election_epoch": 358,
"quorum": [
0,
1
],
"quorum_names": [
"kh11-8",
"kh12-8"
],
"quorum_leader_name": "kh11-8",
"monmap": {
"epoch": 1,
"fsid": "a6ae50db-5c71-4ef8-885e-8137c7793da8",
"modified": "0.000000",
"created": "0.000000",
"mons": [
{
"rank": 0,
"name": "kh11-8",
"addr": "10.64.64.134:6789\/0"
},
{
"rank": 1,
"name": "kh12-8",
"addr": "10.64.64.143:6789\/0"
},
{
"rank": 2,
"name": "kh13-8",
"addr": "10.64.64.151:6789\/0"
}
]
}
}
'''
kh13-8 was the original working node and kh11-8 and kh12-8 were the ones that had fs issues.
kh13-8 was the original working node and kh11-8 and kh12-8 were the ones that had fs issues.
Currently I am at a loss as to what to do as ceph -w and -s commands do not work due to permissions/cephx errors and the original working monitor was kicked out.
Is there any way to regenerate the cephx authentication and recover the monitor map?
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
