On 11/04/2016 01:39 AM, Tracy Reed wrote:
After a lot of messing about I have manually created a monmap and got the two new monitors working for a total of three. But to do that I had to delete the first monitor which for some reason was coming up with a bogus fsid after manipulated the monmap which I checked and it had the correct fsid. So I recreated it with the right fsid and after that I had three monitors with quorum. I had to manually put the keys in place on the recreated monitor too. But now all of my OSDs have disapeared! Apparently the mon I deleted was storing some special knowledge of the OSDs?
The monitors keep information about the osds in the cluster in the form of osdmaps. If you delete the osdmaps from all your monitors, then it's as if the cluster has no osds and, therefore, no data.
The mon log says: 2016-11-03 18:31:26.612012 7f7139529700 0 mon.ceph01@0(leader).data_health(14) update_stats avail 96% total 51175 MB, used 1744 MB, avail 49430 MB 2016-11-03 18:31:26.679911 7f7138d28700 0 cephx server osd.3: couldn't find entity name: osd.3 2016-11-03 18:31:26.876589 7f7138d28700 0 cephx server osd.6: couldn't find entity name: osd.6 2016-11-03 18:31:26.996219 7f7138d28700 0 cephx server osd.14: couldn't find entity name: osd.14 2016-11-03 18:31:27.016283 7f7138d28700 0 cephx server osd.41: couldn't find entity name: osd.41 2016-11-03 18:31:27.016406 7f7138d28700 0 cephx server osd.37: couldn't find entity name: osd.37 2016-11-03 18:31:27.016606 7f7138d28700 0 cephx server osd.40: couldn't find entity name: osd.40 2016-11-03 18:31:27.017276 7f7138d28700 0 cephx server osd.48: couldn't find entity name: osd.48 2016-11-03 18:31:27.291934 7f7138d28700 0 cephx server osd.4: couldn't find entity name: osd.4 2016-11-03 18:31:27.292598 7f7138d28700 0 cephx server osd.5: couldn't find entity name: osd.5 2016-11-03 18:31:27.339803 7f7138d28700 0 cephx server osd.7: couldn't find entity name: osd.7 So how do I tell the mon about the OSDs?
These messages pertain the authentication entities. The monitors also keep auth keys for all clients that communicate with the monitors.
Given you recreated your monitors, the osd keys the monitors were storing vanished and now the osds are unable to contact the monitors.
You can manually add the keys if you so wish (see the docs on how to manually add an osd).
-Joao _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
