I'm guessing you had writeback cache enabled on ceph-mon disk (smartctl -g wcache /dev/sdX) and disk firmware did not care about respecting flush semantics. On 11.08.2016 08:33, Wido den Hollander wrote: > >> Op 11 augustus 2016 om 0:10 schreef Sean Sullivan <seapasulli@xxxxxxxxxxxx>: >> >> >> I think it just got worse:: >> >> all three monitors on my other cluster say that ceph-mon can't open >> /var/lib/ceph/mon/$(hostname). Is there any way to recover if you lose all >> 3 monitors? I saw a post by Sage saying that the data can be recovered as >> all of the data is held on other servers. Is this possible? If so has >> anyone had any experience doing so? > > I have never done so, so I couldn't tell you. > > However, it is weird that on all three it got corrupted. What hardware are you using? Was it properly protected against power failure? > > If you mon store is corrupted I'm not sure what might happen. > > However, make a backup of ALL monitors right now before doing anything. > > Wido > >> _______________________________________________ >> ceph-users mailing list >> ceph-users@xxxxxxxxxxxxxx >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > _______________________________________________ > ceph-users mailing list > ceph-users@xxxxxxxxxxxxxx > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > -- Tomasz Kuzemko tomasz.kuzemko@xxxxxxxxxxxx
Attachment:
signature.asc
Description: OpenPGP digital signature
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
