Hello, We had a power failure and after some trouble 2 of our OSDs started crashing with this error: "FAILED assert(last_e.version.version < e.version.version)" I know what's the problematic PG, and searching the ceph lists and the web I saw that ultimately I should fix that PG using ceph-kvstore-tool, but I can't find an example of how to use it. Can anybody give me a hint? Should I ask in ceph-devel? Thanks in advance. An extract from a debug log, can provide more if needed: -2> 2017-08-01 21:40:32.509895 7f4f506d8880 20 read_log 1793746'173182313 (1793746'173182312) modify 3/44f40dd4/rbd_data.84873da26686612.00000000000000f1/head by client.215472136.0:13700212 2017-08-01 12:26:06.904276 -1> 2017-08-01 21:40:32.509900 7f4f506d8880 20 read_log 1794404'173182313 (1793746'173182220) modify 3/67fc6dd4/rbd_data.13957cd6ab8977c.0000000000000027/head by client.267959820.0:11279245 2017-08-01 12:29:52.696307 0> 2017-08-01 21:40:32.511324 7f4f506d8880 -1 osd/PGLog.cc: In function 'static void PGLog::read_log(ObjectStore*, coll_t, coll_t, ghobject_t, const pg_info_t&, std::map<eversion_t, hobject_t>&, PGLog::IndexedLog&, pg_missing_t&, std::ostringstream&, std::set<std::basic_string<char> >*)' thread 7f4f506d8880 time 2017-08-01 21:40:32.509905 osd/PGLog.cc: 911: FAILED assert(last_e.version.version < e.version.version) Regards, -- Ricardo J. Barberis Usuario Linux Nº 250625: http://counter.li.org/ Usuario LFS Nº 5121: http://www.linuxfromscratch.org/ Senior SysAdmin / IT Architect - www.DonWeb.com _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com