On Thu, May 25, 2017 at 8:39 AM Stuart Harland <s.harland@xxxxxxxxxxxxxxxxxxxxxx> wrote:
Has no-one any idea about this? If needed I can produce more information or diagnostics on request. I find it hard to believe that we are the only people experiencing this, and thus far we have lost about 40 OSDs to corruption due to this.RegardsStuart Harland
The only person who's reported this remotely recently was working on a FreeBSD port. Other than the one tracker bug you found, errors like this are usually the result of failing disks, buggy local filesystems, or incorrect configuration (like turning off barriers).
I assume you didn't just upgrade from a pre-Jewel release that might have been susceptible to that tracker.
2) is there a way to recover the damaged OSD metadata, as I really don’t want to keep having to rebuild large numbers of disks based on something arbitrary.
I saw somewhere (check the list archives?) that you may be able to get around it by removing just the PG which is causing the crash, assuming it has replicas elsewhere.
But more generally you want to figure out how this is happening. Either you've got disk state which was previously broken and undetected (which, if you've been running 10.2.5 on all your OSDs, I don't think is possible), or you've experienced recent failures which are unlikely Ceph software bugs. (They might be! But you'd be the only to report them anywhere I can see.)
-Greg
_______________________________________________SEEK_HOLE is disabled via 'filestore seek data hole' config option-31> 2017-05-24 10:23:10.152349 7f24035e2800 0 genericfilestorebackend(/var/lib/ceph/osd/txc1-1908) detect_features: splice is supported-30> 2017-05-24 10:23:10.182065 7f24035e2800 0 genericfilestorebackend(/var/lib/ceph/osd/txc1-1908) detect_features: syncfs(2) syscall fully supported (by glibc and kernel)-29> 2017-05-24 10:23:10.182112 7f24035e2800 0 xfsfilestorebackend(/var/lib/ceph/osd/txc1-1908) detect_feature: extsize is disabled by conf-28> 2017-05-24 10:23:10.182839 7f24035e2800 1 leveldb: Recovering log #23079-27> 2017-05-24 10:23:10.284173 7f24035e2800 1 leveldb: Delete type=0 #23079-26> 2017-05-24 10:23:10.284223 7f24035e2800 1 leveldb: Delete type=3 #23078-25> 2017-05-24 10:23:10.284807 7f24035e2800 0 filestore(/var/lib/ceph/osd/txc1-1908) mount: enabling WRITEAHEAD journal mode: checkpoint is not enabled-24> 2017-05-24 10:23:10.285581 7f24035e2800 2 journal open /var/lib/ceph/osd/txc1-1908/journal fsid 8dada68b-0d1c-4f2a-bc96-1d861577bc98 fs_op_seq 20363902-23> 2017-05-24 10:23:10.289523 7f24035e2800 1 journal _open /var/lib/ceph/osd/txc1-1908/journal fd 18: 5367660544 bytes, blocksize 4096 bytes, directio = 1, aio = 1-22> 2017-05-24 10:23:10.293733 7f24035e2800 2 journal open advancing committed_seq 20363681 to fs op_seq 20363902-21> 2017-05-24 10:23:10.293743 7f24035e2800 2 journal read_entry -- not readable-20> 2017-05-24 10:23:10.293744 7f24035e2800 2 journal read_entry -- not readable-19> 2017-05-24 10:23:10.293745 7f24035e2800 3 journal journal_replay: end of journal, done.-18> 2017-05-24 10:23:10.297605 7f24035e2800 1 journal _open /var/lib/ceph/osd/txc1-1908/journal fd 18: 5367660544 bytes, blocksize 4096 bytes, directio = 1, aio = 1-17> 2017-05-24 10:23:10.298470 7f24035e2800 1 filestore(/var/lib/ceph/osd/txc1-1908) upgrade-16> 2017-05-24 10:23:10.298509 7f24035e2800 2 osd.1908 0 boot-15> 2017-05-24 10:23:10.300096 7f24035e2800 1 <cls> cls/replica_log/cls_replica_log.cc:141: Loaded replica log class!-14> 2017-05-24 10:23:10.300384 7f24035e2800 1 <cls> cls/user/cls_user.cc:375: Loaded user class!-13> 2017-05-24 10:23:10.300617 7f24035e2800 0 <cls> cls/hello/cls_hello.cc:305: loading cls_hello-12> 2017-05-24 10:23:10.303748 7f24035e2800 1 <cls> cls/refcount/cls_refcount.cc:232: Loaded refcount class!-11> 2017-05-24 10:23:10.304120 7f24035e2800 1 <cls> cls/version/cls_version.cc:228: Loaded version class!-10> 2017-05-24 10:23:10.304439 7f24035e2800 1 <cls> cls/log/cls_log.cc:317: Loaded log class!-9> 2017-05-24 10:23:10.307437 7f24035e2800 1 <cls> cls/rgw/cls_rgw.cc:3359: Loaded rgw class!-8> 2017-05-24 10:23:10.307768 7f24035e2800 1 <cls> cls/timeindex/cls_timeindex.cc:259: Loaded timeindex class!-7> 2017-05-24 10:23:10.307927 7f24035e2800 0 <cls> cls/cephfs/cls_cephfs.cc:202: loading cephfs_size_scan-6> 2017-05-24 10:23:10.308086 7f24035e2800 1 <cls> cls/statelog/cls_statelog.cc:306: Loaded log class!-5> 2017-05-24 10:23:10.315241 7f24035e2800 0 osd.1908 863035 crush map has features 2234490552320, adjusting msgr requires forclients-4> 2017-05-24 10:23:10.315258 7f24035e2800 0 osd.1908 863035 crush map has features 2234490552320 was 8705, adjusting msgr requires for mons-3> 2017-05-24 10:23:10.315267 7f24035e2800 0 osd.1908 863035 crush map has features 2234490552320, adjusting msgr requires forosds-2> 2017-05-24 10:23:10.441444 7f24035e2800 0 osd.1908 863035 load_pgs-1> 2017-05-24 10:23:10.442608 7f24035e2800 -1 osd.1908 863035 load_pgs: have pgid 11.3f5a at epoch 863078, but missing map. Crashing.0> 2017-05-24 10:23:10.444151 7f24035e2800 -1 osd/OSD.cc: In function 'void OSD::load_pgs()' thread 7f24035e2800 time 2017-05-24 10:23:10.442617osd/OSD.cc: 3189: FAILED assert(0 == "Missing map in load_pgs")ceph version 10.2.7 (50e863e0f4bc8f4b9e31156de690d765af245185)1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0x55d1874be6db]2: (OSD::load_pgs()+0x1f9b) [0x55d186e6a26b]3: (OSD::init()+0x1f74) [0x55d186e7aec4]4: (main()+0x29d1) [0x55d186de1d71]5: (__libc_start_main()+0xf5) [0x7f24004fdf45]6: (()+0x356a47) [0x55d186e2aa47]NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
Regards
Stuart Harland
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
