Hello! I have a small strange problem in my a cluster. The cluster works well until i'm add the "user_subvol_rm_allowed" mount option for btrfs filestore in my ceph.conf. After restart an host, in an osd logs i found: *** 2016-09-19 16:47:02.189162 7fd732e21840 2 osd.10 0 mounting /var/lib/ceph/osd/ceph-10 /var/lib/ceph/osd/ceph-10/journal 2016-09-19 16:47:02.189176 7fd732e21840 5 filestore(/var/lib/ceph/osd/ceph-10) basedir /var/lib/ceph/osd/ceph-10 journal /var/lib/ceph/osd/ceph-10/journal 2016-09-19 16:47:02.189195 7fd732e21840 10 filestore(/var/lib/ceph/osd/ceph-10) mount fsid is f1685d33-9572-4500-b61d-91d8462f8df0 2016-09-19 16:47:02.189219 7fd732e21840 10 filestore(/var/lib/ceph/osd/ceph-10) version_stamp_is_valid was 4 vs target 4 2016-09-19 16:47:02.189248 7fd732e21840 0 filestore(/var/lib/ceph/osd/ceph-10) backend btrfs (magic 0x9123683e) 2016-09-19 16:47:02.189594 7fd732e21840 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-10) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option 2016-09-19 16:47:02.189607 7fd732e21840 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-10) detect_features: lseek SEEK_DATA/SEEK_HOLE is supported 2016-09-19 16:47:02.189635 7fd732e21840 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-10) detect_features: splice is supported 2016-09-19 16:47:02.284964 7fd732e21840 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-10) detect_features: syncfs(2) syscall fully supported (by glibc and kernel) 2016-09-19 16:47:02.285125 7fd732e21840 0 btrfsfilestorebackend(/var/lib/ceph/osd/ceph-10) detect_feature: CLONE_RANGE ioctl is supported 2016-09-19 16:47:02.443329 7fd732e21840 0 btrfsfilestorebackend(/var/lib/ceph/osd/ceph-10) detect_feature: SNAP_CREATE is supported 2016-09-19 16:47:02.443531 7fd732e21840 0 btrfsfilestorebackend(/var/lib/ceph/osd/ceph-10) detect_feature: SNAP_DESTROY is supported 2016-09-19 16:47:02.443736 7fd732e21840 0 btrfsfilestorebackend(/var/lib/ceph/osd/ceph-10) detect_feature: START_SYNC is supported (transid 54316) 2016-09-19 16:47:02.526714 7fd732e21840 0 btrfsfilestorebackend(/var/lib/ceph/osd/ceph-10) detect_feature: WAIT_SYNC is supported 2016-09-19 16:47:02.560839 7fd732e21840 0 btrfsfilestorebackend(/var/lib/ceph/osd/ceph-10) detect_feature: SNAP_CREATE_V2 is supported 2016-09-19 16:47:02.561603 7fd732e21840 10 filestore(/var/lib/ceph/osd/ceph-10) current/ seq was 150825 2016-09-19 16:47:02.561612 7fd732e21840 10 filestore(/var/lib/ceph/osd/ceph-10) most recent snap from <1> is 1 2016-09-19 16:47:02.561624 7fd732e21840 -1 filestore(/var/lib/ceph/osd/ceph-10) ERROR: /var/lib/ceph/osd/ceph-10/current/nosnap exists, not rolling back to avoid losing new data 2016-09-19 16:47:02.561627 7fd732e21840 -1 filestore(/var/lib/ceph/osd/ceph-10) Force rollback to old snapshotted version with 'osd use stale snap = true' 2016-09-19 16:47:02.561628 7fd732e21840 -1 filestore(/var/lib/ceph/osd/ceph-10) config option for --osd-use-stale-snap startup argument. 2016-09-19 16:47:02.561635 7fd732e21840 -1 osd.10 0 OSD:init: unable to mount object store 2016-09-19 16:47:02.561642 7fd732e21840 -1 ESC[0;31m ** ERROR: osd init failed: (95) Operation not supportedESC[0m *** If I set the "osd use stale snap" to "true" I saw another an error: *** 2016-09-19 17:14:59.740274 7f4edc7c8840 20 filestore (init)dbobjectmap: seq is 1 2016-09-19 17:14:59.740289 7f4edc7c8840 10 filestore(/var/lib/ceph/osd/ceph-10) open_journal at /var/lib/ceph/osd/ceph-10/journal 2016-09-19 17:14:59.740326 7f4edc7c8840 0 filestore(/var/lib/ceph/osd/ceph-10) mount: PARALLEL journal mode explicitly enabled in conf 2016-09-19 17:14:59.740336 7f4edc7c8840 10 filestore(/var/lib/ceph/osd/ceph-10) list_collections 2016-09-19 17:14:59.740459 7f4edc7c8840 10 journal journal_replay fs op_seq 1 2016-09-19 17:14:59.740471 7f4edc7c8840 2 journal open /var/lib/ceph/osd/ceph-10/journal fsid f1685d33-9572-4500-b61d-91d8462f8df0 fs_op_seq 1 2016-09-19 17:14:59.740500 7f4edc7c8840 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2016-09-19 17:14:59.740509 7f4edc7c8840 10 journal _open journal is not a block device, NOT checking disk write cache on '/var/lib/ceph/osd/ceph-10/journal' 2016-09-19 17:14:59.740516 7f4edc7c8840 1 journal _open /var/lib/ceph/osd/ceph-10/journal fd 18: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 0 2016-09-19 17:14:59.740524 7f4edc7c8840 10 journal read_header 2016-09-19 17:14:59.740510 7f4eced13700 20 filestore(/var/lib/ceph/osd/ceph-10) sync_entry waiting for max_interval 5.000000 2016-09-19 17:14:59.753513 7f4edc7c8840 10 journal header: block_size 4096 alignment 4096 max_size 5368709120 2016-09-19 17:14:59.753528 7f4edc7c8840 10 journal header: start 2399928320 2016-09-19 17:14:59.753530 7f4edc7c8840 10 journal write_pos 4096 2016-09-19 17:14:59.753535 7f4edc7c8840 10 journal open header.fsid = f1685d33-9572-4500-b61d-91d8462f8df0 2016-09-19 17:14:59.781344 7f4edc7c8840 2 journal No further valid entries found, journal is most likely valid 2016-09-19 17:14:59.781355 7f4edc7c8840 10 journal open reached end of journal. 2016-09-19 17:14:59.781404 7f4edc7c8840 -1 journal Unable to read past sequence 2 but header indicates the journal has committed up through 150825, journal is corrupt 2016-09-19 17:14:59.785143 7f4edc7c8840 -1 os/filestore/FileJournal.cc: In function 'bool FileJournal::read_entry(ceph::bufferlist&, uint64_t&, bool*)' thread 7f4edc7c8840 time 2016-09-19 17:14:59.781414 os/filestore/FileJournal.cc: 2031: FAILED assert(0) ceph version 10.2.2 (45107e21c568dd033c2f0a3107dec8f0b0e58374) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x85) [0x7f4edd1f55b5] 2: (FileJournal::read_entry(ceph::buffer::list&, unsigned long&, bool*)+0x90c) [0x7f4edcf99d0c] 3: (JournalingObjectStore::journal_replay(unsigned long)+0x1ee) [0x7f4edcee9cde] 4: (FileStore::mount()+0x3cd6) [0x7f4edcec14d6] 5: (OSD::init()+0x27d) [0x7f4edcb8504d] 6: (main()+0x2c55) [0x7f4edcaeabe5] 7: (__libc_start_main()+0xf5) [0x7f4ed94a5b15] 8: (()+0x353009) [0x7f4edcb35009] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. *** I'm stuck. I can't understand what I did wrong and how recover the OSDs? Googling didn't help me. -- Mike, runs! _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |