Jewel is almost EOL. It looks similar to several related issues, one of which is http://tracker.ceph.com/issues/21826 On Mon, Aug 13, 2018 at 9:19 PM, Alexandru Cucu <me@xxxxxxxxxxx> wrote: > Hi, > > Already tried zapping the disk. Unfortunaltely the same segfaults keep > me from adding the OSD back to the cluster. > > I wanted to open an issue on tracker.ceph.com but I can't find the > "new issue" button. > > --- > Alex Cucu > > On Mon, Aug 13, 2018 at 8:24 AM <ceph@xxxxxxxxxx> wrote: >> >> >> >> Am 3. August 2018 12:03:17 MESZ schrieb Alexandru Cucu <me@xxxxxxxxxxx>: >> >Hello, >> > >> >> Hello Alex, >> >> >Another OSD started randomly crashing with segmentation fault. Haven't >> >managed to add the last 3 OSDs back to the cluster as the daemons keep >> >crashing. >> > >> >> An idea could be to remove the osds completely from the Cluster and add it again after zapping the Disks. >> >> Hth >> - Mehmet >> >> >--- >> > >> > -2> 2018-08-03 12:12:52.670076 7f12b6b15700 4 rocksdb: >> >EVENT_LOG_v1 {"time_micros": 1533287572670073, "job": 3, "event": >> >"table_file_deletion", "file_number": 4350} >> > -1> 2018-08-03 12:12:53.146753 7f12c38d0a80 0 osd.154 89917 load_pgs >> > 0> 2018-08-03 12:12:57.526910 7f12c38d0a80 -1 *** Caught signal >> >(Segmentation fault) ** >> > in thread 7f12c38d0a80 thread_name:ceph-osd >> > ceph version 10.2.11 (e4b061b47f07f583c92a050d9e84b1813a35671e) >> > 1: (()+0x9f1c2a) [0x7f12c42ddc2a] >> > 2: (()+0xf5e0) [0x7f12c1dc85e0] >> > 3: (()+0x34484) [0x7f12c34a6484] >> > 4: (rocksdb::BlockBasedTable::NewIndexIterator(rocksdb::ReadOptions >> >const&, rocksdb::BlockIter*, >> >rocksdb::BlockBasedTable::CachableEntry<rocksdb::BlockBasedTable::IndexReader>*)+0x466) >> >[0x7f12c41e40d6] >> > 5: (rocksdb::BlockBasedTable::Get(rocksdb::ReadOptions const&, >> >rocksdb::Slice const&, rocksdb::GetContext*, bool)+0x297) >> >[0x7f12c41e4b27] >> > 6: (rocksdb::TableCache::Get(rocksdb::ReadOptions const&, >> >rocksdb::InternalKeyComparator const&, rocksdb::FileDescriptor const&, >> >rocksdb::Slice const&, rocksdb::GetContext*, rocksdb::HistogramImpl*, >> >bool >> >, int)+0x2a4) [0x7f12c429ff94] >> > 7: (rocksdb::Version::Get(rocksdb::ReadOptions const&, >> >rocksdb::LookupKey const&, rocksdb::PinnableSlice*, rocksdb::Status*, >> >rocksdb::MergeContext*, rocksdb::RangeDelAggregator*, bool*, bool*, >> >unsigned l >> >ong*)+0x810) [0x7f12c419bb80] >> > 8: (rocksdb::DBImpl::GetImpl(rocksdb::ReadOptions const&, >> >rocksdb::ColumnFamilyHandle*, rocksdb::Slice const&, >> >rocksdb::PinnableSlice*, bool*)+0x5a4) [0x7f12c424e494] >> > 9: (rocksdb::DBImpl::Get(rocksdb::ReadOptions const&, >> >rocksdb::ColumnFamilyHandle*, rocksdb::Slice const&, >> >rocksdb::PinnableSlice*)+0x19) [0x7f12c424ea19] >> > 10: (rocksdb::DB::Get(rocksdb::ReadOptions const&, >> >rocksdb::ColumnFamilyHandle*, rocksdb::Slice const&, >> >std::string*)+0x95) [0x7f12c4252a45] >> > 11: (rocksdb::DB::Get(rocksdb::ReadOptions const&, rocksdb::Slice >> >const&, std::string*)+0x4a) [0x7f12c4251eea] >> > 12: (RocksDBStore::get(std::string const&, std::string const&, >> >ceph::buffer::list*)+0xff) [0x7f12c415c31f] >> > 13: (DBObjectMap::_lookup_map_header(DBObjectMap::MapHeaderLock >> >const&, ghobject_t const&)+0x5e4) [0x7f12c4110814] >> > 14: (DBObjectMap::get_values(ghobject_t const&, std::set<std::string, >> >std::less<std::string>, std::allocator<std::string> > const&, >> >std::map<std::string, ceph::buffer::list, std::less<std::string>, >> >std:: >> >allocator<std::pair<std::string const, ceph::buffer::list> > >*)+0x5f) >> >[0x7f12c411111f] >> > 15: (FileStore::omap_get_values(coll_t const&, ghobject_t const&, >> >std::set<std::string, std::less<std::string>, >> >std::allocator<std::string> > const&, std::map<std::string, >> >ceph::buffer::list, std::less<s >> >td::string>, std::allocator<std::pair<std::string const, >> >ceph::buffer::list> > >*)+0x197) [0x7f12c4031f77] >> >16: (PG::_has_removal_flag(ObjectStore*, spg_t)+0x151) [0x7f12c3d8f7c1] >> > 17: (OSD::load_pgs()+0x5d5) [0x7f12c3cf43e5] >> > 18: (OSD::init()+0x2086) [0x7f12c3d07096] >> > 19: (main()+0x2c18) [0x7f12c3c1e088] >> > 20: (__libc_start_main()+0xf5) [0x7f12c0374c05] >> > 21: (()+0x3c8847) [0x7f12c3cb4847] >> > NOTE: a copy of the executable, or `objdump -rdS <executable>` is >> >needed to interpret this. >> >--- >> > >> >Any help would be appreciated. >> > >> >Thanks, >> >Alex Cucu >> > >> >On Mon, Jul 30, 2018 at 4:55 PM Alexandru Cucu <me@xxxxxxxxxxx> wrote: >> >> >> >> Hello Ceph users, >> >> >> >> We have updated our cluster from 10.2.7 to 10.2.11. A few hours after >> >> the update, 1 OSD crashed. >> >> When trying to add the OSD back to the cluster, other 2 OSDs started >> >> crashing with segmentation fault. Had to mark all 3 OSDs as down as >> >we >> >> had stuck PGs and blocked operations and the cluster status was >> >> HEALTH_ERR. >> >> >> >> We have tried various ways to re-add the OSDs back to the cluster but >> >> after a while they start crashing and won't start anymore. After a >> >> while they can be started again and marked as in but after some >> >> rebalancing they will start the crashing imediately after starting. >> >> >> >> Here are some logs: >> >> https://pastebin.com/nCRamgRU >> >> >> >> Do you know of any existing bug report that might be related? (I >> >> couldn't find anything). >> >> >> >> I will happily provide any information that would help solving this >> >issue. >> >> >> >> Thank you, >> >> Alex Cucu >> >_______________________________________________ >> >ceph-users mailing list >> >ceph-users@xxxxxxxxxxxxxx >> >http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> _______________________________________________ >> ceph-users mailing list >> ceph-users@xxxxxxxxxxxxxx >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > _______________________________________________ > ceph-users mailing list > ceph-users@xxxxxxxxxxxxxx > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com -- Cheers, Brad _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |