Hi,
Already tried zapping the disk. Unfortunaltely the same segfaults keep
me from adding the OSD back to the cluster.
I wanted to open an issue on tracker.ceph.com but I can't find the
"new issue" button.
---
Alex Cucu
On Mon, Aug 13, 2018 at 8:24 AM <ceph@xxxxxxxxxx> wrote:
>
>
>
> Am 3. August 2018 12:03:17 MESZ schrieb Alexandru Cucu <me@xxxxxxxxxxx>:
> >Hello,
> >
>
> Hello Alex,
>
> >Another OSD started randomly crashing with segmentation fault. Haven't
> >managed to add the last 3 OSDs back to the cluster as the daemons keep
> >crashing.
> >
>
> An idea could be to remove the osds completely from the Cluster and add it again after zapping the Disks.
>
> Hth
> - Mehmet
>
> >---
> >
> > -2> 2018-08-03 12:12:52.670076 7f12b6b15700 4 rocksdb:
> >EVENT_LOG_v1 {"time_micros": 1533287572670073, "job": 3, "event":
> >"table_file_deletion", "file_number": 4350}
> > -1> 2018-08-03 12:12:53.146753 7f12c38d0a80 0 osd.154 89917 load_pgs
> > 0> 2018-08-03 12:12:57.526910 7f12c38d0a80 -1 *** Caught signal
> >(Segmentation fault) **
> > in thread 7f12c38d0a80 thread_name:ceph-osd
> > ceph version 10.2.11 (e4b061b47f07f583c92a050d9e84b1813a35671e)
> > 1: (()+0x9f1c2a) [0x7f12c42ddc2a]
> > 2: (()+0xf5e0) [0x7f12c1dc85e0]
> > 3: (()+0x34484) [0x7f12c34a6484]
> > 4: (rocksdb::BlockBasedTable::NewIndexIterator(rocksdb::ReadOptions
> >const&, rocksdb::BlockIter*,
> >rocksdb::BlockBasedTable::CachableEntry<rocksdb::BlockBasedTable::IndexReader>*)+0x466)
> >[0x7f12c41e40d6]
> > 5: (rocksdb::BlockBasedTable::Get(rocksdb::ReadOptions const&,
> >rocksdb::Slice const&, rocksdb::GetContext*, bool)+0x297)
> >[0x7f12c41e4b27]
> > 6: (rocksdb::TableCache::Get(rocksdb::ReadOptions const&,
> >rocksdb::InternalKeyComparator const&, rocksdb::FileDescriptor const&,
> >rocksdb::Slice const&, rocksdb::GetContext*, rocksdb::HistogramImpl*,
> >bool
> >, int)+0x2a4) [0x7f12c429ff94]
> > 7: (rocksdb::Version::Get(rocksdb::ReadOptions const&,
> >rocksdb::LookupKey const&, rocksdb::PinnableSlice*, rocksdb::Status*,
> >rocksdb::MergeContext*, rocksdb::RangeDelAggregator*, bool*, bool*,
> >unsigned l
> >ong*)+0x810) [0x7f12c419bb80]
> > 8: (rocksdb::DBImpl::GetImpl(rocksdb::ReadOptions const&,
> >rocksdb::ColumnFamilyHandle*, rocksdb::Slice const&,
> >rocksdb::PinnableSlice*, bool*)+0x5a4) [0x7f12c424e494]
> > 9: (rocksdb::DBImpl::Get(rocksdb::ReadOptions const&,
> >rocksdb::ColumnFamilyHandle*, rocksdb::Slice const&,
> >rocksdb::PinnableSlice*)+0x19) [0x7f12c424ea19]
> > 10: (rocksdb::DB::Get(rocksdb::ReadOptions const&,
> >rocksdb::ColumnFamilyHandle*, rocksdb::Slice const&,
> >std::string*)+0x95) [0x7f12c4252a45]
> > 11: (rocksdb::DB::Get(rocksdb::ReadOptions const&, rocksdb::Slice
> >const&, std::string*)+0x4a) [0x7f12c4251eea]
> > 12: (RocksDBStore::get(std::string const&, std::string const&,
> >ceph::buffer::list*)+0xff) [0x7f12c415c31f]
> > 13: (DBObjectMap::_lookup_map_header(DBObjectMap::MapHeaderLock
> >const&, ghobject_t const&)+0x5e4) [0x7f12c4110814]
> > 14: (DBObjectMap::get_values(ghobject_t const&, std::set<std::string,
> >std::less<std::string>, std::allocator<std::string> > const&,
> >std::map<std::string, ceph::buffer::list, std::less<std::string>,
> >std::
> >allocator<std::pair<std::string const, ceph::buffer::list> > >*)+0x5f)
> >[0x7f12c411111f]
> > 15: (FileStore::omap_get_values(coll_t const&, ghobject_t const&,
> >std::set<std::string, std::less<std::string>,
> >std::allocator<std::string> > const&, std::map<std::string,
> >ceph::buffer::list, std::less<s
> >td::string>, std::allocator<std::pair<std::string const,
> >ceph::buffer::list> > >*)+0x197) [0x7f12c4031f77]
> >16: (PG::_has_removal_flag(ObjectStore*, spg_t)+0x151) [0x7f12c3d8f7c1]
> > 17: (OSD::load_pgs()+0x5d5) [0x7f12c3cf43e5]
> > 18: (OSD::init()+0x2086) [0x7f12c3d07096]
> > 19: (main()+0x2c18) [0x7f12c3c1e088]
> > 20: (__libc_start_main()+0xf5) [0x7f12c0374c05]
> > 21: (()+0x3c8847) [0x7f12c3cb4847]
> > NOTE: a copy of the executable, or `objdump -rdS <executable>` is
> >needed to interpret this.
> >---
> >
> >Any help would be appreciated.
> >
> >Thanks,
> >Alex Cucu
> >
> >On Mon, Jul 30, 2018 at 4:55 PM Alexandru Cucu <me@xxxxxxxxxxx> wrote:
> >>
> >> Hello Ceph users,
> >>
> >> We have updated our cluster from 10.2.7 to 10.2.11. A few hours after
> >> the update, 1 OSD crashed.
> >> When trying to add the OSD back to the cluster, other 2 OSDs started
> >> crashing with segmentation fault. Had to mark all 3 OSDs as down as
> >we
> >> had stuck PGs and blocked operations and the cluster status was
> >> HEALTH_ERR.
> >>
> >> We have tried various ways to re-add the OSDs back to the cluster but
> >> after a while they start crashing and won't start anymore. After a
> >> while they can be started again and marked as in but after some
> >> rebalancing they will start the crashing imediately after starting.
> >>
> >> Here are some logs:
> >> https://pastebin.com/nCRamgRU
> >>
> >> Do you know of any existing bug report that might be related? (I
> >> couldn't find anything).
> >>
> >> I will happily provide any information that would help solving this
> >issue.
> >>
> >> Thank you,
> >> Alex Cucu
> >_______________________________________________
> >ceph-users mailing list
> >ceph-users@xxxxxxxxxxxxxx
> >http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> _______________________________________________
> ceph-users mailing list
> ceph-users@xxxxxxxxxxxxxx
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
