Hello,
Another OSD started randomly crashing with segmentation fault. Haven't
managed to add the last 3 OSDs back to the cluster as the daemons keep
crashing.
---
-2> 2018-08-03 12:12:52.670076 7f12b6b15700 4 rocksdb:
EVENT_LOG_v1 {"time_micros": 1533287572670073, "job": 3, "event":
"table_file_deletion", "file_number": 4350}
-1> 2018-08-03 12:12:53.146753 7f12c38d0a80 0 osd.154 89917 load_pgs
0> 2018-08-03 12:12:57.526910 7f12c38d0a80 -1 *** Caught signal
(Segmentation fault) **
in thread 7f12c38d0a80 thread_name:ceph-osd
ceph version 10.2.11 (e4b061b47f07f583c92a050d9e84b1813a35671e)
1: (()+0x9f1c2a) [0x7f12c42ddc2a]
2: (()+0xf5e0) [0x7f12c1dc85e0]
3: (()+0x34484) [0x7f12c34a6484]
4: (rocksdb::BlockBasedTable::NewIndexIterator(rocksdb::ReadOptions
const&, rocksdb::BlockIter*,
rocksdb::BlockBasedTable::CachableEntry<rocksdb::BlockBasedTable::IndexReader>*)+0x466)
[0x7f12c41e40d6]
5: (rocksdb::BlockBasedTable::Get(rocksdb::ReadOptions const&,
rocksdb::Slice const&, rocksdb::GetContext*, bool)+0x297)
[0x7f12c41e4b27]
6: (rocksdb::TableCache::Get(rocksdb::ReadOptions const&,
rocksdb::InternalKeyComparator const&, rocksdb::FileDescriptor const&,
rocksdb::Slice const&, rocksdb::GetContext*, rocksdb::HistogramImpl*,
bool
, int)+0x2a4) [0x7f12c429ff94]
7: (rocksdb::Version::Get(rocksdb::ReadOptions const&,
rocksdb::LookupKey const&, rocksdb::PinnableSlice*, rocksdb::Status*,
rocksdb::MergeContext*, rocksdb::RangeDelAggregator*, bool*, bool*,
unsigned l
ong*)+0x810) [0x7f12c419bb80]
8: (rocksdb::DBImpl::GetImpl(rocksdb::ReadOptions const&,
rocksdb::ColumnFamilyHandle*, rocksdb::Slice const&,
rocksdb::PinnableSlice*, bool*)+0x5a4) [0x7f12c424e494]
9: (rocksdb::DBImpl::Get(rocksdb::ReadOptions const&,
rocksdb::ColumnFamilyHandle*, rocksdb::Slice const&,
rocksdb::PinnableSlice*)+0x19) [0x7f12c424ea19]
10: (rocksdb::DB::Get(rocksdb::ReadOptions const&,
rocksdb::ColumnFamilyHandle*, rocksdb::Slice const&,
std::string*)+0x95) [0x7f12c4252a45]
11: (rocksdb::DB::Get(rocksdb::ReadOptions const&, rocksdb::Slice
const&, std::string*)+0x4a) [0x7f12c4251eea]
12: (RocksDBStore::get(std::string const&, std::string const&,
ceph::buffer::list*)+0xff) [0x7f12c415c31f]
13: (DBObjectMap::_lookup_map_header(DBObjectMap::MapHeaderLock
const&, ghobject_t const&)+0x5e4) [0x7f12c4110814]
14: (DBObjectMap::get_values(ghobject_t const&, std::set<std::string,
std::less<std::string>, std::allocator<std::string> > const&,
std::map<std::string, ceph::buffer::list, std::less<std::string>,
std::
allocator<std::pair<std::string const, ceph::buffer::list> > >*)+0x5f)
[0x7f12c411111f]
15: (FileStore::omap_get_values(coll_t const&, ghobject_t const&,
std::set<std::string, std::less<std::string>,
std::allocator<std::string> > const&, std::map<std::string,
ceph::buffer::list, std::less<s
td::string>, std::allocator<std::pair<std::string const,
ceph::buffer::list> > >*)+0x197) [0x7f12c4031f77]
16: (PG::_has_removal_flag(ObjectStore*, spg_t)+0x151) [0x7f12c3d8f7c1]
17: (OSD::load_pgs()+0x5d5) [0x7f12c3cf43e5]
18: (OSD::init()+0x2086) [0x7f12c3d07096]
19: (main()+0x2c18) [0x7f12c3c1e088]
20: (__libc_start_main()+0xf5) [0x7f12c0374c05]
21: (()+0x3c8847) [0x7f12c3cb4847]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is
needed to interpret this.
---
Any help would be appreciated.
Thanks,
Alex Cucu
On Mon, Jul 30, 2018 at 4:55 PM Alexandru Cucu <me@xxxxxxxxxxx> wrote:
>
> Hello Ceph users,
>
> We have updated our cluster from 10.2.7 to 10.2.11. A few hours after
> the update, 1 OSD crashed.
> When trying to add the OSD back to the cluster, other 2 OSDs started
> crashing with segmentation fault. Had to mark all 3 OSDs as down as we
> had stuck PGs and blocked operations and the cluster status was
> HEALTH_ERR.
>
> We have tried various ways to re-add the OSDs back to the cluster but
> after a while they start crashing and won't start anymore. After a
> while they can be started again and marked as in but after some
> rebalancing they will start the crashing imediately after starting.
>
> Here are some logs:
> https://pastebin.com/nCRamgRU
>
> Do you know of any existing bug report that might be related? (I
> couldn't find anything).
>
> I will happily provide any information that would help solving this issue.
>
> Thank you,
> Alex Cucu
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
