Hi All!
I'm observing following crash when running Bluestore StoreTest.Synthetic
test casr( ceph_test_objectstore
--gtest_filter=ObjectStore/StoreTest.Synthetic/2 ):
-20> 2016-08-11 16:09:30.155436 7f6c1437b680 15
bluestore(store_test_temp_dir) collection_list 555.0_head start GHMIN
end GHMAX max 10
-19> 2016-08-11 16:09:30.155443 7f6c1437b680 20
bluestore(store_test_temp_dir) collection_list range
'--'0x7ffffffffffffdd300000000'.' to '--'0x7ff
ffffffffffdd3ffffffff':' and '--'0x800000000000022b00000000'.' to
'--'0x800000000000022bffffffff':' start GHMIN
-18> 2016-08-11 16:09:30.155482 7f6c1437b680 20
bluestore(store_test_temp_dir) collection_list pend
'--'0x7ffffffffffffdd3ffffffff':'
-17> 2016-08-11 16:09:30.155509 7f6c1437b680 20
bluestore(store_test_temp_dir) collection_list key
'--'0x800000000000022b00000000'.!=OBJ_1049aaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa!'0xff
fffffffffffffe0000000000000000 > GHMAX
-16> 2016-08-11 16:09:30.155515 7f6c1437b680 30
bluestore(store_test_temp_dir) collection_list switch to non-temp namespace
-15> 2016-08-11 16:09:30.155521 7f6c1437b680 30
bluestore(store_test_temp_dir) collection_list pend
'--'0x800000000000022bffffffff':'
-14> 2016-08-11 16:09:30.155524 7f6c1437b680 20
bluestore(store_test_temp_dir) collection_list key
'--'0x800000000000022b00000000'.!=OBJ_1049aaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa!'0xff
fffffffffffffe0000000000000000
-13> 2016-08-11 16:09:30.155536 7f6c1437b680 20
bluestore(store_test_temp_dir) collection_list key
'--'0x800000000000022b00000000'.!=OBJ_1105aaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa!'0x00
0000005df155b0ffffffffffffffff
-12> 2016-08-11 16:09:30.155544 7f6c1437b680 20
bluestore(store_test_temp_dir) collection_list key
'--'0x800000000000022b00000000'.!=OBJ_255aaaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa!'0x000
000007e448de90000000000000000
-11> 2016-08-11 16:09:30.155555 7f6c1437b680 20
bluestore(store_test_temp_dir) collection_list key
'--'0x800000000000022b00000000'.!=OBJ_511aaaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa!'0xfff
ffffffffffffe0000000000000001
-10> 2016-08-11 16:09:30.155563 7f6c1437b680 20
bluestore(store_test_temp_dir) collection_list key
'--'0x800000000000022b00000000'.!=OBJ_767aaaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa!'0x000
000005cab38c60000000000000001
-9> 2016-08-11 16:09:30.155574 7f6c1437b680 20
bluestore(store_test_temp_dir) collection_list key
'--'0x800000000000022b01000000'.!=OBJ_127aaaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa!'0x000
0000026f324ba0000000000000000
-8> 2016-08-11 16:09:30.155584 7f6c1437b680 20
bluestore(store_test_temp_dir) collection_list key
'--'0x800000000000022b01000000'.!=OBJ_639aaaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa!'0x000
000007365f1ee0000000000000001
-7> 2016-08-11 16:09:30.155592 7f6c1437b680 20
bluestore(store_test_temp_dir) collection_list key
'--'0x800000000000022b02000000'.!=OBJ_575aaaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa!'0x000
00000060a14630000000000000000
-6> 2016-08-11 16:09:30.155601 7f6c1437b680 20
bluestore(store_test_temp_dir) collection_list key
'--'0x800000000000022b02000000'.!=OBJ_831aaaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa!'0xfff
ffffffffffffe0000000000000000
-5> 2016-08-11 16:09:30.155612 7f6c1437b680 20
bluestore(store_test_temp_dir) collection_list key
'--'0x800000000000022b03000000 (bnode, skipping)
-4> 2016-08-11 16:09:30.155617 7f6c1437b680 20
bluestore(store_test_temp_dir) collection_list key
'--'0x800000000000022b03000000'.!=OBJ_1013aaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa!'0xff
fffffffffffffe0000000000000002
-3> 2016-08-11 16:09:30.155626 7f6c1437b680 20
bluestore(store_test_temp_dir) collection_list key
'--'0x800000000000022b03000000'.!=OBJ_1243aaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa!'0x00
0000000c94deb00000000000000000
-2> 2016-08-11 16:09:30.155631 7f6c1437b680 20
bluestore(store_test_temp_dir) collection_list reached max 10
-1> 2016-08-11 16:09:30.155636 7f6c1437b680 20
bluestore.2QCache(0xa282410) trim onodes 500 / 500 buffers 1978368 / 2000000
0> 2016-08-11 16:09:56.642872 7f6c1437b680 -1 *** Caught signal
(Segmentation fault) **
in thread 7f6c1437b680 thread_name:ceph_test_objec
ceph version v11.0.0-1434-gc6e561a
(c6e561a5385105dfbd70f642bcd5732fac0bca89)
1: bin/ceph_test_objectstore() [0xa1f207]
2: (()+0xfc90) [0x7f6c13f7bc90]
3: bin/ceph_test_objectstore() [0x8cadab]
4: (operator<<(std::ostream&, hobject_t const&)+0x229) [0x8d1709]
5: (operator<<(std::ostream&, ghobject_t const&)+0x107) [0x8d1c37]
6:
(BlueStore::collection_list(boost::intrusive_ptr<ObjectStore::CollectionImpl>&,
ghobject_t, ghobject_t, bool, int, std::vector<ghobject_t, std::al
locator<ghobject_t> >*, ghobject_t*)+0xb6d) [0x78680d]
7: (BlueStore::collection_list(coll_t const&, ghobject_t, ghobject_t,
bool, int, std::vector<ghobject_t, std::allocator<ghobject_t> >*,
ghobject_t*)+
0x1eb) [0x79d64b]
8: (SyntheticWorkloadState::shutdown()+0x2ab) [0x68b88b]
9: (doSyntheticTest(boost::scoped_ptr<ObjectStore>&, unsigned long,
unsigned long, unsigned long)+0x471) [0x651741]
It looks like it was introduced by the recent PR #10410 merge
(https://github.com/ceph/ceph/pull/10410). I don't observe the issue
prior to this patch.
Brief analysis under GDB shows that ghobject_t instance pointed by pnext
passed to operator<< at Bluestore.cc:4167 is broken - oid.name field has
huge size..
The issue is 100% reproducible.
Thanks,
Igor
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
