We've been doing some performance testing on Bluestore to see whether it could be viable to use in the future.
The good news we are seeing significant performance improvements on using it, so thank you for all the work that has gone into it.
The bad news is we keep encountering crashes and corruption requiring the rebuild: Example log extract looks like the following:
2016-09-03 17:56:57.337756 7f593fe9b700 -1 freelist release bad release 564521787392~4096 overlaps with 564521787392~4096
2016-09-03 17:56:57.340169 7f593fe9b700 -1 os/bluestore/FreelistManager.cc: In function 'int FreelistManager::release(uint64_t, uint64_t, KeyValueDB::Transaction)' thread 7f593fe9b700 time 2016-09-03 17:56:57.338393
os/bluestore/FreelistManager.cc: 245: FAILED assert(0 == "bad release overlap")
ceph version 10.2.2 (45107e21c568dd033c2f0a3107dec8f0b0e58374)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x85) [0x7f59643945b5]
2: (FreelistManager::release(unsigned long, unsigned long, std::shared_ptr<KeyValueDB::TransactionImpl>)+0x533) [0x7f596402c963]
3: (BlueStore::_txc_update_fm(BlueStore::TransContext*)+0x317) [0x7f5963fda0d7]
4: (BlueStore::_kv_sync_thread()+0x9b0) [0x7f5964003690]
5: (BlueStore::KVSyncThread::entry()+0xd) [0x7f59640299dd]
6: (()+0x7dc5) [0x7f59622c4dc5]
7: (clone()+0x6d) [0x7f596095021d]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
--- begin dump of recent events ---
-10000> 2016-09-03 17:43:23.940910 7f593fe9b700 5 rocksdb: EmitPhysicalRecord: log 37 offset 70561038 len 23 crc 696327790
-9999> 2016-09-03 17:43:23.965781 7f593fe9b700 5 rocksdb: EmitPhysicalRecord: log 37 offset 70561061 len 2178 crc 2109984297
-9998> 2016-09-03 17:43:23.965833 7f593fe9b700 5 rocksdb: EmitPhysicalRecord: log 37 offset 70563239 len 2175 crc 493419836
-9997> 2016-09-03 17:43:23.965867 7f593fe9b700 5 rocksdb: EmitPhysicalRecord: log 37 offset 70565414 len 23 crc 1766806723
...
This appears to match this existing bug: http://tracker.ceph.com/issues/15659
Are there any know work-arounds to prevent the issue from happening?
What information and support can we provide to help the fix on the issue to be progressed? We seem to be able to reliably reproduce the issue after about 6 hours or so of test running so would be able to test any proposed fixes if that would be helpful,
Thanks,
Thomas
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
