Hi Igor, sure, i would be appreciated if you can share your ceph config and i will try on master branch. Thanks, Sheng On Fri, Mar 31, 2017 at 10:45 AM, Igor Fedotov <ifedotov@xxxxxxxxxxxx> wrote: > Sheng, > > thanks a lot for your advise. > > Speaking of bluestore failures - would you mind to try master branch? > > I used FIO + bluestore extensively and and it works for me just fine, > perhaps some issues in kraken... > > You may try my ceph config if you like too... > > Thanks, > > Igor > > > > On 31.03.2017 20:35, sheng qiu wrote: >> >> Hi Igor, >> >> thanks for your suggestions. i was able to run fio + kstore built with >> tcmalloc. If you have problem with it, maybe try add >> LD_PRELOAD=/path/to/libfio_ceph_objectstore.so before your fio >> command. >> it fixed my problem with tcmalloc and propose better performance than >> jemalloc. >> my problem is with bluestore that produces the above segmentation fault. >> >> Sage, >> the code is cloned from ceph.git branch kraken. >> If i run fio with qd=1, it's relatively stable, but if use qd > 1, >> bluestore case frequently pop out the above segmentation fault. >> >> Thanks, >> Sheng >> >> >> >> On Fri, Mar 31, 2017 at 9:49 AM, Igor Fedotov <ifedotov@xxxxxxxxxxxx> >> wrote: >>> >>> Not sure it relates to your issue but I was unable to run fio + >>> objectstore >>> plugin built with tcmalloc. >>> >>> No issues after switching to jemalloc though, please do full revuild >>> starting with >>> >>> ./do_cmake.sh -DWITH_FIO=ON -DFIO_INCLUDE_DIR=/root/fio/ >>> -DALLOCATOR=jemalloc >>> >>> >>> Hope this helps, >>> >>> Igor >>> >>> >>> >>> On 31.03.2017 19:43, sheng qiu wrote: >>>> >>>> Hi, >>>> >>>> i am trying to use fio_ceph_objectstore as an external engine used by >>>> fio to directly generate IO to objectstore backend without running a >>>> ceph cluster. The purpose is to understand the performance of >>>> objectstore backend. >>>> >>>> i was able to run kstore relatively stable, however with bluestore, it >>>> frequently crash during the test. >>>> here's the crash log: >>>> >>>> *** Caught signal (Segmentation fault) ** >>>> in thread 7f6bc67f4700 thread_name:bstore_kv_sync >>>> ceph version ccd5a2 (4ccd5a2aafa15ccb6830fa0e339d57be67a50e24) >>>> 1: (()+0x4f309e) [0x7f6bff71509e] >>>> 2: (()+0x11390) [0x7f6bed0a8390] >>>> 3: (malloc_usable_size()+0x28) [0x7f6becb4e4e8] >>>> 4: (rocksdb::Arena::AllocateNewBlock(unsigned long)+0x7c) >>>> [0x7f6bff80072c] >>>> 5: (rocksdb::Arena::AllocateFallback(unsigned long, bool)+0x45) >>>> [0x7f6bff8008a5] >>>> 6: (rocksdb::ConcurrentArena::AllocateAligned(unsigned long, unsigned >>>> long, rocksdb::Logger*)+0x16e) [0x7f6bff78a46e] >>>> 7: (()+0x5a3c0b) [0x7f6bff7c5c0b] >>>> 8: (rocksdb::MemTable::Add(unsigned long, rocksdb::ValueType, >>>> rocksdb::Slice const&, rocksdb::Slice const&, bool, >>>> rocksdb::MemTablePostProcessInfo*)+0x8d9) [0x7f6bff7871d9] >>>> 9: (rocksdb::MemTableInserter::PutCF(unsigned int, rocksdb::Slice >>>> const&, rocksdb::Slice const&)+0x30f) [0x7f6bff7c215f] >>>> 10: (rocksdb::WriteBatch::Iterate(rocksdb::WriteBatch::Handler*) >>>> const+0x53d) [0x7f6bff7bd3cd] >>>> 11: >>>> >>>> (rocksdb::WriteBatchInternal::InsertInto(rocksdb::autovector<rocksdb::WriteThread::Writer*, >>>> 8ul> const&, unsigned long, rocksdb::ColumnFamilyMemTables*, >>>> rocksdb::FlushScheduler*, bool, unsigned long, rocksdb::DB*, >>>> bool)+0x140) [0x7f6bff7bfd50] >>>> 12: (rocksdb::DBImpl::WriteImpl(rocksdb::WriteOptions const&, >>>> rocksdb::WriteBatch*, rocksdb::WriteCallback*, unsigned long*, >>>> unsigned long, bool)+0x1617) [0x7f6bff73e3c7] >>>> 13: (rocksdb::DBImpl::Write(rocksdb::WriteOptions const&, >>>> rocksdb::WriteBatch*)+0x2a) [0x7f6bff73ec5a] >>>> 14: >>>> >>>> (RocksDBStore::submit_transaction(std::shared_ptr<KeyValueDB::TransactionImpl>)+0x249) >>>> [0x7f6bff644469] >>>> 15: (BlueStore::_kv_sync_thread()+0x12a7) [0x7f6bff5e5037] >>>> 16: (BlueStore::KVSyncThread::entry()+0xd) [0x7f6bff6299ed] >>>> 17: (Thread::entry_wrapper()+0x75) [0x7f6bff923335] >>>> 18: (()+0x76ba) [0x7f6bed09e6ba] >>>> 19: (clone()+0x6d) [0x7f6becbd082d] >>>> >>>> is there any clue how to fix it? any help would be appreciated. >>>> >>>> Thanks, >>>> Sheng >>>> -- >>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in >>>> the body of a message to majordomo@xxxxxxxxxxxxxxx >>>> More majordomo info at http://vger.kernel.org/majordomo-info.html >>> >>> > -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
