How you configured bluestore, all default ? i.e all in single partition , no separate partition for db/wal ? Wondering if you are out of db space/disk space ? We had some issues in this front sometimes back which was fixed, may be a new issue (?). Need verbose log for at least bluefs (debug_bluefs = 20/20) BTW, what is your workload (block size, IO pattern ) ? -----Original Message----- From: Kamble, Nitin A [mailto:Nitin.Kamble@xxxxxxxxxxxx] Sent: Friday, September 16, 2016 12:00 PM To: Somnath Roy Cc: Sage Weil; Ceph Development Subject: Re: Bluestore OSD support in ceph-disk > On Sep 16, 2016, at 11:43 AM, Somnath Roy <Somnath.Roy@xxxxxxxxxxx> wrote: > > Please send the snippet (very first trace , go up in the log) where it is actually printing the assert. > BTW, what workload you are running ? > > Thanks & Regards > Somnath > Here it is. 2016-09-16 08:49:30.605845 7fb5a96ba700 -1 /build/nitin/nightly_builds/20160914_125459-master/ceph.git/rpmbuild/BUILD/ceph-v11.0.0-2309.g9096ad3/src/os/bluestore/BlueFS.cc: In function 'int BlueFS::_allocate(uint8_t, uint64_t, std::vecto r<bluefs_extent_t>*)' thread 7fb5a96ba700 time 2016-09-16 08:49:30.602139 /build/nitin/nightly_builds/20160914_125459-master/ceph.git/rpmbuild/BUILD/ceph-v11.0.0-2309.g9096ad3/src/os/bluestore/BlueFS.cc: 1686: FAILED assert(0 == "allocate failed... wtf") ceph version v11.0.0-2309-g9096ad3 (9096ad37f2c0798c26d7784fb4e7a781feb72cb8) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0x7fb5bf43a11b] 2: (BlueFS::_allocate(unsigned char, unsigned long, std::vector<bluefs_extent_t, std::allocator<bluefs_extent_t> >*)+0x8ad) [0x7fb5bf2735dd] 3: (BlueFS::_flush_and_sync_log(std::unique_lock<std::mutex>&, unsigned long, unsigned long)+0xb4f) [0x7fb5bf27aa1f] 4: (BlueFS::_fsync(BlueFS::FileWriter*, std::unique_lock<std::mutex>&)+0x29b) [0x7fb5bf27bc9b] 5: (BlueRocksWritableFile::Sync()+0x4e) [0x7fb5bf29125e] 6: (rocksdb::WritableFileWriter::SyncInternal(bool)+0x139) [0x7fb5bf388699] 7: (rocksdb::WritableFileWriter::Sync(bool)+0x88) [0x7fb5bf389238] 8: (rocksdb::DBImpl::WriteImpl(rocksdb::WriteOptions const&, rocksdb::WriteBatch*, rocksdb::WriteCallback*, unsigned long*, unsigned long, bool)+0x13cf) [0x7fb5bf2e0a2f] 9: (rocksdb::DBImpl::Write(rocksdb::WriteOptions const&, rocksdb::WriteBatch*)+0x27) [0x7fb5bf2e1637] 10: (RocksDBStore::submit_transaction_sync(std::shared_ptr<KeyValueDB::TransactionImpl>)+0x5b) [0x7fb5bf21a14b] 11: (BlueStore::_kv_sync_thread()+0xf5a) [0x7fb5bf1e7ffa] 12: (BlueStore::KVSyncThread::entry()+0xd) [0x7fb5bf1f5a6d] 13: (()+0x80a4) [0x7fb5bb4a70a4] 14: (clone()+0x6d) [0x7fb5ba32004d] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. Thanks, Nitin PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies). -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
