Re: Fio randwrite does not work on Centos 7.2 VM

[Mansour Shafaei Moghaddam <mansoor.shafaei@xxxxxxxxx>]

It fails at "FileStore.cc: 2761". Here is a more complete log:

    -9> 2016-06-15 10:55:13.205014 7fa2dcd85700 -1 dump_open_fds unable to open /proc/self/fd

    -8> 2016-06-15 10:55:13.205085 7fa2cb402700  2 filestore(/var/lib/ceph/osd/ceph-0) waiting 51 > 50 ops || 328390 > 104857600

    -7> 2016-06-15 10:55:13.205094 7fa2cd406700  2 filestore(/var/lib/ceph/osd/ceph-0) waiting 51 > 50 ops || 328389 > 104857600

    -6> 2016-06-15 10:55:13.205111 7fa2cac01700  2 filestore(/var/lib/ceph/osd/ceph-0) waiting 51 > 50 ops || 328317 > 104857600

    -5> 2016-06-15 10:55:13.205118 7fa2ca400700  2 filestore(/var/lib/ceph/osd/ceph-0) waiting 51 > 50 ops || 328390 > 104857600

    -4> 2016-06-15 10:55:13.205121 7fa2cdc07700  2 filestore(/var/lib/ceph/osd/ceph-0) waiting 51 > 50 ops || 328390 > 104857600

    -3> 2016-06-15 10:55:13.205153 7fa2de588700  5 -- op tracker -- seq: 1476, time: 2016-06-15 10:55:13.205153, event: journaled_completion_queued, op: osd_op(client.4109.0:1457 rb.0.100a.6b8b4567.000000006b6c [set-alloc-hint object_size 4194304 write_size 4194304,write 1884160~4096] 0.cbe1d8a4 ack+ondisk+write e9)

    -2> 2016-06-15 10:55:13.205183 7fa2de588700  5 -- op tracker -- seq: 1483, time: 2016-06-15 10:55:13.205183, event: write_thread_in_journal_buffer, op: osd_op(client.4109.0:1464 rb.0.100a.6b8b4567.00000000524d [set-alloc-hint object_size 4194304 write_size 4194304,write 3051520~4096] 0.6778c255 ack+ondisk+write e9)

    -1> 2016-06-15 10:55:13.205400 7fa2de588700  5 -- op tracker -- seq: 1483, time: 2016-06-15 10:55:13.205400, event: journaled_completion_queued, op: osd_op(client.4109.0:1464 rb.0.100a.6b8b4567.00000000524d [set-alloc-hint object_size 4194304 write_size 4194304,write 3051520~4096] 0.6778c255 ack+ondisk+write e9)

     0> 2016-06-15 10:55:13.206559 7fa2dcd85700 -1 os/FileStore.cc: In function 'unsigned int FileStore::_do_transaction(ObjectStore::Transaction&, uint64_t, int, ThreadPool::TPHandle*)' thread 7fa2dcd85700 time 2016-06-15 10:55:13.205018

os/FileStore.cc: 2761: FAILED assert(0 == "unexpected error")

 ceph version 0.94.6 (e832001feaf8c176593e0325c8298e3f16dfb403)

 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x78) [0xacd718]

 2: (FileStore::_do_transaction(ObjectStore::Transaction&, unsigned long, int, ThreadPool::TPHandle*)+0xa24) [0x8b8114]

 3: (FileStore::_do_transactions(std::list<ObjectStore::Transaction*, std::allocator<ObjectStore::Transaction*> >&, unsigned long, ThreadPool::TPHandle*)+0x64) [0x8bcf34]

 4: (FileStore::_do_op(FileStore::OpSequencer*, ThreadPool::TPHandle&)+0x17e) [0x8bd0ce]

 5: (ThreadPool::worker(ThreadPool::WorkThread*)+0xa56) [0xabe326]

 6: (ThreadPool::WorkThread::entry()+0x10) [0xabf3d0]

 7: (()+0x7dc5) [0x7fa2e88f3dc5]

 8: (clone()+0x6d) [0x7fa2e73d528d]

On Wed, Jun 15, 2016 at 2:05 PM, Somnath Roy <Somnath.Roy@xxxxxxxxxxx> wrote:

There should be a line in the log specifying which assert is failing , post that along with say 10 lines from top of that..

 

Thanks & Regards

Somnath

 

From: ceph-users [mailto:ceph-users-bounces@xxxxxxxxxxxxxx] On Behalf Of Mansour Shafaei Moghaddam
Sent: Wednesday, June 15, 2016 1:57 PM
To: ceph-users@xxxxxxxxxxxxxx
Subject: Fio randwrite does not work on Centos 7.2 VM

 

Hi All,

 

Has anyone faced a similar issue? I do not have a problem with random read, sequential read, and sequential writes though. Everytime I try running fio for random writes, one osd in the cluster crashes. Here is the what I see at the tail of the log:

 

 ceph version 0.94.6 (e832001feaf8c176593e0325c8298e3f16dfb403)

 1: ceph-osd() [0x9d6334]

 2: (()+0xf100) [0x7fa2e88fb100]

 3: (gsignal()+0x37) [0x7fa2e73145f7]

 4: (abort()+0x148) [0x7fa2e7315ce8]

 5: (__gnu_cxx::__verbose_terminate_handler()+0x165) [0x7fa2e7c189d5]

 6: (()+0x5e946) [0x7fa2e7c16946]

 7: (()+0x5e973) [0x7fa2e7c16973]

 8: (()+0x5eb93) [0x7fa2e7c16b93]

 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x24a) [0xacd8ea]

 10: (FileStore::_do_transaction(ObjectStore::Transaction&, unsigned long, int, ThreadPool::TPHandle*)+0xa24) [0x8b8114]

 11: (FileStore::_do_transactions(std::list<ObjectStore::Transaction*, std::allocator<ObjectStore::Transaction*> >&, unsigned long, ThreadPool::TPHandle*)+0x64) [0x8bcf34]

 12: (FileStore::_do_op(FileStore::OpSequencer*, ThreadPool::TPHandle&)+0x17e) [0x8bd0ce]

 13: (ThreadPool::worker(ThreadPool::WorkThread*)+0xa56) [0xabe326]

 14: (ThreadPool::WorkThread::entry()+0x10) [0xabf3d0]

 15: (()+0x7dc5) [0x7fa2e88f3dc5]

 16: (clone()+0x6d) [0x7fa2e73d528d]

 

 

PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies).

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com