Hi Sage:
In my virtual machine environment, when the queue depth set with 64, then the iops slowed to 0 immediately.
But in physics environment, the queue depth must be larger if we want to reproduct.
Slow request is seen at ceph.log, I opened the osd_enable_op_tracker option.
Here are some logs:
2016-07-03 04:28:02.870324 7fe57c454700 10 journal op_submit_start 92941 <======= *** start
2016-07-03 04:28:02.870336 7fe57c454700 10 journal op_journal_transactions 92941 0x3dfb5f8
2016-07-03 04:28:02.870345 7fe57c454700 5 journal submit_entry seq 92941 len 6440 (0x7e8ae70)
2016-07-03 04:28:02.870356 7fe57c454700 10 journal op_submit_finish 92941
2016-07-03 04:28:02.870365 7fe58d5bf700 20 journal write_thread_entry woke up
2016-07-03 04:28:02.870369 7fe58d5bf700 10 journal room 10731565055 max_size 10736369664 pos 5793300480 header.start 5788499968 top 4096
2016-07-03 04:28:02.870373 7fe58d5bf700 10 journal check_for_full at 5793300480 : 8192 < 10731565055
2016-07-03 04:28:02.870374 7fe58d5bf700 15 journal prepare_single_write 1 will write 5793300480 : seq 92941 len 6440 -> 8192 (head 40 pre_pad 0 ebl 6440 post_pad 1672 tail 40) (ebl alignment -1)
2016-07-03 04:28:02.870393 7fe58d5bf700 20 journal prepare_multi_write queue_pos now 5793308672
2016-07-03 04:28:02.870394 7fe58d5bf700 15 journal do_aio_write writing 5793300480~8192
2016-07-03 04:28:02.870406 7fe58d5bf700 10 journal align_bl total memcopy: 8192
2016-07-03 04:28:02.870407 7fe58d5bf700 20 journal write_aio_bl 5793300480~8192 seq 92941
2016-07-03 04:28:02.870413 7fe58d5bf700 20 journal write_aio_bl .. 5793300480~8192 in 1
...
2016-07-03 04:29:19.644934 7fe58ddc0700 10 journal commit_start nothing to do
2016-07-03 04:29:19.644936 7fe58ddc0700 10 journal commit_start
2016-07-03 04:29:24.645001 7fe58ddc0700 10 journal commit_start max_applied_seq 92940, open_ops 0
2016-07-03 04:29:24.645013 7fe58ddc0700 10 journal commit_start blocked, all open_ops have completed
2016-07-03 04:29:24.645015 7fe58ddc0700 10 journal commit_start nothing to do
2016-07-03 04:29:24.645017 7fe58ddc0700 10 journal commit_start
2016-07-03 04:29:29.645082 7fe58ddc0700 10 journal commit_start max_applied_seq 92940, open_ops 0
2016-07-03 04:29:29.645096 7fe58ddc0700 10 journal commit_start blocked, all open_ops have completed
2016-07-03 04:29:29.645098 7fe58ddc0700 10 journal commit_start nothing to do
2016-07-03 04:29:29.645100 7fe58ddc0700 10 journal commit_start
2016-07-03 04:29:34.645165 7fe58ddc0700 10 journal commit_start max_applied_seq 92940, open_ops 0
2016-07-03 04:29:34.645182 7fe58ddc0700 10 journal commit_start blocked, all open_ops have completed
2016-07-03 04:29:34.645183 7fe58ddc0700 10 journal commit_start nothing to do
2016-07-03 04:29:34.645185 7fe58ddc0700 10 journal commit_start
2016-07-03 04:29:37.065045 7fe58cdbe700 10 journal write_finish_thread_entry aio 5793300480~8192 done
2016-07-03 04:29:37.065065 7fe58cdbe700 20 journal check_aio_completion
2016-07-03 04:29:37.065067 7fe58cdbe700 20 journal check_aio_completion completed seq 92941 5793300480~8192 <======= *** end
======================
The seq 92941 take one and a half mimute to write into journal, when we running fio with ioengine=libaio. It must be affected by fio , we tested some times.
======================
On Fri, 1 Jul 2016, Zhangzengran wrote:
> Hi, all:
> Recently we found that, when running fio with ioengine=libaio on one cluster node , some ops will become slow quest if the iodepth is large enough.
> We track the log ,found that the slow quest blocked too long when the journal entry writen with io_submit.
> Is it a problem of libaio parallel? Thank you!
How deep is the queue depth? And where are you seeing the slow requests?
At some point you will just saturate the OSDs and latency will go up, but it should take a lot of requests for that to happen.
sage
-------------------------------------------------------------------------------------------------------------------------------------
本邮件及其附件含有杭州华三通信技术有限公司的保密信息，仅限于发送给上面地址中列出
的个人或群组。禁止任何其他人以任何形式使用（包括但不限于全部或部分地泄露、复制、
或散发）本邮件中的信息。如果您错收了本邮件，请您立即电话或邮件通知发件人并删除本
邮件！
This e-mail and its attachments contain confidential information from H3C, which is
intended only for the person or entity whose address is listed above. Any use of the
information contained herein in any way (including, but not limited to, total or partial
disclosure, reproduction, or dissemination) by persons other than the intended
recipient(s) is prohibited. If you receive this e-mail in error, please notify the sender
by phone or email immediately and delete it!
��.n��������+%������w��{.n����z��u���ܨ}���Ơz�j:+v�����w����ޙ��&�)ߡ�a����z�ޗ���ݢj��w�f
