Any suggestion to deal with slow request?

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hi Cephers,

We have a Ceph cluster running 0.80.9, which consists of 36 OSDs with 3 replicas. Recently, some OSDs keep reporting slow request and the cluster has a performance downgrade.

From the log of one OSD, I observe that all the slow requests are resulted from waiting for the replicas to complete. And the replication OSDs are not always some specific ones but could be any other two OSDs.
2016-01-06 08:17:11.887016 7f175ef25700  0 log [WRN] : slow request 1.162776 seconds old, received at 2016-01-06 08:17:11.887092: osd_op(client.13302933.0:839452 rbd_data.c2659c728b0ddb.0000000000000024 [stat,set-alloc-hint object_size 16777216 write_size 16777216,write 12099584~8192] 3.abd08522 ack+ondisk+write e4661) v4 currently waiting for subops from 24,31
I dumped out the historic Ops of the OSD and noticed the following information:
1) wait about 8 seconds for the replies from the replica OSDs.
                    { "time": "2016-01-06 08:17:03.879264",
                      "event": "op_applied"},
                    { "time": "2016-01-06 08:17:11.684598",
                      "event": "sub_op_applied_rec"},
                    { "time": "2016-01-06 08:17:11.687016",
                      "event": "sub_op_commit_rec"},

2) spend more than 3 seconds in writeq and 2 seconds to write the journal.
                  { "time": "2016-01-06 08:19:16.887519",
                      "event": "commit_queued_for_journal_write"},
                    { "time": "2016-01-06 08:19:20.109339",
                      "event": "write_thread_in_journal_buffer"},
                    { "time": "2016-01-06 08:19:22.177952",
                      "event": "journaled_completion_queued"},

Any ideas or suggestions?

BTW, I checked the underlying network with iperf, it works fine.

Thanks,
Jevon
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux