Re: requests are blocked

[Dan Nica <dan.nica@xxxxxxxxxxxxxxxxxxxx>]

That is strange, maybe there is a sysctl option to tweak on OSDs ? this will be nasty if it goes into our production!

 

--

Dan

 

From: Wade Holler [mailto:wade.holler@xxxxxxxxx]
Sent: Tuesday, December 22, 2015 4:36 PM
To: Dan Nica <dan.nica@xxxxxxxxxxxxxxxxxxxx>; ceph-users@xxxxxxxxxxxxxx
Subject: Re: [ceph-users] requests are blocked

 

I had major host stability problems under load with -327  . Repeatable test cases under high load with XFS or BTRFS would result in hung kernel tasks and of course the sympathetic behavior you mention. 

requests are blocked mean that the op tracker in ceph hasn't received a timely response from the osd usually.  I'm sure someone more seasoned can provide a better explanation.

-Wade

 

On Tue, Dec 22, 2015 at 9:24 AM Dan Nica <dan.nica@xxxxxxxxxxxxxxxxxxxx> wrote:

Hi

 

I try to run a bench test on a RBD image and I get from time to time the following in ceph status

 

    cluster 046b0180-dc3f-4846-924f-41d9729d48c8

     health HEALTH_WARN

            2 requests are blocked > 32 sec

     monmap e1: 3 mons at {alder=10.6.250.249:6789/0,ash=10.6.250.248:6789/0,aspen=10.6.250.247:6789/0}

            election epoch 18, quorum 0,1,2 aspen,ash,alder

     osdmap e114: 6 osds: 6 up, 6 in

            flags sortbitwise

      pgmap v3816: 192 pgs, 1 pools, 23062 MB data, 5814 objects

            46406 MB used, 44624 GB / 44670 GB avail

                 192 active+clean

  client io 6083 B/s rd, 18884 kB/s wr, 75 op/s

 

 

what does  “requests are blocked” mean ? and performance drops to almost  0 ?

I am running infernalis version on Centos 7 kernel 3.10.0-327.3.1.el7.x86_64

 

Thanks

--

Dan

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com