On Sat, Feb 20, 2016 at 2:26 AM, <aaaler@xxxxxxxxx> wrote:
Hi All.
We're running 180-node cluster in docker containers -- official ceph:hammer.
Recently, we've found a rarely reproducible problem on it: sometimes
data transfer freezes for significant time (5-15 minutes). The issue
is taking place while using radosgw & librados apps
(docker-distribution). This problem can be worked around with "ms tcp
read timeout" parameter decreased to 2-3 seconds on the client side,
but that does not seem to be a good solution.
I've written bash script, getting every object (and it's omap/xattr)
with 'rados' cli utility from data pool in infinite cycle, to
reproduce the problem. Running that on 3 hosts simultaneously on
docker-distribution's pool (4mb objects) during 8 hours resulted in 25
reads, each of them took more than 60 seconds.
Script results here (hostnames substituted):
https://gist.github.com/aaaler/cb190c1eb636564519a5#file-distribution-pool-err-sorted
But there's nothing suspicious on corresponding OSD logs.
For example, take a look on the one of these faulty reads:
21:44:32 consumed 1891 seconds reading
blob:daa46e8d-170e-43ab-8c00-526782f95e02-0 on host1(192.168.1.133)
osdmap e80485 pool 'distribution' (17) object
'blob:daa46e8d-170e-43ab-8c00-526782f95e02-0' -> pg 17.97f485f (17.5f)
-> up ([139,149,167], p139) acting ([139,149,167], p139)
Thus, we've got 1891 seconds of waiting, and after that the client has
just proceed without any errors occurred, so I tried to find something
useful in osd.139 logs
(https://gist.github.com/aaaler/cb190c1eb636564519a5#file-osd-139-log),
but could not find anything interesting.
Another example (next line in script output) shew us 2983 seconds of
reading blob:f5c22093-6e6d-41a6-be36-462330b36c67-71 from osd.56.
Again, nothing in osd.56 logs during that time:
https://gist.github.com/aaaler/cb190c1eb636564519a5#file-osd-56-log
How can I troubleshoot this? As too excessive logging on 180-node
cluster will make bunch of traffic and bring problems with finding the
right host to check log :(
So you don't find any slow request with ceph -s or at /var/log/ceph/ceph.log in monitor side?
You mentioned "ms tcp read timeout" has some effects on your case, I guess we may has some problems on using poll call.
The only potential related PR is https://github.com/ceph/ceph/pull/6971
Few words about underlying configuration:
- ceph:hammer containers in docker 1.9.1 (--net=host)
- gentoo with 3.14.18/3.18.10 kernel.
- 1gbps LAN
- osd using directory in /var
- hosts share osd workload with some other php-fpm's
The configuration is pretty default, except some osd parameters
configured to reduce scrubbing workload:
[osd]
osd disk thread ioprio class = idle
osd disk thread ioprio priority = 5
osd recovery max active = 1
osd max backfills = 2
--
Sincerely, Alexey Griazin
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Best Regards,
Wheat
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com