| Hi all we have a ceph 0.67.4 cluster with 24 OSDs I have noticed that the two servers that have 9 OSD each, have around 10'000 threads running - a number that went up significantly 2 weeks ago. Looking at the threads: root@h2:/var/log/ceph# ps -efL | grep ceph-osd | awk '{ print $2 }' | uniq -c | sort -n 1 17583 856 3151 874 11946 1034 3173 1038 3072 1040 3175 1052 3068 1062 3149 1068 3060 1070 3190 root@h2:/var/log/ceph# ps axwu | grep ceph-osd root 3060 1.0 0.2 2224068 312456 ? Ssl Nov01 392:17 /usr/bin/ceph-osd --cluster=ceph -i 5 -f root 3068 1.2 0.2 2140988 356208 ? Ssl Nov01 441:22 /usr/bin/ceph-osd --cluster=ceph -i 9 -f root 3072 1.2 0.2 2049608 370236 ? Ssl Nov01 443:18 /usr/bin/ceph-osd --cluster=ceph -i 4 -f root 3149 1.2 0.3 2122548 402236 ? Ssl Nov01 440:16 /usr/bin/ceph-osd --cluster=ceph -i 8 -f root 3151 1.2 0.3 1917856 426224 ? Ssl Nov01 453:30 /usr/bin/ceph-osd --cluster=ceph -i 7 -f root 3173 0.8 0.2 1978252 264732 ? Ssl Nov01 325:01 /usr/bin/ceph-osd --cluster=ceph -i 11 -f root 3175 1.1 0.3 2186676 422112 ? Ssl Nov01 401:27 /usr/bin/ceph-osd --cluster=ceph -i 12 -f root 3190 1.1 0.3 2140480 412844 ? Ssl Nov01 421:31 /usr/bin/ceph-osd --cluster=ceph -i 6 -f root 11946 1.7 0.3 2060968 445368 ? Ssl Nov14 302:36 /usr/bin/ceph-osd --cluster=ceph -i 10 -f root 17589 0.0 0.0 9456 952 pts/25 S+ 16:13 0:00 grep --color=auto ceph-osd we see each osd process with around 1000 threads. Is this normal and expected? One theory we have, is that this has to do with the number of placement groups - I had increased the number of PGs in one of the pools: root@h2:/var/log/ceph# ceph osd pool get images pg_num pg_num: 1000 root@h2:/var/log/ceph# ceph osd pool get volumes pg_num pg_num: 128 That could possibly have been on the day, the number of treads started to rise. Feedback appreciated! thanks Jens-Christian -- SWITCH Jens-Christian Fischer, Peta Solutions Werdstrasse 2, P.O. Box, 8021 Zurich, Switzerland phone +41 44 268 15 15, direct +41 44 268 15 71 jens-christian.fischer@xxxxxxxxx |
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
