On 07/28/2015 04:21 PM, Mark Nelson wrote:
On 07/17/2015 07:50 AM, Gregory Farnum wrote:
On Fri, Jul 17, 2015 at 1:13 PM, Kenneth Waegeman
<kenneth.waegeman@xxxxxxxx> wrote:
Hi all,
I've read in the documentation that OSDs use around 512MB on a healthy
cluster.(http://ceph.com/docs/master/start/hardware-recommendations/#ram)
Now, our OSD's are all using around 2GB of RAM memory while the
cluster is
healthy.
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+
COMMAND
29784 root 20 0 6081276 2.535g 4740 S 0.7 8.1 1346:55
ceph-osd
32818 root 20 0 5417212 2.164g 24780 S 16.2 6.9 1238:55
ceph-osd
25053 root 20 0 5386604 2.159g 27864 S 0.7 6.9 1192:08
ceph-osd
33875 root 20 0 5345288 2.092g 3544 S 0.7 6.7 1188:53
ceph-osd
30779 root 20 0 5474832 2.090g 28892 S 1.0 6.7 1142:29
ceph-osd
22068 root 20 0 5191516 2.000g 28664 S 0.7 6.4 31:56.72
ceph-osd
34932 root 20 0 5242656 1.994g 4536 S 0.3 6.4 1144:48
ceph-osd
26883 root 20 0 5178164 1.938g 6164 S 0.3 6.2 1173:01
ceph-osd
31796 root 20 0 5193308 1.916g 27000 S 16.2 6.1 923:14.87
ceph-osd
25958 root 20 0 5193436 1.901g 2900 S 0.7 6.1 1039:53
ceph-osd
27826 root 20 0 5225764 1.845g 5576 S 1.0 5.9 1031:15
ceph-osd
36011 root 20 0 5111660 1.823g 20512 S 15.9 5.8 1093:01
ceph-osd
19736 root 20 0 2134680 0.994g 0 S 0.3 3.2 46:13.47
ceph-osd
[root@osd003 ~]# ceph status
2015-07-17 14:03:13.865063 7f1fde5f0700 -1 WARNING: the following
dangerous
and experimental features are enabled: keyvaluestore
2015-07-17 14:03:13.887087 7f1fde5f0700 -1 WARNING: the following
dangerous
and experimental features are enabled: keyvaluestore
cluster 92bfcf0a-1d39-43b3-b60f-44f01b630e47
health HEALTH_OK
monmap e1: 3 mons at
{mds01=10.141.16.1:6789/0,mds02=10.141.16.2:6789/0,mds03=10.141.16.3:6789/0}
election epoch 58, quorum 0,1,2 mds01,mds02,mds03
mdsmap e17218: 1/1/1 up {0=mds03=up:active}, 1 up:standby
osdmap e25542: 258 osds: 258 up, 258 in
pgmap v2460163: 4160 pgs, 4 pools, 228 TB data, 154 Mobjects
270 TB used, 549 TB / 819 TB avail
4152 active+clean
8 active+clean+scrubbing+deep
We are using erasure code on most of our OSDs, so maybe that is a
reason.
But also the cache-pool filestore OSDS on 200GB SSDs are using 2GB of
RAM.
Our erasure code pool (16*14 osds) have a pg_num of 2048; our cache pool
(2*14 OSDS) has a pg_num of 1024.
Are these normal values for this configuration, and is the
documentation a
bit outdated, or should we look into something else?
2GB of RSS is larger than I would have expected, but not unreasonable.
In particular I don't think we've gathered numbers on either EC pools
or on the effects of the caching processes.
FWIW, here's statistics for ~36 ceph-osds on the wip-promote-prob branch
after several hours of cache tiering tests (30 OSD base, 6 OS cache
tier) using an EC6+2 pool. At the time of this test, 4K random
read/writes were being performed. The cache tier OSDs specifically use
quite a bit more memory than the base tier. Interestingly in this test
major pagefaults are showing up for the cache tier OSDs which is
annoying. I may need to tweak kernel VM settings on this box.
Ah, we see the same here with our cache OSDS: those small OSDS are
taking the most memory, on some servers they are taking 3G of RAM.
Even If I restart these , they take up the same amount again.
# PROCESS SUMMARY (counters are /sec)
#Time PID User PR PPID THRD S VSZ RSS CP SysT UsrT
Pct AccuTime RKB WKB MajF MinF Command
09:58:48 715 root 20 1 424 S 1G 271M 8 0.19 0.43
6 30:12.64 0 0 0 2502 /usr/local/bin/ceph-osd
09:58:48 1363 root 20 1 424 S 1G 325M 8 0.14 0.33
4 26:50.54 0 0 0 68 /usr/local/bin/ceph-osd
09:58:48 2080 root 20 1 420 S 1G 276M 1 0.21 0.49
7 23:49.36 0 0 0 2848 /usr/local/bin/ceph-osd
09:58:48 2747 root 20 1 424 S 1G 283M 8 0.25 0.68
9 25:16.63 0 0 0 1391 /usr/local/bin/ceph-osd
09:58:48 3451 root 20 1 424 S 1G 331M 6 0.13 0.14
2 27:36.71 0 0 0 148 /usr/local/bin/ceph-osd
09:58:48 4172 root 20 1 424 S 1G 301M 6 0.19 0.43
6 29:44.56 0 0 0 2165 /usr/local/bin/ceph-osd
09:58:48 4935 root 20 1 420 S 1G 310M 9 0.18 0.28
4 29:09.78 0 0 0 2042 /usr/local/bin/ceph-osd
09:58:48 5750 root 20 1 420 S 1G 267M 2 0.11 0.14
2 26:55.31 0 0 0 866 /usr/local/bin/ceph-osd
09:58:48 6544 root 20 1 424 S 1G 299M 7 0.22 0.62
8 26:46.35 0 0 0 3468 /usr/local/bin/ceph-osd
09:58:48 7379 root 20 1 424 S 1G 283M 8 0.16 0.47
6 25:47.86 0 0 0 538 /usr/local/bin/ceph-osd
09:58:48 8183 root 20 1 424 S 1G 269M 4 0.25 0.67
9 35:09.85 0 0 0 2968 /usr/local/bin/ceph-osd
09:58:48 9026 root 20 1 424 S 1G 261M 1 0.19 0.46
6 26:27.36 0 0 0 539 /usr/local/bin/ceph-osd
09:58:48 9859 root 20 1 420 S 1G 289M 2 0.17 0.44
6 31:08.99 0 0 0 2407 /usr/local/bin/ceph-osd
09:58:48 10668 root 20 1 420 S 1G 264M 9 0.13 0.15
2 25:56.95 0 0 0 1312 /usr/local/bin/ceph-osd
09:58:48 11461 root 20 1 424 S 1G 240M 3 0.12 0.13
2 23:25.74 0 0 0 250 /usr/local/bin/ceph-osd
09:58:48 12262 root 20 1 420 S 1G 302M 4 0.24 0.62
8 26:17.65 0 0 0 2608 /usr/local/bin/ceph-osd
09:58:48 13197 root 20 1 424 S 1G 234M 2 0.15 0.41
5 23:15.40 0 0 0 371 /usr/local/bin/ceph-osd
09:58:48 14092 root 20 1 428 S 1G 267M 9 0.09 0.22
3 24:05.74 0 0 0 110 /usr/local/bin/ceph-osd
09:58:48 14907 root 20 1 436 S 2G 905M 1 6.46 22.21
286 02:05:09 0 0 16 24K /usr/local/bin/ceph-osd
09:58:48 15760 root 20 1 432 S 2G 894M 7 5.09 16.40
214 01:58:33 0 0 15 8730 /usr/local/bin/ceph-osd
09:58:48 16653 root 20 1 432 S 2G 931M 3 6.42 19.04
254 02:04:16 0 0 10 23K /usr/local/bin/ceph-osd
09:58:48 17541 root 20 1 432 S 2G 896M 4 5.41 15.22
206 01:53:47 0 0 9 5913 /usr/local/bin/ceph-osd
09:58:48 18396 root 20 1 432 S 2G 1G 5 6.03 19.26
252 02:06:52 0 0 11 24K /usr/local/bin/ceph-osd
09:58:48 19339 root 20 1 400 S 2G 878M 1 5.92 16.84
227 02:09:23 0 0 17 20K /usr/local/bin/ceph-osd
09:58:48 27136 root 20 1 428 S 1G 306M 4 0.19 0.32
5 26:06.79 0 0 0 2742 /usr/local/bin/ceph-osd
09:58:48 27518 root 20 1 424 S 1G 294M 21 0.22 0.36
5 25:33.46 0 0 0 2678 /usr/local/bin/ceph-osd
09:58:48 27913 root 20 1 424 S 1G 305M 8 0.12 0.16
2 25:48.54 0 0 0 152 /usr/local/bin/ceph-osd
09:58:48 28320 root 20 1 424 S 1G 279M 8 0.17 0.43
6 28:46.26 0 0 0 1696 /usr/local/bin/ceph-osd
09:58:48 28735 root 20 1 424 S 1G 281M 5 0.19 0.43
6 28:51.23 0 0 0 2120 /usr/local/bin/ceph-osd
09:58:48 29162 root 20 1 424 S 1G 297M 18 0.23 0.55
7 26:49.48 0 0 0 3936 /usr/local/bin/ceph-osd
09:58:48 29603 root 20 1 424 S 1G 308M 21 0.13 0.20
3 26:03.96 0 0 0 227 /usr/local/bin/ceph-osd
09:58:48 30269 root 20 1 424 S 1G 311M 3 0.14 0.23
3 26:01.05 0 0 0 2058 /usr/local/bin/ceph-osd
09:58:48 30801 root 20 1 424 S 1G 296M 21 0.16 0.36
5 25:33.12 0 0 0 212 /usr/local/bin/ceph-osd
09:58:48 31369 root 20 1 420 S 1G 306M 11 0.18 0.51
6 31:25.69 0 0 0 3621 /usr/local/bin/ceph-osd
09:58:48 31944 root 20 1 420 S 1G 258M 5 0.15 0.30
4 30:16.04 0 0 0 1686 /usr/local/bin/ceph-osd
09:58:48 32526 root 20 1 420 S 1G 326M 8 0.17 0.39
5 27:29.20 0 0 0 2544 /usr/local/bin/ceph-osd
-Greg
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
