Re: Math behind : : OSD count vs OSD process vs OSD ports

[Дмитрий Глушенок <glush@xxxxxxxxxx>]

Hi Vickey,

18 нояб. 2015 г., в 11:36, Vickey Singh <vickey.singh22693@xxxxxxxxx> написал(а):

Can anyone please help me understand this.

Thank You

On Mon, Nov 16, 2015 at 5:55 PM, Vickey Singh <vickey.singh22693@xxxxxxxxx> wrote:

Hello Community

Need your help in understanding this.

I have the below node, which is hosting 60 physical disks, running 1 OSD per disk so total 60 Ceph OSD daemons

[root@node01 ~]# service ceph status | grep -i osd | grep -i running | wc -l

60

[root@node01 ~]#

However if i check OSD processes it shows that there are 120 OSD process are running.

[root@node01 ~]# ps -ef | grep -i osd | grep -v grep | wc -l

120

[root@node01 ~]#

Question 1 : why its 120 processes ? , i it should be 60 (because of 60 OSD on the system)

My guess : is this because of multithreading ?

No, it is the way OSD processes were launched. Try following (quotes are necessary):

$ bash -c "sleep 1; sleep 999" &

$ ps -ef | grep sleep | grep -v grep

You will see that bash process will wait for "sleep 999" process to finish. OSD processes are launched similarly.

Now if i check the number of ports used by OSD its comming out to be 240

[root@node01 ~]# netstat -plunt | grep -i osd | wc -l

240

[root@node01 ~]#

Question 2 : Now why its 240 ports ? It should be 60 ( because of 60 OSD on the system)

It is because each OSD uses four ports: http://docs.ceph.com/docs/master/rados/configuration/network-config-ref/#osd-ip-tables

If i grep a specific OSD port , its shows 2 ports are occupied by OSD process 260519

[root@node01 ~]# netstat -plunt | grep -i osd | grep -i 6819

tcp        0      0 10.101.50.1:6819        0.0.0.0:*               LISTEN      260519/ceph-osd

tcp        0      0 10.102.50.1:6819        0.0.0.0:*               LISTEN      260519/ceph-osd

[root@node01 ~]#

Question 3 : Now based on the scenario 2 it should be 4 ( so 60 OSD x 4 ports = 240 ports in total)

I have two public and cluster network configured in ceph.conf , is all these because of two different networks ?

If you will grep 260519 (PID) instead of 6819 (port) you should see four ports listened (two for private network and two for public).

I would really appreciate if some knowledgeable person share his understanding with me.

Thank you in advance.

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

--

Dmitry Glushenok

Jet Infosystems

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com