Re: 160 Thousand ceph-client.admin.*.asok files : Wired problem , never seen before

[Robert LeBlanc <robert@xxxxxxxxxxxxx>]

Probably the same trigger. Openstack does some check on the Ceph cluster and the process doesn't let go of the socket, or something like that, I don't know all the details. I only know that there is already a fix, but I don't have the commit for that fix. You can try and look it up to backport it or just disable the admin socket until the next version comes out.

Since we only use the admin socket for troubleshooting we decided to disable it until the fixed version is out. We were running into problems where openstack would not start new VMS because the nova process ran out of file descriptors. Restarting the nova process would let it work until the file descriptors were exhausted again.

Robert LeBlanc

Sent from a mobile device please excuse any typos.

On Aug 10, 2015 1:13 AM, "Vickey Singh" <vickey.singh22693@xxxxxxxxx> wrote:

Hi Thank you for your reply

No its these nodes are just Ceph nodes , nothing shared with OpenSack. Infact i am not using openstack with this ceph cluster.

However i have configured Calamari on this cluster , but not sure if that has caused the problem. I also tried to remove calamari configurations and packages from entire cluster but still files are getting generated.

See the below trend, files are getting generated every 1 Minute 

srwxr-xr-x.  1 root root    0 Aug 10 09:56 ceph-client.admin.3104224.53185360.asok

srwxr-xr-x.  1 root root    0 Aug 10 09:57 ceph-client.admin.3104606.45857616.asok

srwxr-xr-x.  1 root root    0 Aug 10 09:58 ceph-client.admin.3104987.45837136.asok

srwxr-xr-x.  1 root root    0 Aug 10 09:59 ceph-client.admin.3105378.63122256.asok

srwxr-xr-x.  1 root root    0 Aug 10 10:00 ceph-client.admin.3105763.62139216.asok

srwxr-xr-x.  1 root root    0 Aug 10 10:01 ceph-client.admin.3106175.47811408.asok

srwxr-xr-x.  1 root root    0 Aug 10 10:02 ceph-client.admin.3106553.45525840.asok

srwxr-xr-x.  1 root root    0 Aug 10 10:03 ceph-client.admin.3106937.57162576.asok

srwxr-xr-x.  1 root root    0 Aug 10 10:04 ceph-client.admin.3107316.68578128.asok

srwxr-xr-x.  1 root root    0 Aug 10 10:05 ceph-client.admin.3107701.54725456.asok

srwxr-xr-x.  1 root root    0 Aug 10 10:06 ceph-client.admin.3108044.65157968.asok

srwxr-xr-x.  1 root root    0 Aug 10 10:07 ceph-client.admin.3108513.50350928.asok

srwxr-xr-x.  1 root root    0 Aug 10 10:08 ceph-client.admin.3108900.59702096.asok

srwxr-xr-x.  1 root root    0 Aug 10 10:09 ceph-client.admin.3109276.57756496.asok

drwxr-xr-x.  2 root root  13M Aug 10 10:09 .

[root@node1 ceph]# date

Mon Aug 10 10:09:57 EEST 2015

[root@node1 ceph]# 

Any help to fix this is appreciated. 

Regards

Vicky

On Mon, Aug 10, 2015 at 2:32 AM, Nigel Williams <nigel.williams@xxxxxxxxxxx> wrote:

On 10/08/2015 12:02 AM, Robert LeBlanc wrote:
> I'm guessing this is on an OpenStack node? There is a fix for this and I
> think it will come out in the next release. For now we have had to disable
> the admin sockets.

Do you know what triggers the fault? we've not seen it on Firefly+RBD for Openstack.

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com