Cannot shutdown monitors

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hi

I am running a small cluster of 8 machines (80 osds), with three monitors on Ubuntu 16.04. Ceph version 10.2.5.

I cannot reboot the monitors without physically going into the datacenter and power cycling them. What happens is that while shutting down, ceph gets stuck trying to contact the other monitors but networking has already shut down or something like that. I get an endless stream of:

libceph: connect 10.20.0.10:6789 error -101
libceph: connect 10.20.0.13:6789 error -101
libceph: connect 10.20.0.17:6789 error -101

where in this case 10.20.0.10 is the machine I am trying to shut down and all three IPs are the MONs.

At this stage of the shutdown, the machine doesn't respond to pings, and I cannot even log in on any of the virtual terminals. Nothing to do but poweroff at the server.

The other non-mon servers shut down just fine, and the cluster was healthy at the time I was rebooting the mon (I only reboot one machine at a time, waiting for it to come up before I do the next one).

Also worth mentioning that if I execute 

sudo systemctl stop ceph\*.service ceph\*.target

on the server, the only things I see are:

root     11143     2  0 18:40 ?        00:00:00 [ceph-msgr]
root     11162     2  0 18:40 ?        00:00:00 [ceph-watch-noti]

and even then, when no ceph daemons are left running, doing a reboot goes into the same loop.

I can't really find any mention of this online, but I feel someone must have hit this. Any idea how to fix it? It's really annoying because its hard for me to get access to the datacenter.

Thanks
Michael
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux