Re: ceph-mon not starting on boot with systemd and Ubuntu 16.04

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Can you check that the dependencies have started? Anything about those in the logs?

network-online.target local-fs.target ceph-create-keys@%i.service

Jan


> On 13 May 2016, at 14:30, Sage Weil <sage@xxxxxxxxxxxx> wrote:
> 
> This is starting to sound like a xenial systemd issue to me.  Maybe poke 
> the canonical folks?
> 
> You might edit the unit file and make it touch something in /tmp instead 
> of starting Ceph just to rule out ceph...
> 
> sage
> 
> 
> On Fri, 13 May 2016, Wido den Hollander wrote:
> 
>> No luck either. After a reboot only the Ceph OSD starts, but the monitor not.
>> 
>> I have checked:
>> - service is enabled
>> - tried to re-enable the service
>> - check the MON logs to see if it was started, it wasn't
>> - systemd log to see if it wants to start the MON, it doesn't
>> 
>> My systemd-foo isn't that good either, so I don't know what is happening here.
>> 
>> Wido
>> 
>>> Op 12 mei 2016 om 15:31 schreef Jan Schermer <jan@xxxxxxxxxxx>:
>>> 
>>> 
>>> Btw try replacing
>>> 
>>> WantedBy=ceph-mon.target
>>> 
>>> With: WantedBy=default.target
>>> then systemctl daemon-reload.
>>> 
>>> See if that does the trick
>>> 
>>> I only messed with systemctl to have my own services start, I still hope it goes away eventually... :P
>>> 
>>> Jan
>>> 
>>>> On 12 May 2016, at 15:01, Wido den Hollander <wido@xxxxxxxx> wrote:
>>>> 
>>>> 
>>>> To also answer Sage's question: No, this is a fresh Jewel install in a few test VMs. This system was not upgraded.
>>>> 
>>>> It was installed 2 hours ago.
>>>> 
>>>>> Op 12 mei 2016 om 14:51 schreef Jan Schermer <jan@xxxxxxxxxxx>:
>>>>> 
>>>>> 
>>>>> Can you post the contents of ceph-mon@.service file?
>>>>> 
>>>> 
>>>> Yes, here you go:
>>>> 
>>>> root@charlie:~# cat /lib/systemd/system/ceph-mon@.service 
>>>> [Unit]
>>>> Description=Ceph cluster monitor daemon
>>>> 
>>>> # According to:
>>>> #   http://www.freedesktop.org/wiki/Software/systemd/NetworkTarget
>>>> # these can be removed once ceph-mon will dynamically change network
>>>> # configuration.
>>>> After=network-online.target local-fs.target ceph-create-keys@%i.service
>>>> Wants=network-online.target local-fs.target ceph-create-keys@%i.service
>>>> 
>>>> PartOf=ceph-mon.target
>>>> 
>>>> [Service]
>>>> LimitNOFILE=1048576
>>>> LimitNPROC=1048576
>>>> EnvironmentFile=-/etc/default/ceph
>>>> Environment=CLUSTER=ceph
>>>> ExecStart=/usr/bin/ceph-mon -f --cluster ${CLUSTER} --id %i --setuser ceph --setgroup ceph
>>>> ExecReload=/bin/kill -HUP $MAINPID
>>>> PrivateDevices=yes
>>>> ProtectHome=true
>>>> ProtectSystem=full
>>>> PrivateTmp=true
>>>> TasksMax=infinity
>>>> Restart=on-failure
>>>> StartLimitInterval=30min
>>>> StartLimitBurst=3
>>>> 
>>>> [Install]
>>>> WantedBy=ceph-mon.target
>>>> root@charlie:~#
>>>> 
>>>>> what does
>>>>> systemctl is-enabled ceph-mon@charlie
>>>>> say?
>>>>> 
>>>> 
>>>> root@charlie:~# systemctl is-enabled ceph-mon@charlie
>>>> enabled
>>>> root@charlie:~#
>>>> 
>>>>> However, this looks like it was just started at a bad moment and died - nothing in logs?
>>>>> 
>>>> 
>>>> No, I checked the ceph-mon logs in /var/log/ceph. No sign of it even trying to start after boot. In /var/log/syslog there also is not a trace of ceph-mon.
>>>> 
>>>> Only the OSD starts.
>>>> 
>>>> Wido
>>>> 
>>>>> Jan
>>>>> 
>>>>> 
>>>>>> On 12 May 2016, at 14:44, Sage Weil <sage@xxxxxxxxxxxx> wrote:
>>>>>> 
>>>>>> On Thu, 12 May 2016, Wido den Hollander wrote:
>>>>>>> Hi,
>>>>>>> 
>>>>>>> I am setting up a Jewel cluster in VMs with Ubuntu 16.04.
>>>>>>> 
>>>>>>> ceph version 10.2.0 (3a9fba20ec743699b69bd0181dd6c54dc01c64b9)
>>>>>>> 
>>>>>>> After a reboot the Ceph Monitors don't start and I have to do so manually.
>>>>>>> 
>>>>>>> Three machines, alpha, bravo and charlie all have the same problem.
>>>>>>> 
>>>>>>> root@charlie:~# systemctl status ceph-mon@charlie
>>>>>>> ● ceph-mon@charlie.service - Ceph cluster monitor daemon
>>>>>>> Loaded: loaded (/lib/systemd/system/ceph-mon@.service; enabled; vendor preset: enabled)
>>>>>>> Active: inactive (dead)
>>>>>>> root@charlie:~#
>>>>>>> 
>>>>>>> I can start it and it works
>>>>>> 
>>>>>> Hmm.. my systemd-fu is weak, but if it's enabled it seems like it shoud 
>>>>>> come up.
>>>>>> 
>>>>>> Was this an upgraded package?  What if you do 'systemctl reenable 
>>>>>> ceph-mon@charlie'?
>>>>>> 
>>>>>> sage
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>>> 
>>>>>>> root@charlie:~# systemctl start ceph-mon@charlie
>>>>>>> root@charlie:~# systemctl status ceph-mon@charlie
>>>>>>> ● ceph-mon@charlie.service - Ceph cluster monitor daemon
>>>>>>> Loaded: loaded (/lib/systemd/system/ceph-mon@.service; enabled; vendor preset: enabled)
>>>>>>> Active: active (running) since Thu 2016-05-12 16:08:56 CEST; 1s ago
>>>>>>> Main PID: 1368 (ceph-mon)
>>>>>>> 
>>>>>>> I tried removing the /var/log/ceph/ceph-mon.charlie.log file and reboot to see if the mon was actually invoked, but it wasn't.
>>>>>>> 
>>>>>>> ceph.target has been started and so is the OSD on the machine. It is just the monitor which hasn't been started.
>>>>>>> 
>>>>>>> In the syslog I see:
>>>>>>> 
>>>>>>> May 12 16:11:19 charlie systemd[1]: Starting Ceph object storage daemon...
>>>>>>> May 12 16:11:19 charlie systemd[1]: Starting LSB: Start Ceph distributed file system daemons at boot time...
>>>>>>> May 12 16:11:19 charlie systemd[1]: Started LSB: Start Ceph distributed file system daemons at boot time.
>>>>>>> May 12 16:11:20 charlie systemd[1]: Started Ceph object storage daemon.
>>>>>>> May 12 16:11:20 charlie systemd[1]: Started Ceph disk activation: /dev/sdb2.
>>>>>>> May 12 16:11:21 charlie systemd[1]: Started Ceph object storage daemon.
>>>>>>> May 12 16:11:21 charlie systemd[1]: Started Ceph disk activation: /dev/sdb1.
>>>>>>> 
>>>>>>> Am I missing something or is this a bug?
>>>>>>> 
>>>>>>> Wido
>>>>>>> _______________________________________________
>>>>>>> ceph-users mailing list
>>>>>>> ceph-users@xxxxxxxxxxxxxx
>>>>>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>>>>> _______________________________________________
>>>>>> ceph-users mailing list
>>>>>> ceph-users@xxxxxxxxxxxxxx
>>>>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>>>> 
>>> 
>> 

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com




[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux