Re: ceph-mon not starting on boot with systemd and Ubuntu 16.04

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



> Op 13 mei 2016 om 14:56 schreef Jan Schermer <jan@xxxxxxxxxxx>:
> 
> 
> Can you check that the dependencies have started? Anything about those in the logs?
> 
> network-online.target local-fs.target ceph-create-keys@%i.service
> 

May 13 16:59:15 alpha systemd[1]: Reached target Local File Systems (Pre).
May 13 16:59:15 alpha systemd[1]: Reached target Local File Systems.
..
..
May 13 16:59:18 alpha systemd[1]: Reached target Network.
May 13 16:59:18 alpha systemd[1]: Reached target Network is Online.

In the systemd logs there is no trace of 'ceph-create-keys@%i.service' starting up, so that seems to be the cullprit.

Wido

> Jan
> 
> 
> > On 13 May 2016, at 14:30, Sage Weil <sage@xxxxxxxxxxxx> wrote:
> > 
> > This is starting to sound like a xenial systemd issue to me.  Maybe poke 
> > the canonical folks?
> > 
> > You might edit the unit file and make it touch something in /tmp instead 
> > of starting Ceph just to rule out ceph...
> > 
> > sage
> > 
> > 
> > On Fri, 13 May 2016, Wido den Hollander wrote:
> > 
> >> No luck either. After a reboot only the Ceph OSD starts, but the monitor not.
> >> 
> >> I have checked:
> >> - service is enabled
> >> - tried to re-enable the service
> >> - check the MON logs to see if it was started, it wasn't
> >> - systemd log to see if it wants to start the MON, it doesn't
> >> 
> >> My systemd-foo isn't that good either, so I don't know what is happening here.
> >> 
> >> Wido
> >> 
> >>> Op 12 mei 2016 om 15:31 schreef Jan Schermer <jan@xxxxxxxxxxx>:
> >>> 
> >>> 
> >>> Btw try replacing
> >>> 
> >>> WantedBy=ceph-mon.target
> >>> 
> >>> With: WantedBy=default.target
> >>> then systemctl daemon-reload.
> >>> 
> >>> See if that does the trick
> >>> 
> >>> I only messed with systemctl to have my own services start, I still hope it goes away eventually... :P
> >>> 
> >>> Jan
> >>> 
> >>>> On 12 May 2016, at 15:01, Wido den Hollander <wido@xxxxxxxx> wrote:
> >>>> 
> >>>> 
> >>>> To also answer Sage's question: No, this is a fresh Jewel install in a few test VMs. This system was not upgraded.
> >>>> 
> >>>> It was installed 2 hours ago.
> >>>> 
> >>>>> Op 12 mei 2016 om 14:51 schreef Jan Schermer <jan@xxxxxxxxxxx>:
> >>>>> 
> >>>>> 
> >>>>> Can you post the contents of ceph-mon@.service file?
> >>>>> 
> >>>> 
> >>>> Yes, here you go:
> >>>> 
> >>>> root@charlie:~# cat /lib/systemd/system/ceph-mon@.service 
> >>>> [Unit]
> >>>> Description=Ceph cluster monitor daemon
> >>>> 
> >>>> # According to:
> >>>> #   http://www.freedesktop.org/wiki/Software/systemd/NetworkTarget
> >>>> # these can be removed once ceph-mon will dynamically change network
> >>>> # configuration.
> >>>> After=network-online.target local-fs.target ceph-create-keys@%i.service
> >>>> Wants=network-online.target local-fs.target ceph-create-keys@%i.service
> >>>> 
> >>>> PartOf=ceph-mon.target
> >>>> 
> >>>> [Service]
> >>>> LimitNOFILE=1048576
> >>>> LimitNPROC=1048576
> >>>> EnvironmentFile=-/etc/default/ceph
> >>>> Environment=CLUSTER=ceph
> >>>> ExecStart=/usr/bin/ceph-mon -f --cluster ${CLUSTER} --id %i --setuser ceph --setgroup ceph
> >>>> ExecReload=/bin/kill -HUP $MAINPID
> >>>> PrivateDevices=yes
> >>>> ProtectHome=true
> >>>> ProtectSystem=full
> >>>> PrivateTmp=true
> >>>> TasksMax=infinity
> >>>> Restart=on-failure
> >>>> StartLimitInterval=30min
> >>>> StartLimitBurst=3
> >>>> 
> >>>> [Install]
> >>>> WantedBy=ceph-mon.target
> >>>> root@charlie:~#
> >>>> 
> >>>>> what does
> >>>>> systemctl is-enabled ceph-mon@charlie
> >>>>> say?
> >>>>> 
> >>>> 
> >>>> root@charlie:~# systemctl is-enabled ceph-mon@charlie
> >>>> enabled
> >>>> root@charlie:~#
> >>>> 
> >>>>> However, this looks like it was just started at a bad moment and died - nothing in logs?
> >>>>> 
> >>>> 
> >>>> No, I checked the ceph-mon logs in /var/log/ceph. No sign of it even trying to start after boot. In /var/log/syslog there also is not a trace of ceph-mon.
> >>>> 
> >>>> Only the OSD starts.
> >>>> 
> >>>> Wido
> >>>> 
> >>>>> Jan
> >>>>> 
> >>>>> 
> >>>>>> On 12 May 2016, at 14:44, Sage Weil <sage@xxxxxxxxxxxx> wrote:
> >>>>>> 
> >>>>>> On Thu, 12 May 2016, Wido den Hollander wrote:
> >>>>>>> Hi,
> >>>>>>> 
> >>>>>>> I am setting up a Jewel cluster in VMs with Ubuntu 16.04.
> >>>>>>> 
> >>>>>>> ceph version 10.2.0 (3a9fba20ec743699b69bd0181dd6c54dc01c64b9)
> >>>>>>> 
> >>>>>>> After a reboot the Ceph Monitors don't start and I have to do so manually.
> >>>>>>> 
> >>>>>>> Three machines, alpha, bravo and charlie all have the same problem.
> >>>>>>> 
> >>>>>>> root@charlie:~# systemctl status ceph-mon@charlie
> >>>>>>> ● ceph-mon@charlie.service - Ceph cluster monitor daemon
> >>>>>>> Loaded: loaded (/lib/systemd/system/ceph-mon@.service; enabled; vendor preset: enabled)
> >>>>>>> Active: inactive (dead)
> >>>>>>> root@charlie:~#
> >>>>>>> 
> >>>>>>> I can start it and it works
> >>>>>> 
> >>>>>> Hmm.. my systemd-fu is weak, but if it's enabled it seems like it shoud 
> >>>>>> come up.
> >>>>>> 
> >>>>>> Was this an upgraded package?  What if you do 'systemctl reenable 
> >>>>>> ceph-mon@charlie'?
> >>>>>> 
> >>>>>> sage
> >>>>>> 
> >>>>>> 
> >>>>>> 
> >>>>>>> 
> >>>>>>> root@charlie:~# systemctl start ceph-mon@charlie
> >>>>>>> root@charlie:~# systemctl status ceph-mon@charlie
> >>>>>>> ● ceph-mon@charlie.service - Ceph cluster monitor daemon
> >>>>>>> Loaded: loaded (/lib/systemd/system/ceph-mon@.service; enabled; vendor preset: enabled)
> >>>>>>> Active: active (running) since Thu 2016-05-12 16:08:56 CEST; 1s ago
> >>>>>>> Main PID: 1368 (ceph-mon)
> >>>>>>> 
> >>>>>>> I tried removing the /var/log/ceph/ceph-mon.charlie.log file and reboot to see if the mon was actually invoked, but it wasn't.
> >>>>>>> 
> >>>>>>> ceph.target has been started and so is the OSD on the machine. It is just the monitor which hasn't been started.
> >>>>>>> 
> >>>>>>> In the syslog I see:
> >>>>>>> 
> >>>>>>> May 12 16:11:19 charlie systemd[1]: Starting Ceph object storage daemon...
> >>>>>>> May 12 16:11:19 charlie systemd[1]: Starting LSB: Start Ceph distributed file system daemons at boot time...
> >>>>>>> May 12 16:11:19 charlie systemd[1]: Started LSB: Start Ceph distributed file system daemons at boot time.
> >>>>>>> May 12 16:11:20 charlie systemd[1]: Started Ceph object storage daemon.
> >>>>>>> May 12 16:11:20 charlie systemd[1]: Started Ceph disk activation: /dev/sdb2.
> >>>>>>> May 12 16:11:21 charlie systemd[1]: Started Ceph object storage daemon.
> >>>>>>> May 12 16:11:21 charlie systemd[1]: Started Ceph disk activation: /dev/sdb1.
> >>>>>>> 
> >>>>>>> Am I missing something or is this a bug?
> >>>>>>> 
> >>>>>>> Wido
> >>>>>>> _______________________________________________
> >>>>>>> ceph-users mailing list
> >>>>>>> ceph-users@xxxxxxxxxxxxxx
> >>>>>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >>>>>> _______________________________________________
> >>>>>> ceph-users mailing list
> >>>>>> ceph-users@xxxxxxxxxxxxxx
> >>>>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >>>>> 
> >>> 
> >> 
>
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com




[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux