----- Original Message -----
From: "James Hogarth" <james.hogarth@xxxxxxxxx>
To: "centos" <centos@xxxxxxxxxx>
Sent: Tuesday, August 29, 2017 2:03:44 PM
Subject: Re: Syncronize systemctl status with reality?
On 29 Aug 2017 17:58, "Leroy Tennison" <leroy@xxxxxxxxxxxxxxxx> wrote:
The particular issue is with puppetmaster (which admittedly takes 4 minutes
to actually start, setting TimeoutStartSec=300 in it's unit file stopped
the false timeout report) but I have seen it one other time (don't remember
the details).
systemctl status puppetmaster
● puppetmaster.service - Puppet master
Loaded: loaded (/lib/systemd/system/puppetmaster.service; enabled; vendor
preset: enabled)
Active: failed (Result: resources) since Tue 2017-08-29 11:24:36 CDT; 22min
ago
Process: 897 ExecStart=/usr/bin/puppet master (code=exited,
status=0/SUCCESS)
Aug 29 11:22:39 puppetmaster02 systemd[1]: Starting Puppet master...
Aug 29 11:24:36 puppetmaster02 puppet-master[1233]: Reopening log files
Aug 29 11:24:36 puppetmaster02 puppet-master[1233]: Starting Puppet master
version 3.8.5
Aug 29 11:24:36 puppetmaster02 puppet-master[1233]: Could not run: Address
already in use - listen(2)
Aug 29 11:24:36 puppetmaster02 systemd[1]: puppetmaster.service: PID 1233
read from file /run/puppet/master.pid does not exist or is a zombie.
Aug 29 11:24:36 puppetmaster02 systemd[1]: Failed to start Puppet master.
Aug 29 11:24:36 puppetmaster02 systemd[1]: puppetmaster.service: Unit
entered failed state.
Aug 29 11:24:36 puppetmaster02 systemd[1]: puppetmaster.service: Failed
with result 'resources'.
However, ps -ef | grep puppet (run just after the above) returns
puppet 1380 1 0 11:26 ? 00:00:08 Passenger RubyApp: /usr/share/puppet/rack/
puppetmasterd
root 2015 1341 0 11:48 pts/0 00:00:00 grep --color=auto puppet
Earlier ps .. also reported
puppet 1355 1166 3 11:26 ? 00:00:01 Passenger AppPreloader:
/usr/share/puppet/rack/puppetmasterd
And, the "bottom line", puppet agent -t on a client works. It reports
finishing the catalog run and the client's yaml files on puppetmaster are
up to date.
Is there a command to tell systemd to re-scan running state and update its
understanding on what it finds? I tried systemctl daemon-reload just to be
sure that didn't solve the problem before posting this.
_______________________________________________
CentOS mailing list
CentOS@xxxxxxxxxx
https://lists.centos.org/mailman/listinfo/centos
First glance ity looks like someone has started that puppetmaster manually
at some point. As such it's not in a cgroup systemd is tracking so it isn't
aware of it.
Your attempts to start the service are failing because that manually
started instance already has the port open.
Kill it with pkill -f puppet and then use ss -tnp to check for the port
being freed (wait for any time_wait states to go... which is why I'm not
filtering by listen).
Once it's clear then try starting with systemctl
_______________________________________________
CentOS mailing list
CentOS@xxxxxxxxxx
https://lists.centos.org/mailman/listinfo/centos
OK, something weird is definitely going on here, I have the luxury of rebooting this system so it did. Here's what I got, note the time stamps.
ps -ef | grep puppet
root 932 1 0 15:23 ? 00:00:00 /usr/bin/ruby /usr/bin/puppet master
root 1343 1327 0 15:24 pts/0 00:00:00 grep --color=auto puppet
(immediately afterward as fast as I could type:) uptime
15:24:56 up 1 min, 1 user, load average: 0.16, 0.07, 0.02
systemctl status puppetmaster
● puppetmaster.service - Puppet master
Loaded: loaded (/lib/systemd/system/puppetmaster.service; enabled; vendor preset: enabled)
Active: activating (start) since Tue 2017-08-29 15:23:44 CDT; 1min 24s ago
Control: 932 (puppet)
Tasks: 1
Memory: 2.4M
CPU: 4ms
CGroup: /system.slice/puppetmaster.service
└─932 /usr/bin/ruby /usr/bin/puppet master
Aug 29 15:23:44 puppetmaster02 systemd[1]: Starting Puppet master...
After a short delay:
systemctl status puppetmaster
● puppetmaster.service - Puppet master
Loaded: loaded (/lib/systemd/system/puppetmaster.service; enabled; vendor preset: enabled)
Active: failed (Result: resources) since Tue 2017-08-29 15:25:11 CDT; 11s ago
Process: 932 ExecStart=/usr/bin/puppet master (code=exited, status=0/SUCCESS)
Aug 29 15:23:44 puppetmaster02 systemd[1]: Starting Puppet master...
Aug 29 15:25:11 puppetmaster02 puppet-master[1360]: Reopening log files
Aug 29 15:25:11 puppetmaster02 puppet-master[1360]: Starting Puppet master version 3.8.5
Aug 29 15:25:11 puppetmaster02 puppet-master[1360]: Could not run: Address already in use - listen(2)
Aug 29 15:25:11 puppetmaster02 systemd[1]: puppetmaster.service: PID 1360 read from file /run/puppet/master.pid does not exist or is a zombie.
Aug 29 15:25:11 puppetmaster02 systemd[1]: Failed to start Puppet master.
Aug 29 15:25:11 puppetmaster02 systemd[1]: puppetmaster.service: Unit entered failed state.
Aug 29 15:25:11 puppetmaster02 systemd[1]: puppetmaster.service: Failed with result 'resources'.
ps -ef | grep puppet
root 1373 1327 0 15:25 pts/0 00:00:00 grep --color=auto puppet
date
Tue Aug 29 15:26:04 CDT 2017
uptime
15:26:08 up 2 min, 1 user, load average: 0.08, 0.06, 0.02
(Getting a little smarter...)
date;uptime;ps -ef | grep puppet note the AppPreloader
Tue Aug 29 15:27:41 CDT 2017
15:27:41 up 4 min, 1 user, load average: 0.04, 0.05, 0.02
puppet 1377 1167 1 15:26 ? 00:00:01 Passenger AppPreloader: /usr/share/puppet/rack/puppetmasterd
puppet 1402 1377 4 15:26 ? 00:00:03 Passenger RubyApp: /usr/share/puppet/rack/puppetmasterd
root 1457 1327 0 15:27 pts/0 00:00:00 grep --color=auto puppet
date;uptime;systemctl status puppetmaster
Tue Aug 29 15:28:48 CDT 2017
15:28:48 up 5 min, 1 user, load average: 0.15, 0.10, 0.03
● puppetmaster.service - Puppet master
Loaded: loaded (/lib/systemd/system/puppetmaster.service; enabled; vendor preset: enabled)
Active: failed (Result: resources) since Tue 2017-08-29 15:25:11 CDT; 3min 37s ago
Process: 932 ExecStart=/usr/bin/puppet master (code=exited, status=0/SUCCESS)
Aug 29 15:23:44 puppetmaster02 systemd[1]: Starting Puppet master...
Aug 29 15:25:11 puppetmaster02 puppet-master[1360]: Reopening log files
Aug 29 15:25:11 puppetmaster02 puppet-master[1360]: Starting Puppet master version 3.8.5
Aug 29 15:25:11 puppetmaster02 puppet-master[1360]: Could not run: Address already in use - listen(2)
Aug 29 15:25:11 puppetmaster02 systemd[1]: puppetmaster.service: PID 1360 read from file /run/puppet/master.pid does not exist or is a zombie.
Aug 29 15:25:11 puppetmaster02 systemd[1]: Failed to start Puppet master.
Aug 29 15:25:11 puppetmaster02 systemd[1]: puppetmaster.service: Unit entered failed state.
Aug 29 15:25:11 puppetmaster02 systemd[1]: puppetmaster.service: Failed with result 'resources'.
(And finally)
date;uptime;ps -ef | grep puppet
Tue Aug 29 15:36:48 CDT 2017
15:36:48 up 13 min, 1 user, load average: 0.00, 0.01, 0.00
puppet 1402 1 0 15:26 ? 00:00:03 Passenger RubyApp: /usr/share/puppet/rack/puppetmasterd
root 1704 1327 0 15:36 pts/0 00:00:00 grep --color=auto puppet
date;uptime;systemctl status puppetmaster
Tue Aug 29 15:36:57 CDT 2017
15:36:57 up 13 min, 1 user, load average: 0.00, 0.01, 0.00
● puppetmaster.service - Puppet master
Loaded: loaded (/lib/systemd/system/puppetmaster.service; enabled; vendor preset: enabled)
Active: failed (Result: resources) since Tue 2017-08-29 15:25:11 CDT; 11min ago
Process: 932 ExecStart=/usr/bin/puppet master (code=exited, status=0/SUCCESS)
Aug 29 15:23:44 puppetmaster02 systemd[1]: Starting Puppet master...
Aug 29 15:25:11 puppetmaster02 puppet-master[1360]: Reopening log files
Aug 29 15:25:11 puppetmaster02 puppet-master[1360]: Starting Puppet master version 3.8.5
Aug 29 15:25:11 puppetmaster02 puppet-master[1360]: Could not run: Address already in use - listen(2)
Aug 29 15:25:11 puppetmaster02 systemd[1]: puppetmaster.service: PID 1360 read from file /run/puppet/master.pid does not exist or is a zombie.
Aug 29 15:25:11 puppetmaster02 systemd[1]: Failed to start Puppet master.
Aug 29 15:25:11 puppetmaster02 systemd[1]: puppetmaster.service: Unit entered failed state.
Aug 29 15:25:11 puppetmaster02 systemd[1]: puppetmaster.service: Failed with result 'resources'.
The AppPreloader is doing things (and probably confusing systemd in the process) but I didn't start that, it was a part of the reboot. I looked at /etc/init.d/puppetmaster to see if something strange was being done and it was one of the simpler init scripts I've seen, uses start-stop-daemon.
I don't have any control over this, 'systemctl list-unit-files | grep puppet' shows the puppetmaster.service is enabled, I just need a "cleanup" solution, any ideas? And, BTW, thanks for any feedback.
_______________________________________________
CentOS mailing list
CentOS@xxxxxxxxxx
https://lists.centos.org/mailman/listinfo/centos
