On 10/11/18 6:33 AM, Bryce Pepper wrote:
Adrian,
Thanks for being willing to dig into this.
You are correct there are other scripts being called from mine (delivered by BMC with their software). In order to stay in support and work with their updates I use the vendor supplied scripts/programs.
The Control-M product is installed on this single server and is broken down into the following parts:
Enterprise server with dedicated postgresql instance
Distributed datacenter with agent and dedicated postgresql instance
Linux datacenter with with agent and dedicated postgresql instance
To cut down on the noise, my post only focused on the "Distributed" side and shutdown process -- although the ControlM_Shutdown.service unit stop script manages all of the above components.
In the ControlM_Shutdown.service there is a requires statement identifying that network must be available while this systemd unit runs.
You noticed that the eth0 disconnected in the /var/log/messages. I showed that to highlight that the unit was not executing in the order I had intended, again refer to the requires statement.
The second shebang is from one of the invoked subscripts (stop_ctmdist_server.sh) and is the "main" shutdown sequence for the Distributed datacenter (I think the "SQL server" echo from BMC is because it can be configured with other databases and they use it in a generic term --- not meaning sqlserver from Microsoft).
The dbversion check is being used to verify pgsql instance for this datacenter is running and returns a non-zero return code if the instance is unreachable (I could use pg_isready or pg_ctl but would diverge further from the BMC supported technique).
You probably also noticed in the earlier posted shutdown service a requires of CTM_Postgre.service. This was one of my attempts to ensure the instance was available by actually starting the instance outside of the BMC routines (if it is already running the BMC routines will not start -- the dbversion check is on the start side also). I thought if I managed the postgresql instance outside of the product I could ensure it was running. Unfortunately that didn't work as the instance shutdown on its own, presumably a resource (perhaps network) was terminated and postgresql shutdown.
So to restate the original post... It appears the postgresql instance is unavailable when the stop script runs.
Thanks,
Bryce
[root@kccontrolmt01 ~]# systemctl --full cat ControlM_Shutdown.service
# /etc/systemd/system/ControlM_Shutdown.service
[Unit]
Description=Run ControlM shutdown process
Requires=graphical.target multi-user.target network.target network.service sockets.target
DefaultDependencies=no
Before=shutdown.target reboot.target halt.target poweroff.target kexec.target
Again I am not a systemd expert, but I believe the Before line above is
the opposite of what you want:
https://serverfault.com/questions/812584/in-systemd-whats-the-difference-between-after-and-requires#812589
Above quotes man
page(https://www.freedesktop.org/software/systemd/man/systemd.unit.html):
"... Note that when two units with an ordering dependency between them
are shut down, the inverse of the start-up order is applied. i.e. if a
unit is configured with After= on another unit, the former is stopped
before the latter if both are shut down. ..."
[Service]
Type=oneshot
RemainAfterExit=true
ExecStart=/bin/true
ExecStop=/bin/bash /root/scripts/control-m_shutdown.sh
TimeoutStopSec=4min
[Install]
WantedBy=multi-user.target
[root@kccontrolmt01 ~]#
--
Adrian Klaver
adrian.klaver@xxxxxxxxxxx
