I noticed a couple of days back that in the event of an apt-get update of the target services, our services did not start.
Apparently, the service was in a sleeping state when it was upgraded. We are also capturing the SIGTERM signal to do a graceful shutdown(when systemctl tell us to stop) but since the service was sleeping for a definite period the shutdown was delayed. (more than 90s) causing the systemctl to kill the process by sending SIGKILL.
We expected the systemctl to restart the update binary thereafter but it didn't
Trying to replicate the same thing behaviour thus far has been futile
I check the syslog but no helpful information can be derived
Jun 1 07:22:21 edge systemd[1]: Stopping A service which handles the interaction among other edge services...
Jun 1 07:22:43 edge systemd[1]: Started Utility to generate JSON containing server statistics and health parameters.
Jun 1 07:23:51 edge systemd[1]: sbagent.service: State 'stop-sigterm' timed out. Killing.
Jun 1 07:23:51 edge systemd[1]: sbagent.service: Killing process 30165 (sbagent) with signal SIGKILL.
Jun 1 07:23:51 edge systemd[1]: sbagent.service: Main process exited, code=killed, status=9/KILL
Jun 1 07:23:51 edge systemd[1]: sbagent.service: Failed with result 'timeout'.
Jun 1 07:23:51 edge systemd[1]: Stopped A service which handles the interaction among other edge services.
I can say with confidence that the services binary both old and new were not buggy and they did not crash on start / bootstrap
Hence I reaching out to understand if anyone else has seen this behaviour and how can I reproduce it.
Thanks
Disclaimer: This e-mail and any documents, files, or previous e-mail messages appended or attached to it may contain confidential and/or privileged information. If you are not the intended recipient (or have received this email in error) please notify the sender immediately and delete this e-mail. Any unauthorized copying, disclosure or distribution of the material in this email is strictly prohibited & unlawful. The recipient acknowledges that Margo Networks Private Limited (SugarBox) may be unable to exercise control or ensure or guarantee the integrity of the text of the email message and the text is not warranted as to completeness and accuracy. Before opening and accessing the attachment, if any, please check and scan for virus
