I have a system that uses a web server (Apache) to allow non-technical users to start long-running jobs on our systems using a browser. The system starts the job and returns info that allows the browser to poll a JSON file that the long-running job maintains to give the user status info about the job (simplified description.)
But I determined that if Apache restarts on that server, it kills that process. I had done this about 10 years ago and it worked without doing this.
I tried to detach the process by forking, and I also tried to have catch SIGTERM. As a result, Apache restart takes over a minute because something in the server (or apachctl I’m not sure) tries a number of time to send that process a SIGTERM (which it ignores) and then just sends it a SIGKILL.
Here’s what syslog shows:
Jan 13 04:09:57 wcmdev systemd[1]: apache2.service: State 'stop-final-sigterm' timed out. Killing.
Jan 13 04:09:57 wcmdev systemd[1]: apache2.service: Killing process 113975 (php) with signal SIGKILL.
Jan 13 04:09:57 wcmdev systemd[1]: apache2.service: Failed with result 'timeout'.
Jan 13 04:09:57 wcmdev systemd[1]: Stopped The Apache HTTP Server.
Jan 13 04:09:57 wcmdev systemd[1]: apache2.service: Found left-over process 113975 (php) in control group while starting unit. Ignoring.
Jan 13 04:09:57 wcmdev systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Jan 13 04:09:57 wcmdev systemd[1]: Starting The Apache HTTP Server...
Here’s what the long running process looks like. Note that the PPID is 1, so it is detached.
root@wcmdev:/var/www/wcm# ps -f -p 110733
UID PID PPID C STIME TTY TIME CMD
www-data 110733 1 0 02:58 ? 00:00:00 php /media/sf_Shared/wcm/src/business/api/system/../../../jobs/daemon/testDaemon.php --id=3852114
I search the list for stop-final-sigterm, and could not find anything.
root@wcmdev:/var/www/wcm# /usr/sbin/apache2 -v
Server version: Apache/2.4.41 (Ubuntu)
Server built: 2020-08-12T19:46:17
root@wcmdev:/var/www/wcm# uname -a
Linux wcmdev 5.4.0-48-generic #52-Ubuntu SMP Thu Sep 10 10:58:49 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
|