|
I have a system that uses a web server (Apache) to allow non-technical users to start long-running jobs on our systems using a browser. The system starts the job and returns info that allows the browser to
poll a JSON file that the long-running job maintains to give the user status info about the job (simplified description.) But I determined that if Apache restarts on that server, it kills that process. I had done this about 10 years ago and it worked without doing this. I tried to detach the process by forking, and I also tried to have catch SIGTERM. As a result, Apache restart takes over a minute because something in the server (or apachctl I’m not sure) tries a number of
time to send that process a SIGTERM (which it ignores) and then just sends it a SIGKILL. Here’s what syslog shows: Jan 13 04:09:57 wcmdev systemd[1]: apache2.service: State 'stop-final-sigterm' timed out. Killing. Jan 13 04:09:57 wcmdev systemd[1]: apache2.service: Killing process 113975 (php) with signal SIGKILL. Jan 13 04:09:57 wcmdev systemd[1]: apache2.service: Failed with result 'timeout'. Jan 13 04:09:57 wcmdev systemd[1]: Stopped The Apache HTTP Server. Jan 13 04:09:57 wcmdev systemd[1]: apache2.service: Found left-over process 113975 (php) in control group while starting unit. Ignoring. Jan 13 04:09:57 wcmdev systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. Jan 13 04:09:57 wcmdev systemd[1]: Starting The Apache HTTP Server... Here’s what the long running process looks like. Note that the PPID is 1, so it is detached. root@wcmdev:/var/www/wcm# ps -f -p 110733 UID
PID
PPID C STIME TTY
TIME CMD www-data
110733
1 0 02:58 ?
00:00:00 php /media/sf_Shared/wcm/src/business/api/system/../../../jobs/daemon/testDaemon.php --id=3852114 I search the list for stop-final-sigterm, and could not find anything. root@wcmdev:/var/www/wcm# /usr/sbin/apache2 -v Server version: Apache/2.4.41 (Ubuntu) Server built:
2020-08-12T19:46:17 root@wcmdev:/var/www/wcm# uname -a Linux wcmdev 5.4.0-48-generic #52-Ubuntu SMP Thu Sep 10 10:58:49 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
|