It was more than 9 months ago I discovered a problem with the graceful restarts on a default Virtualmin installation with the default execution mode (mod_fcgid), but recently I had the time to dig deeper and experiment. Since Virtualmin uses Apache + mod_fcgid by default, the experiments will probably lead to the same results on any Apache 2.2 + mod_fcgid 2.3.7 installation. This is not the widely known problem with leftover processes that never get killed on a graceful restart, this is something else - the processes get forcefully killed way to soon and you don't get the output to the browser. Please, test it on your setup and report back the result.
What is the setup:
CentOS 6.4 x86_64 minimal installation
Virtualmin 4.02.gpl GPL installed by the automatic .sh script, all default settings (you can skip this, the problem is probably not virtualmin related)
mod_fcgid.x86_64 2.3.7-1.el6 from the virtualmin repo (other should work too)
httpd.x86_64 1:2.2.15-29.el6.vm.1 from the virtualmin repo (other should work too)
php 5.3.3 from the official repo
Single virtual domain, running under the default FCGId execution mode, with 90 sec php execution time and fcgid IO wait.
Single test.php file containing
<?php
for($i = 1; $i <= 30; $i++) {
echo $i."\n";
sleep(1);
}
?>
What is the error:
Run the script via browser, then go and do a graceful restart on apache (service httpd graceful). After around 12 seconds you are going to see "No data received" error in you browser (Chrome) and the following in the apache error log:
(22)Invalid argument: mod_fcgid: can't lock process table in pid 25570
(the pid number will be different of course)
Further experiments show that this script gets forcefully killed before ending.
If you reduce the time the script executes to 5 seconds ($i <= 4), you'll get the same result, this time after 5 seconds.
Further experiments show this process completes, but you still get the errors both in the browser and the error log.
Try it and post your result.
Dig:
It is probably a problem of mod_fcgid
I tweaked the experiment adding a file write at the end of the script which shows which script completes and which gets killed before that. I got the result above.
Add this inside the loop:
file_put_contents("test.txt", "test run for: ".$i." seconds");
So why 12 seconds and where is this set. After some time I discovered that increasing FcgidErrorScanInterval to 60 will let the second process to complete (but still you get the errors).
If you check the code of mod_fcgid In fcgid_pm_main.c, the graceful restart should be performed by the function kill_all_subprocess() but obviously the scan_errorlist() is also executed even if there is a check for procmgr_must_exit().
The error in the log "can't lock process table in pid 25570" probably means that some information about the process is destroyed immediately upon the graceful restart (the mutex), so we will never get the result back.
Even if we get around the early termination of the processes increasing FcgidErrorScanInterval the second problem is actually bigger - all your users are going to see this error.
Do you get the same errors and do you have idea how to fix mod_fcgid?
