Re: Graceful draining of KeepAlive Connections (Apache Users)

We had a similar issue with Keep-Alives so I catch USR1 in the application code and set a flag and then at the end of that request I set connection closed header and call child_terminate to kill off that child.

IIRC, MaxRequestsPerChild doesn't include the number of keep-alive requests, so another thing to keep in mind.

Did you try reducing the keep-alive timeout? 60 seconds seems like a long time to tie up a process.

On Tue, Nov 15, 2016 at 10:59 AM, Alex Kaiser <alextkaiser@xxxxxxxxx> wrote:

Hello,

I am having trouble gracefully draining connections from httpd when we have KeepAlive turned on. The problem seems specific to idle KeepAlive connections. It looks like when a process has an established KeepAlive connection and we send the main httpd process a WINCH or a USR1 signal, the KeepAlive connection will stay alive until it gets another request, and then once it is finished with that request it will shut down. However, if it doesn't get another request on the connection, the process will hang around fro KeepAliveTimeout since you issued the signal. I would like for idle KeepAlive connections to shut down immediately, is this not technically possible for some reason? If it is possible, is it a bug that the connections stick around?

The problem with them sticking around, is that when I want to fully shut down the server some of these KeepAlive connections might finally get a request, and thus if I hard kill a server before waiting KeepAliveTimeout, browsers can see errors.

When I look at the server-status page, these processes change from K to G immediately, but will stay in the G mode the whole time they are waiting. When the mode of operation changes the SS column (seconds since last requests) doesn't get reset and will keep going up, so this could possibly go all the way up to KeepAliveTimeout * 2 if you start the restart process when a connection is near the KeepAliveTimeout.

I've done most of my testing using Apache httpd 2.2.31 with mod_jk 1.2.31, I also did some testing with Apache httpd 2.4.23 with mod_jk 1.2.42 and saw the same behavior. I did most of my testing using the prefork module, but I also saw the same behavior when I twas testing with the event module. The KeepAlive settings I was using are:

KeepAlive on
KeepAliveTimeout 60
MaxKeepAliveRequests 100

The small python script I was using to test keepAlive connections (the behavior was the same when I was testing using browsers):

import requests
import time

s = requests.Session()
r = s.get("<server address>")
print("Sleeping")
time.sleep(500)

Thanks,
Alex Kaiser

Bill Moseley
moseley@xxxxxxxx