Re: Apache 2.0.58 + Solaris 5.9: status "...reading..." & TCP state "FIN_WAIT_2"

Sander Temme <sctemme@xxxxxxxxxx> · Fri, 26 Jan 2007 09:18:46 -0800

Chirouze,

On Jan 26, 2007, at 8:41 AM, Chirouze Olivier wrote:

Thanks for your prompt reply. We will definitively upgrade soon, just
"to be up to date"... But because nothing is said about that point in
docs I'm wondering if that will make any difference...

I doubt it. The Apache 2.0.x branch at this moment mainly gets  

security fixes, and not a lot else.  The difference between  

subsequent versions tends to be fairly small, and we tend to tell you  

what the fixes are.

You tell us that you are seeing an unusually high number of children  

sitting in 'R' stage, which means they are attempting to read a full  

request.  Reading requests *should* not take very long (this also  

depends on the kind of requests you expect to get), and it may be  

that you are under attack.  This may be deliberate or accidental.

Do you have something like a Firewall in front of your server that  

drops idle TCP connections?  If this happens, your Apache server will  

not notice this and keep trying to read the incoming data.

In any case, you may try to reduce the value of the Timeout directive  

in your configuration file.

The high number of FIN_WAIT_2 connections you observe may be normal,  

but it may be the result of the situation sketched above.  Once  

Apache is done with a connection, it'll close that and forget about  

it.  The Apache child will then get ready to accept a new connection,  

and the kernel takes care of closing the TCP connection.  It does  

this by sending a FIN packet to the client, who sends an ACK back.   

Then the client is supposed to send its own FIN, and while this  

happens the server will have that socket in FIN_WAIT_2 state.  After  

the client sends its FIN, the server ACKnowledges and puts the socket  

into TIME_WAIT before it can be re-used.  You can see a full diagram  

of this process in Figure 18.12 in TCP Illustrated, Volume 1 by W.  

Richard Stevens.

If you are dealing with clients that don't bother sending their FIN,  

or have something in-between that messes with the connection and its  

termination, you may be stuck with this.  You can use kernel tunables  

like the ones you mention below to reduce the overhead.

Once again, Apache has nothing to do with the TCP handshake process.   

However, you may be able to reduce the effect of these 'R'  

connections by reducing the timeout and raising the number of  

available child processes (if your log file tells you that the server  

reached MaxClients).

Regards,

Sander

Anyway, we're providing both HTTP and HTTPS. Might be interesting  

to try

recognize if this happens on both? I will have a look at it.

Do you think you might give me the values of the following Unix params
on your Solaris 9 installs?

tcp_fin_wait_2_flush_interval
tcp_keepalive_interval
tcp_ip_abort_interval

Thanks in advance,

Olivier

Olivier CHIROUZE
I&0 Infrastructure
Volvo Information Technology

-----Original Message-----
From: Richard de Vries [mailto:richard_devries@xxxxxxxxx]
Sent: 26 January 2007 17:35
To: users@xxxxxxxxxxxxxxxx
Subject: Re:  Apache 2.0.58 + Solaris 5.9: status
"...reading..." & TCP state "FIN_WAIT_2"

Interesting problem.

I am running Apache 2.0.59 as a reverse proxy on multiple Solaris 9  

and

AIX servers and have never encountered these types of issues. Perhaps
you should try upgrading to 2.0.59 on one of your development machines
and see if that makes a difference. If not, it is most likely an OS
and/or configuration issue.

What other plugins are you running? Also, is this HTTP proxying, or
HTTPS?

----- Original Message ----
From: Chirouze Olivier <olivier.chirouze@xxxxxxxxx>
To: users@xxxxxxxxxxxxxxxx
Sent: Friday, January 26, 2007 9:56:46 AM
Subject:  Apache 2.0.58 + Solaris 5.9: status
"...reading..." & TCP state "FIN_WAIT_2"

Hi all,

I'm facing a quite tricky situation with Apache 2.0.58 running on
Solaris 5.9.

Apache is running as a reverse proxy (mod_proxy + mod_rewrite).
The maximum concurrent connections is set to 150.

Because we reached the maximum a few times and got the reverse proxy
saturated, we started monitoring the Apache status page (/status).
We noticed that many requests were in the "..reading.." state (up to
40!), and they block a lot of slots.

At first, we upgraded from 2.0.47 to 2.0.58 because it seemed there  

was

a security hole in the earlier, fixed in 2.0.48.
I found some explanation here:
http://www.monkeybrains.net/~rudy/example/server_busy_state.html.

The thing is, the situation is starting to appear again with 2.0.58.

We've gone down to Unix and found that most of these requests were in
"FIN_WAIT_2" TCP state, and for a while (approx. 8min!!).

We found this: http://httpd.apache.org/docs/2.0/misc/fin_wait_2.html.
What it says, in a word, is that these things can happen and are

"normal": the connection stays in "FIN_WAIT_2" state until the  

timeout,

if clients do not close it properly. They just say it can be a problem
on the Unix point of view because.

I don't know if this is still true for 2.0 because the article was  

just

copied from 1.3. The thing is, it says that "The connections in
FIN_WAIT_2 do not tie up an httpd process". For us, IT DOES! Every
"..reading.." request happend to be in the "FIN_WAIT_2" state.

We have contacted Sun to get their opinion. The short answer is  

"you can

change the FIN_WAIT_2 timeout but be careful because wrong tuning will

have negative impact. Maybe you should wonder why these connections  

stay

alive". As far as I understood, the connection is not closed by the

client. The server (Apache) does nothing wrong. But maybe it does,  

as it

doesn't leave the process free?

My questions are:
Does anyone have heard about similar problems?
Why do these connections hold a process of Apache while the
documentation says it doesn't?
Do you recon tuning the Unix timeout would help? (current value of
tcp_fin_wait_2_flush_interval: 675000 ms - 11min!! This looks just
huge!)

Thanks in advance,

Olivier

Olivier CHIROUZE
I&0 Infrastructure

Volvo Information Technology

---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server
Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@xxxxxxxxxxxxxxxx
   "   from the digest: users-digest-unsubscribe@xxxxxxxxxxxxxxxx
For additional commands, e-mail: users-help@xxxxxxxxxxxxxxxx

______________________________________________________________________ 

__

____________
Do you Yahoo!?
Everyone is raving about the all-new Yahoo! Mail beta.
http://new.mail.yahoo.com

---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server
Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@xxxxxxxxxxxxxxxx
   "   from the digest: users-digest-unsubscribe@xxxxxxxxxxxxxxxx
For additional commands, e-mail: users-help@xxxxxxxxxxxxxxxx

---------------------------------------------------------------------

The official User-To-User support forum of the Apache HTTP Server  

Project.

See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@xxxxxxxxxxxxxxxx
   "   from the digest: users-digest-unsubscribe@xxxxxxxxxxxxxxxx
For additional commands, e-mail: users-help@xxxxxxxxxxxxxxxx

--
sctemme@xxxxxxxxxx            http://www.temme.net/sander/
Open Source Software Consultant
PGP FP: 51B4 8727 466A 0BC3 69F4  B7B8 B2BE BC40 1529 24AF

smime.p7s