-----Original Message-----
From: William A. Rowe Jr. [mailto:wrowe@xxxxxxxxxxxxx]
Sent: Wednesday, March 09, 2011 4:46 PM
To: users@xxxxxxxxxxxxxxxx
Subject: Re: Slow performance of Apache on Windows
> On 3/9/2011 4:44 PM, William A. Rowe Jr. wrote:
>> On 3/9/2011 3:15 PM, Feldhacker, Chris wrote:
>>>
>>> It was still odd that the problem only seemed to occur from our internal server, HTTP downloads from the internet seemed fine. Out of curiosity we setup a Windows Server 2008 running Apache, and a RedHat Linux server running Apache. The performance of the 2008 server was the same as the 2003 server, but the performance of the Linux server was double that of the Windows machines (10-11MB/s)! In fact, not only did the Windows 7 client maintain 10-11MB/s, but the Windows XP client had the same 10-11MB/s performance as well!
>>
>> On this one issue, did you try 'Enable Sendfile off' on the windows box?
> Sorry, asked and answered in the original summary, I just missed it the first time through.
We did get this problem resolved -- you were right.
If you'd like the tail-between-our-legs explanation and hopefully to benefit anyone else that might encounter this problem:
Ultimately, we did not discover the solution sooner due to inconsistent testing and shortcuts. (Isn't that how it usually goes?) Most testing was done in an isolated lab (especially network captures), but sometimes we'd play with settings on the production network out of convenience. And since the problem occurred with both XP and W7, we would try different things with different clients. And, once we made a change and it didn't work, we would set things back to "default" and go on.
We did try setting the three Apache options for windows: EnableSendfile Off, EnableMMAP Off, Win32DisableAcceptEx. But we must have done this on the production network and tested with the XP client, which only got the expected slow(er) result. However, apparently the EnableSendfile directive really did make a difference, we just didn't realize it (we can only assume the production network must have been under some load at the time). So, we removed the settings back to default and kept searching...
Per your suggestion, we focused on this setting again in the isolated lab -- the XP client achieved the 10MB/s, as did the W7 client. Nice.
Before finding this solution we were giving additional focus to the network traces (after all, the answer must be there, right?). In case anyone has interest, focusing first on traces from the Windows 7 client:
The TCP window size are slightly different, the client was always consistent at 65700, but Apache=64099, IIS=64094, SHTTPS=64094. (Don't know why Apache is 5 bytes higher, but anyway...) The window size really doesn't seem to matter. Looking at the traffic patterns, the server sends 2 packets, and then the client ACKs the last packet sent. But sometimes the server must be sending packets fast enough that 3 packets get through and the client ACKs the last packet sent. So, sometimes the client is ACKing every other packet, sometimes it's every 3rd packet. Which means sometimes the last packet from the server with the Push flag can be 1st, 2nd, or 3rd in the final "set". When the packet is 2nd or 3rd, the client has no problems and an ACK is sent right away with very little delay. When the packet is 1st, then (with Apache) the server waits for an ACK from the client before continuing. Of course, this is where Nagel's algorithm comes into play, which explains the client delay (looking at the traces, it varies between 35-210ms).
I say I don't think the window size matters because the same traffic pattern appears with IIS and SHTTPS -- sometimes the W7 client ACKs every 3rd packet instead of every other packet, so it can still end up on an "odd" boundary. The difference being, with IIS and SHTTPS, when the last packet with the Push flag comes through and if it happens to be 1st, *the server starts on the next sequence and keeps sending packets without stopping to wait for the client ACK*. So, IIS and SHTTPS are continuously streaming without waiting for ACKs of the Push packets, but Apache stops and waits.
(You would think using the TCP_NODELAY option on the client should workaround this problem. We definitely tested this previously with no luck, so we double-checked to make sure and still no luck. We tested this using curl with the "--tcp-nodelay" option, so unless there are issues with curl, I'm not sure why this doesn't seem to result in any improvement...)
So, what does the traffic look like with a W7 client against Apache with the EnableSendfile Off directive? Client ACKs are still sent after every 2 to 3 packets, but when the packet with the Push flag is sent Apache no longer stops and waits for an ACK but continues to stream packets -- just like IIS and SHTTPS did. No only that, but Push packets appear a LOT more frequently, which must mean Apache is streaming content in smaller chunks. There are 5 regular packets and the 6th contains the Push (8192 bytes total). With EnableSendfile On, there are 44 regular packets and the 45th contains the Push (65536 bytes total).
I don't know what Apache is doing differently with the EnableSendfile Off, but it seems to resolve the performance issue by continuously streaming data despite apparently using a smaller buffer size.
(Could probably squeeze an additional 6% performance improvement if the buffer size was larger so that every 6th packet wasn't only 60% full, but at this point I'm not complaining!)
It's hard to draw any conclusions from the XP traces -- time precision of the XP machine must be pretty poor as almost all of the time deltas display as zero; there's occasionally a 15ms blip with no apparent pattern, I assume it just happens to be whenever the clock happens to tick. However, the XP client always ACKs every 1-2 packets, even mid-stream; there is never a sequence of 3 packets. It's almost like the XP client is configured to ACK every packet rather than every other one (strange, nothing in the registry suggests this...perhaps something under a different key). So, with EnableSendfile On, it looks like Apache is still waiting for an ACK, but because it appears the client is trying to ACK every packet there is no delay. With EnableSendfile Off, the traffic appears the same, but again it's really hard to draw any conclusions given the low precision...
So, there you have it.
I think it's really unfortunate that a default install of Apache on Windows has this issue -- and because the symptoms vary so drastically based on the client OS, it's very easy to get mislead and think the problem is with the client...or the application...or the server...or the network.
Does anyone know if there is even a valid scenario when EnableSendfile MUST be turned On on a Windows machine? Seems like Off should be the default on Windows...
Thanks for the feedback and helping us double-check our work!
-----Message Disclaimer-----
This e-mail message is intended only for the use of the individual or
entity to which it is addressed, and may contain information that is
privileged, confidential and exempt from disclosure under applicable law.
If you are not the intended recipient, any dissemination, distribution or
copying of this communication is strictly prohibited. If you have
received this communication in error, please notify us immediately by
reply email to Connect@xxxxxxxxxxxxx and delete or destroy all copies of
the original message and attachments thereto. Email sent to or from the
Principal Financial Group or any of its member companies may be retained
as required by law or regulation.
Nothing in this message is intended to constitute an Electronic signature
for purposes of the Uniform Electronic Transactions Act (UETA) or the
Electronic Signatures in Global and National Commerce Act ("E-Sign")
unless a specific statement to the contrary is included in this message.
While this communication may be used to promote or market a transaction
or an idea that is discussed in the publication, it is intended to provide
general information about the subject matter covered and is provided with
the understanding that The Principal is not rendering legal, accounting,
or tax advice. It is not a marketed opinion and may not be used to avoid
penalties under the Internal Revenue Code. You should consult with
appropriate counsel or other advisors on all matters pertaining to legal,
tax, or accounting obligations and requirements.
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@xxxxxxxxxxxxxxxx
" from the digest: users-digest-unsubscribe@xxxxxxxxxxxxxxxx
For additional commands, e-mail: users-help@xxxxxxxxxxxxxxxx
|