Summary:
Apache HTTP Server 2.2.17 running on Windows Server 2003 & 2008 has slight HTTP transfer performance problems with Windows XP clients, and *extreme* performance problems with Windows 7 clients.
Apache HTTP Server running on RedHat Linux 5 performs fine with both XP and W7 clients.
IIS and Small HTTP Server on Windows 2003/2008 perform fine with both XP and W7 clients.
The purpose of this email is simply to report our findings for the benefit of anyone else that may have encountered similar problems. (And if anyone has the ability or desire to dive in and test other clients or scenarios, all the better. Ideally, it would be great if others could reproduce and confirm the findings so that this ultimately gets investigated and resolved with the Apache HTTP server for Windows.)
I know, I know, Apache server can't possibly have problems running on the Windows platform that causes different clients to perform differently -- it must be something else. Depending on your perspective and the tests run, the evidence will point in different directions and send you in circles. We're just reporting our findings.
Background:
We initially noticed problems using IBM Installation Manager which downloads several GBs of files from our internal HTTP server (using Apache on Windows Server 2003) in order to install/update various IBM products. With XP clients, transfers were fairly fast at 4000-5000KB/s. With Win7 clients, transfers were pretty slow at 350-1000KB/s. HTTP downloads from the internet seem normal, it seems the problem only occurred when using HTTP against an internal server. We were able to reproduce the problem using Curl as the HTTP client, so HTTP client didn't seem to be the problem.
Obviously the problem is with the client OS. Yes, there are widespread reports of network performance problems with Vista and Windows 7. Microsoft docs indicate the most common cause is problematic network devices that don't handle TCP window scaling or other RFC 1323 options very well, so there are various workarounds to adjust the TCP settings, most noteably the auto-tuning level. Network traces confirmed that window scaling and other 1323 flags were indeed present and being preserved between the client and the server. Nonetheless, we fully explored all the Windows 7 TCP global parameters: rss, chimney offloading, dma, dca, auto-tuning, congestion control provider, ecn, timestamps. Nothing resolved the issue.
Network traces taken in an isolated testing lab with desktop clients running on identical hardware on a 100mbit network also seemed to point to problems with the client OS: after the server sends a window of packets (44) and the client dutifully acknowledges every other one, when the last packet is sent with the push flag, the Windows 7 client takes between 100-300ms longer to send the acknowledgement compared to the Windows XP client. (Hmm...200ms sounds like an issue with selective acknowledgements/nagle's algorithm/TCP_NO_DELAY -- nope, explored that too.)
Again, obviously a problem with the client. Maybe it's a 64/32-bit issue (nope), maybe it's a virusscanner or firewall issue (nope and nope), maybe results would be different if we installed W7 SP1 (nope), removed the QoS provider (nope), or disabled IPv6 (nope). Thinking it could be a network driver issue we made sure ALL drivers were up-to-date, and verified the driver settings were the same between the XP and W7 machines (all the same, W7 has more options but nothing we tried changing improved performance). Monitoring the port on the network switch, we could confirm that duplexing mismatch was not an issue. Oh, we also disabled some RDC feature that's present in Windows 7, no luck.
It was still odd that the problem only seemed to occur from our internal server, HTTP downloads from the internet seemed fine. Out of curiosity we setup a Windows Server 2008 running Apache, and a RedHat Linux server running Apache. The performance of the 2008 server was the same as the 2003 server, but the performance of the Linux server was double that of the Windows machines (10-11MB/s)! In fact, not only did the Windows 7 client maintain 10-11MB/s, but the Windows XP client had the same 10-11MB/s performance as well!
Obviously it's a problem with the server. Since Apache was being used across all platforms, we initially looked at the OS. Again, we tried changing various registry changes to tweak the TCP settings (TcpWindowSize both above and below 64K, Tcp1323Opts (0,1,3), SackOpts, RSS, ICMPRedirect, etc.) -- pretty much everything we could find to no avail. When we setup a file share and copied a large file from the Windows server to the clients directly using Explorer, both the XP and W7 clients performed about the same (roughly 8MB/s). We installed the Test TCP (TTCP) tool on the Windows servers to test the raw TCP traffic and both the XP and W7 clients obtained 10-11MB/s.
We were about to open a ticket with Microsoft and see what they had to say, but we wanted to get to a state where only Microsoft products were involved -- so Microsoft could only blame themselves. So, we already experienced this problem with different HTTP clients (IBM IM, curl), but we went ahead and reproduced the problem with Wget and Firefox and IE8. (Now, performance does vary slightly between clients, but no HTTP client climbs above 1MB/s or so -- certainly far short of the 10-11MB/s that we know is possible!) But, at least the problem still occurs with Microsoft IE as the HTTP client.
Next, we installed IIS on the server. HTTP performance with both XP and W7 clients was now 10-11MB/s, but if we hit against Apache we were still getting 4MB/s with XP and at most 1MB/s with W7! Surely our Apache HTTP server must be setup wrong. But, we just had a default install of Apache HTTP 2.2.17 with absolutely no changes at all. It's quite odd that the default install should perform so poorly, but we started stripping down the config file to make it was as bare-bones as possible -- removed as many modules and directives as we could; after all, we were just providing a direct URL to a large file and just wanted it to transfer, nothing more. We've tried the platform specific options (EnableSendfile Off, EnableMMAP Off, Win32DisableAcceptEx), and tweaking the settings for the number of clients, threads, or age doesn't matter since we're just testing one client and one request. No luck.
Of course IIS performs better on Windows than Apache, Microsoft must have tweaked it or optimized it to perform better, right? The number of alternative HTTP servers for Windows appears to be quite small, and the only one that I found that I could install and setup quickly was "Small HTTP Server". Using Small HTTP Server, HTTP performance with both XP and W7 clients was still 10-11MB/s! I guess the deck really isn't stacked -- if Small HTTP Server can achieve 10-11MB/s, then shouldn't Apache be able to achieve this as well?
End result: We have a Windows 2003 Server with Apache HTTP Server, IIS, and Small HTTP Server all installed and running. If a Windows XP client hits IIS or SHTTPS they get 10-11MB/s, but if they hit Apache it gets 4MB/s. If a Windows 7 client hits IIS or SHTTPS it also gets 10-11MB/s, but if it hits Apache it gets AT MOST 1MB/s.
Conclusion: Apache HTTP Server on Windows exhibits performance issues with certain client OSes -- performance is 2 times as slow for XP clients, and 10-30 times slower for W7 clients. I have no explanation as to why. (And I completely acknowledge that the behavior observed doesn't even make sense, but it is what it is!) It seems our Apache/Windows server has had performance issues this whole time we've been using it and didn't realize it with our XP clients, it wasn't until we started testing with Windows 7 clients that the performance problems seem to be amplified.
In our mind we've resolved our issue -- use IIS on Windows Server instead of Apache, or switch to Apache on Linux. Exactly why Apache/Windows has performance issues is unresolved and a lot more investigation could be done, namely testing with additional clients besides just Windows, more detailed network tracing and analysis, or OS/Apache debugging and troubleshooting.
I apologize for the novel, but depending on where you start, what results you focus on, and what biases you carry, the evidence will lead you in different directions. There are definitely "known" performance issues in the network, client and server OSes that had to be eliminated as possible causes first. You really have to get the complete story or spin in circles a few times on your own before the conclusion becomes clear.
Hopefully the "end result" above can serve as simple "steps to reproduce" and allow other users to reproduce and confirm the problem quickly and easily.
I hope somebody else will someday gain some benefit from these findings.
-----Message Disclaimer-----
This e-mail message is intended only for the use of the individual or
entity to which it is addressed, and may contain information that is
privileged, confidential and exempt from disclosure under applicable law.
If you are not the intended recipient, any dissemination, distribution or
copying of this communication is strictly prohibited. If you have
received this communication in error, please notify us immediately by
reply email to Connect@xxxxxxxxxxxxx and delete or destroy all copies of
the original message and attachments thereto. Email sent to or from the
Principal Financial Group or any of its member companies may be retained
as required by law or regulation.
Nothing in this message is intended to constitute an Electronic signature
for purposes of the Uniform Electronic Transactions Act (UETA) or the
Electronic Signatures in Global and National Commerce Act ("E-Sign")
unless a specific statement to the contrary is included in this message.
While this communication may be used to promote or market a transaction
or an idea that is discussed in the publication, it is intended to provide
general information about the subject matter covered and is provided with
the understanding that The Principal is not rendering legal, accounting,
or tax advice. It is not a marketed opinion and may not be used to avoid
penalties under the Internal Revenue Code. You should consult with
appropriate counsel or other advisors on all matters pertaining to legal,
tax, or accounting obligations and requirements.
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@xxxxxxxxxxxxxxxx
" from the digest: users-digest-unsubscribe@xxxxxxxxxxxxxxxx
For additional commands, e-mail: users-help@xxxxxxxxxxxxxxxx
|