Hello, we are having a hard time tracking down why a very small number of our apache connections have close_notify packet sent back to the clients after the request is received and not a http response. Has anyone seen this type of behavior before? I'm attaching a screenshot of the packet capture, But here is what I understand to be happening: 1) one of our apache servers (IP start with 54.) sends back a 200 response from a previous request 2) a client (IP starts with 10.) makes another request on the same connection 3) apache sends an ACK 4) apache then sends a close_notify 5) client ACK the close_notify 6) apache sends a FIN ACK 7) client sends a close_notify as well and the connection is closed At this point: the apache server log does not have this attempted request in it at all. While the screenshot shows a GET, we have also noticed this behavior for POST requests as well. You can tell from the timestamps in the screenshot that no long period of time has passed, and no we don't think any timeouts should have been triggered. A little about our setup: We have many client apps that constantly long poll on the same URIs. Our backends will hold onto that request for 15 seconds and then respond with the data needed. Then the clients will make another long poll request, and repeat this behavior forever. All our requests are over https, we use KeepAlive, we are on 2.4.27, we use mpm_event and all requests then use mod_rewrite + mod_proxy to get sent to a backend tomcat app over https. OS is linux kernel 4.14.88, openssl 1.0.2k. Apache has 5 vhosts configured, each with a different ServerAlias and wildcard certificate for that different hostname. We have at least 10 child processes for all of our apache servers. None of our servers appear to near any worker or thread limits. And no gracefuls are occuring at this times. Here are some of our settings that might be of interest: KeepAlive on KeepAliveTimeout 5 TimeOut 60 MaxKeepAliveRequests 100 We do not set MaxConnectionsPerChild, so it should default to 0. We have been unable to replicate this problem in a non-production environment, And we are unable to turn on trace level logging in prod because the volume of data is just too great since this does not happen at any regular cadence. We have tried using these settings to disable keepalives and change connection shutdown behavior: SetEnvIf Host "xyz\.abc\.com" nokeepalive ssl-unclean-shutdown downgrade-1.0 force-response-1.0 It appears to be masking the issue so for, but tests need to be run for a lot longer to be truly confident. Since disabling keepalives seems to mask the problem, I went digging into the source code to see how keepalive connections are closed. But I could not find any place in the normal request response flow where the close_notify would be sent back before the response and terminate the request early. Has anyone seen this type of connection behavior before? Any advice on how to debug this further? Thanks in advance - chris
Attachment:
close_notify.png
Description: close_notify.png
Attachment:
close_notify_packet.png
Description: close_notify_packet.png
--------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscribe@xxxxxxxxxxxxxxxx For additional commands, e-mail: users-help@xxxxxxxxxxxxxxxx
|