Good evening Squid experts, We have a serious issue with persistent connections. We are testing a large REST-Application which heavily depends on caching and correct cache behaviour in particular concering the rather scarcely used HTTP-Methods like PUT.The scenario is the following: Client <---> Squid Forward Proxy Cache in the same Network <---> Server The Squid (Version 3.1.11 on Ubuntu Linux 11.04 running in VmWare 8) is configured as an explicit proxy, i.e. our client sends the full path-uri and host of the origin server over a tcp connection to the proxy on port 3128. Our client normally is the Restlet-Java-Framework, but we also used nodejs, telnet and curl to verify the behaviour, which we don't understand: GET works as expected: Client <-GET; persistent tcp connection-> Squid Forward Cache in the same Network <-GET; persistent tcp connection-> Server POST and PUT don't: Client <-POST/PUT; persistent tcp connection-> Squid Forward Cache in the same Network <-POST/PUT; NO! persistent tcp connection-> Server The Squid finishes the TCP-Connection, although via Connection: keep-alive it explicitly announces not to do so. That is a very bad performance killer since the TCP slow start algorithm is - yes indeed - slow. Squid opens a new TCP connection for every POST and PUT for the server. With a single unexplicable exception: Client <-PUT without Body (Content); persistent tcp connection-> Squid Forward Cache in the same Network <-PUT without body; persistent tcp connection-> Server So only if the PUT doesn't contain an HTTP-Body, the squid<->server connection is persistent. The client<->squid connection on the other hand is persistent (we use both the non-standard Proxy-Connection: keep-alive and the correct hopy-by-hop Connection: keep-alive in our request). Here is a detailed example of a successful GET: 1. Client sends to Squid: GET http://192.168.14.194/ HTTP/1.1Cache-Control: no-cache, max-age=0Date: Tue, 11 Oct 2011 16:53:44 GMTContent-Length: 0Accept: application/x-java-serialized-object, application/x-java-serialized-object+xmlHost: 192.168.14.194User-Agent: Restlet-Framework/2.1rc1Connection: keep-aliveProxy-Connection: keep-alive 2. Squid sends to Server: GET / HTTP/1.1Date: Tue, 11 Oct 2011 16:53:44 GMTContent-Length: 0Accept: application/x-java-serialized-object, application/x-java-serialized-object+xmlHost: 192.168.14.194User-Agent: Restlet-Framework/2.1rc1Via: 1.1 localhost (squid/3.1.11)X-Forwarded-For: 192.168.14.194Cache-Control: no-cache, max-age=0Connection: keep-alive 3. Server sends to Squid: HTTP/1.1 200 OKDate: Tue, 11 Oct 2011 16:53:44 GMTServer: Restlet-Framework/2.1rc1Vary: Accept-Charset, Accept-Encoding, Accept-Language, AcceptConnection: keep-aliveContent-Length: 5Content-Type: text/plain; charset=UTF-8 blubb 4. Squid sends to Client: HTTP/1.0 200 OKDate: Tue, 11 Oct 2011 16:53:44 GMTServer: Restlet-Framework/2.1rc1Vary: Accept-Charset, Accept-Encoding, Accept-Language, AcceptContent-Length: 5Content-Type: text/plain; charset=UTF-8X-Cache: MISS from localhostX-Cache-Lookup: MISS from localhost:3128Via: 1.0 localhost (squid/3.1.11)Connection: keep-alive blubb 5. Both TCP-Connections Client<->Squid and Squid<->Server are still open and used for subsequent GET Requests. And now an example of a failed persistent connection between squid and server: 1. Client sends to Squid: POST http://192.168.14.194/ HTTP/1.1Cache-Control: no-cache, max-age=0Date: Tue, 11 Oct 2011 16:53:46 GMTContent-Length: 4Content-Type: text/plain; charset=UTF-8Accept: application/x-java-serialized-object, application/x-java-serialized-object+xmlHost: 192.168.14.194User-Agent: Restlet-Framework/2.1rc1Connection: keep-aliveProxy-Connection: keep-alive test 2. Squid does Threeway-SYN-SYN/ACK-ACK-Handshake with Server and sends: POST / HTTP/1.1Date: Tue, 11 Oct 2011 16:53:45 GMTContent-Length: 4Content-Type: text/plain; charset=UTF-8Accept: application/x-java-serialized-object, application/x-java-serialized-object+xmlHost: 192.168.14.194User-Agent: Restlet-Framework/2.1rc1Via: 1.1 localhost (squid/3.1.11)X-Forwarded-For: 192.168.14.194Cache-Control: no-cache, max-age=0Connection: keep-alive test 3. Server replies over that connection: HTTP/1.1 200 OKDate: Tue, 11 Oct 2011 16:53:45 GMTServer: Restlet-Framework/2.1rc1Vary: Accept-Charset, Accept-Encoding, Accept-Language, AcceptConnection: keep-aliveContent-Length: 4Content-Type: text/plain; charset=UTF-8 test 4. Wih FIN-ACK the Squid closes the connection to the server and replies to the client: HTTP/1.0 200 OKDate: Tue, 11 Oct 2011 16:53:46 GMTServer: Restlet-Framework/2.1rc1Vary: Accept-Charset, Accept-Encoding, Accept-Language, AcceptContent-Length: 4Content-Type: text/plain; charset=UTF-8X-Cache: MISS from localhostX-Cache-Lookup: MISS from localhost:3128Via: 1.0 localhost (squid/3.1.11)Connection: keep-alive test 5. For the subsequent requests, the client uses the same connection to the Squid, which is still open and the Squid opens a new connection to server to forward a PUT or POST. If the Squid isn't used at all, the client<->squid connection is persistent. To us that behaviour seems inconsistent and faulty. RFC 2616 doesn't make any comments about closing a persistent connection if a POST/PUT/DELETE was send. Thus it's not reasonable at all. Is it a bug? Btw, yesterday we already reported an unrelated TCP issue of Squid, which totally abandons a Connection if a Request having an If-Match Header results in a Cache Hit (http://bugs.squid-cache.org/show_bug.cgi?id=3379 ). We are very thankful for any suggestions concerning our persistent connection problem, which keeps us up at night. We are trying to run a benchmark comparison and the TCP-Connection-kills totally ruin the performance. Thanks in advance and bye,Flo & Felix, University of Hamburg
