Hello, thanks for your help. We followed your advice and built the newest Squid version (squid/3.2.0.13-20111019-r11381) - same issues. The option server_persistent_connections is ON. Also the FIN definitely stems from the Squid. The problem still is: Squid closes its server-side TCP-Connection upon PUT {body}, POST, POST {body}, DELETE {body}. The connections remains persistent upon GET, PUT, DELETE (without content): Method Client <-> Squid Squid <-> Server GET persistent persistent PUT persistent persistent PUT {body} persistent non-persistent POST persistent non-persistent POST {body} persistent non-persistent DELETE persistent persistent DELETE {body} persistent non-persistent Here is as a visual presentation of the problem: http://betablogger.info/persistent_connection.png This behaviour appears both if the server answers with 204 No Content and 200 Success. For instance: Squid says: PUT /PUT HTTP/1.1 Content-Type: text/plain Host: 192.168.14.200 Content-Length: 14 Via: 1.1 ubuntu (squid/3.2.0.13-20111019-r11381) X-Forwarded-For: 192.168.14.200 Cache-Control: max-age=259200 Connection: keep-alive put, put, put. Server replies: HTTP/1.1 200 OK Date: Thu Oct 20 2011 16:29:14 GMT+0200 (Mitteleurop.ische Sommerzeit) Server: Restlet-Framework/2.1rc1 Content-Length: 17 Content-Type: text/plain Connection: Keep-Alive I am your Server. Squid sends FIN ACK in reply. If you want to reproduce this obviously buggy behaviour, here is a NodeJs-client and server you can easily use to reproduce it (you need a Squid) (http://nodejs.org ). Just change the respective IP-addresses. Client.js (you can also download it here: http://betablogger.info/client.js ) var http = require('http'); var originServer = '192.168.14.200'; var proxyServer = '192.168.14.194'; var proxyPort = 3128; var Repeater = function(method, content, times) { var self = this; var options = { host: proxyServer, port: proxyPort, path: 'http://' + originServer + '/' + method, method: method, headers: { 'Proxy-Connection' : 'keep-alive', 'Content-Type' : 'text/plain', Host: originServer, Connection: 'keep-alive' } }; if(content == null) content = ''; if(times == null) times = 10; this.makeRequests = function() { for(var i = 1; i <= times; i++) { console.log(options.method + '-Request no ' + i + ' ist being sent...'); var req = http.request(options, onResponse); var body = new Buffer(content); req.setHeader('Content-Length', body.length); req.end(body); } } var onEnd = function() { console.log('Finished receiving Response for ' + options.method + '-Request.'); }; var onResponse = function(res) { res.on('end', onEnd); }; } function test(method, content, times) { r = new Repeater(method, content, times); r.makeRequests(); }; test('GET'); test('PUT'); test('PUT', 'put, put, put.'); test('POST'); test('POST','My new post.'); test('DELETE'); test('DELETE', 'Die!'); and Server.js (http://betablogger.info/server.js ) var http = require('http'); var originServer = '192.168.14.200'; var content = 'I am your Server.'; http.createServer(function (req, res) { console.log(req.method + '-Request received.'); var body = new Buffer(content); req.on('end', function() { res.writeHead(content === '' ? 204 : 200, { 'Date': new Date(), 'Server': 'Restlet-Framework/2.1rc1', 'Content-Length': body.length, 'Content-Type': 'text/plain', 'Connection': 'Keep-Alive' }); res.end(body); }); }).listen(80, originServer); console.log('Server running at http://' + originServer + ':80/'); Use a network monitor like Wireshark (http://www.wireshark.org/ ) or Microsoft Network Monitor to observe the faulty behaviour of the Squid. Should we report this as a bug? We are frankly quite aghast, that a bug of these proportions and effects went unobserved for such a long time. Thanks in advance, Florian and Felix ---------------------------------------- > > NP: inserted a few empty lines to make the text paragraphs > identifiable. > > On Tue, 11 Oct 2011 20:25:06 +0200, Franz Kafka wrote: > > Good evening Squid experts, > > We have a serious issue with persistent connections. We are testing a > > large REST-Application which heavily depends on caching and correct > > cache behaviour in particular concering the rather scarcely used > > Then if you have not already done so please look over the compliance > checklist at: > http://wiki-squid-cache.org/Features/HTTP11 > > It covers milestone releases in the common Squid versions which > advertise 1.1. There are still numbers of networks using Squid 2.6 and > 2.5 though which have 1.0 compliance. And networks using un-rated > releases. So this is not a complete picture, just an idea of what can be > expected out in the Internet. > > HTTP/1.1 correct behaviour is an ongoing effort. Sponsorship is > appreciated, working patches even more so. > > <snip> > > So only if the PUT doesn't contain an HTTP-Body, the squid<->server > > connection is persistent. The client<->squid connection on the other > > hand is persistent (we use both the non-standard Proxy-Connection: > > keep-alive and the correct hopy-by-hop Connection: keep-alive in our > > request). > > You should be able to drop Proxy-Connection entirely. Squid-3.1 emits > it by mistake in some releases. All squid accept Connection:. > > > Here is a detailed example of a successful GET: > > 1. Client sends to Squid: > > GET http://192.168.14.194/ HTTP/1.1Cache-Control: no-cache, > > max-age=0Date: Tue, 11 Oct 2011 16:53:44 GMTContent-Length: 0Accept: > > application/x-java-serialized-object, > > application/x-java-serialized-object+xmlHost: > > 192.168.14.194User-Agent: Restlet-Framework/2.1rc1Connection: > > keep-aliveProxy-Connection: keep-alive > > 2. Squid sends to Server: > > GET / HTTP/1.1Date: Tue, 11 Oct 2011 16:53:44 GMTContent-Length: > > 0Accept: application/x-java-serialized-object, > > application/x-java-serialized-object+xmlHost: > > 192.168.14.194User-Agent: Restlet-Framework/2.1rc1Via: 1.1 localhost > > (squid/3.1.11)X-Forwarded-For: 192.168.14.194Cache-Control: no-cache, > > max-age=0Connection: keep-alive > > 3. Server sends to Squid: > > HTTP/1.1 200 OKDate: Tue, 11 Oct 2011 16:53:44 GMTServer: > > Restlet-Framework/2.1rc1Vary: Accept-Charset, Accept-Encoding, > > Accept-Language, AcceptConnection: keep-aliveContent-Length: > > 5Content-Type: text/plain; charset=UTF-8 > > blubb > > > > 4. Squid sends to Client: > > HTTP/1.0 200 OKDate: Tue, 11 Oct 2011 16:53:44 GMTServer: > > Restlet-Framework/2.1rc1Vary: Accept-Charset, Accept-Encoding, > > Accept-Language, AcceptContent-Length: 5Content-Type: text/plain; > > charset=UTF-8X-Cache: MISS from localhostX-Cache-Lookup: MISS from > > localhost:3128Via: 1.0 localhost (squid/3.1.11)Connection: keep-alive > > blubb > > 5. Both TCP-Connections Client<->Squid and Squid<->Server are still > > open and used for subsequent GET Requests. > > And now an example of a failed persistent connection between squid > > and server: > > 1. Client sends to Squid: > > POST http://192.168.14.194/ HTTP/1.1Cache-Control: no-cache, > > max-age=0Date: Tue, 11 Oct 2011 16:53:46 GMTContent-Length: > > 4Content-Type: text/plain; charset=UTF-8Accept: > > application/x-java-serialized-object, > > application/x-java-serialized-object+xmlHost: > > 192.168.14.194User-Agent: Restlet-Framework/2.1rc1Connection: > > keep-aliveProxy-Connection: keep-alive > > test > > 2. Squid does Threeway-SYN-SYN/ACK-ACK-Handshake with Server and > > sends: > > POST / HTTP/1.1Date: Tue, 11 Oct 2011 16:53:45 GMTContent-Length: > > 4Content-Type: text/plain; charset=UTF-8Accept: > > application/x-java-serialized-object, > > application/x-java-serialized-object+xmlHost: > > 192.168.14.194User-Agent: Restlet-Framework/2.1rc1Via: 1.1 localhost > > (squid/3.1.11)X-Forwarded-For: 192.168.14.194Cache-Control: no-cache, > > max-age=0Connection: keep-alive > > test > > 3. Server replies over that connection: > > HTTP/1.1 200 OKDate: Tue, 11 Oct 2011 16:53:45 GMTServer: > > Restlet-Framework/2.1rc1Vary: Accept-Charset, Accept-Encoding, > > Accept-Language, AcceptConnection: keep-aliveContent-Length: > > 4Content-Type: text/plain; charset=UTF-8 > > test > > 4. Wih FIN-ACK the Squid closes the connection to the server and > > replies to the client: > > HTTP/1.0 200 OKDate: Tue, 11 Oct 2011 16:53:46 GMTServer: > > Restlet-Framework/2.1rc1Vary: Accept-Charset, Accept-Encoding, > > Accept-Language, AcceptContent-Length: 4Content-Type: text/plain; > > charset=UTF-8X-Cache: MISS from localhostX-Cache-Lookup: MISS from > > localhost:3128Via: 1.0 localhost (squid/3.1.11)Connection: keep-alive > > test > > 5. For the subsequent requests, the client uses the same connection > > to the Squid, which is still open and the Squid opens a new > > connection > > to server to forward a PUT or POST. If the Squid isn't used at all, > > the client<->squid connection is persistent. > > > To us that behaviour seems inconsistent and faulty. RFC 2616 doesn't > > make any comments about closing a persistent connection if a > > POST/PUT/DELETE was send. Thus it's not reasonable at all. Is it a > > bug? > > > Btw, yesterday we already reported an unrelated TCP issue of Squid, > > which totally abandons a Connection if a Request having an If-Match > > Header results in a Cache Hit > > (http://bugs.squid-cache.org/show_bug.cgi?id=3379 ). > > We are very thankful for any suggestions concerning our persistent > > connection problem, which keeps us up at night. We are trying to run > > a > > benchmark comparison and the TCP-Connection-kills totally ruin the > > performance. > > At least you are getting realistic results out of it. > > > I think sending keep-alive then FIN is a bug. if in fact that FIN is > actually coming from Squid. The rest seems perfectly compliant > behaviour. Annoying and inefficient, but compliant. > > Check you have server_persistent_connections ON. I think OFF there > would send close and FIN. But its possible to be what you see. > > RFC 2616 also specifies that POST is non-idempotent. So there are > tricky conditions around whether new connections are required. Whether a > particular Squid uses persistence or not varies. Though they should at > least be keeping it alive afterwards since Content-Length is present. > > PUT and DELETE, should have gone through the persistent connections > AFAIK, but may have been caught up in the same logics as POST. > > Please also check whether this still occurs with the latest daily > bundle before reporting. 3.1.11 is a few months old now and we have had > a focus on performance fixes this year. > > Amos >