On 11/01/2013 7:11 a.m., Steve Hill wrote:
On 09/01/13 21:07, Amos Jeffries wrote:
Does the CONNECT request contain Connection:close or
Connection:keep-alive? Squid supports keep-alive on CONNECT requests in
these situations where the CONNECT size is known and may be waiting for
another client request.
The client sends "Proxy-Connection: keep-alive", so it would indeed be
possible for the connection to be reused. I should add that I'm
seeing this with a non-transparent configuration (i.e. without Tproxy
and without ssl_bump).
Please upgrade to 3.2.6 (should anyway for the CVE resolved there) and
see if this issue is gone there.
I've now upgraded my test server and the problem remains.
The "403 Forbidden" being sent back to the client didn't have a
Content Length header, which of course caused the client to drop the
connection. If I add a Content Length, the connection stays alive and
I think it's a bit more obvious whats happening:
1. Client connects to Squid and sends a CONNECT.
2. Squid passes the request to the ICAP REQMOD service.
3. ICAP rewrites the request.
4. Squid returns a "403 Forbidden" to the client in the clear. At
this point, the connection is still alive with empty queues. Squid's
cache.log shows:
2013/01/10 17:52:18 kid1| abandoning local=[2a00:1a90:5::9]:3128
remote=[2001:4d48:ad51:501:226:bbff:fe18:f3ff]:49926 FD 19 flags=1
Now, the user retries the connection:
5. Client sends CONNECT over the existing connection.
6. Squid does not read the socket (netstat shows the rx queue on the
socket is 220 octets long, which is the whole request the client just
sent).
7. The client sits there spinning.
8. Eventually the client times out and drops the connection.
The connection is now in CLOSE_WAIT on the Squid server and will
remain like that indefinitely.
So it seems apparent that after Squid delivers the clear-text
response, it abandons the socket but never closes it. From looking in
the source, this is client_side.cc, and it has a comment:
// XXX: Can this happen? CONNECT tunnels have deferredRequest set.
It looks to me as if the (conn->flags.readMore) section above should
be the bit being executed, although I don't quite understand deferred
requests. In either case, it seems like we should close the socket if
it ever gets abandoned?
A few notes on the REQMOD rewrite:
My ICAP service can generate a "403 Forbidden" response during REQMOD,
or rewrite the request to a GET to a local web server. In the latter
case, the local webserver generates a "403 Forbidden" response. In
both cases the client sees a similar response to its request, and in
both cases this bug manifests.
Interestingly, if the ICAP service closes the ICAP connection instead
of returning a response, Squid generates a "500 Internal Server Error"
and does not abandon the socket (the client then drops the connection,
which squid handles correctly, and therefore doesn't end in CLOSE_WAIT).
Okay. So the source of the problem is that Squid thinks there is
something using the socket, but it never got into the tunnel code which
would have closed it?
Is the 403 message being generated inside Squid or by ICAP?
In the case where the server generates the 403 and things hang it will
be because Squid is expecting a close to arrive from server or client.
Once a tunnel is open and transmitting data it is up to those endpoints
to ensure keep-alive and other such details, although Squid applies a
timeout since last byte transferred.
Amos
