Hi All,
I have two things to clarify:
1) In earlier email (snipped below), Amos told that is caching and scheduled to download - does it mean that we got the answer and do some override?
-------------------- ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
> This is what I see when the download is in progress:
>
> KEY 44000000000000000902000000000000
> STORE_PENDING NOT_IN_MEMORY SWAPOUT_NONE PING_DONE
> RELEASE_REQUEST,DISPATCHED,PRIVATE,VALIDATED
So file stored in memory and scheduled for removal.
>
> KEY 44000000000000000902000000000000
> STORE_PENDING NOT_IN_MEMORY SWAPOUT_NONE PING_DONE
> RELEASE_REQUEST,DISPATCHED,PRIVATE,VALIDATED
So file stored in memory and scheduled for removal.
-------------------- ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
2) With more debug_options enabled, I see that it is not caching because the response is part of authenticated flow. Is there a way I can override this?
- Initially when this file download starts, it gets authorized (I think that is why I see HEAD request is sent to target which has Cache-Control: max-age=0) and then the subsequent GET requests (dont have any cache-control header) download the chunk using adjusted range and offset.
- But the subsequent GET reply seems set with auth flag, see the code below snipped from source (https://github.com/squid-cache/squid/blob/4f1c93a7a0d14eec223e199275ce570d840f71bc/src/http.cc).
// RFC 2068, sec 14.9.4 - MUST NOT cache any response with Authentication UNLESS certain CC controls are present
// allow HTTP violations to IGNORE those controls (ie re-block caching Auth)
if (request && (request->flags.auth || request->flags.authSent)) {
if (!rep->cache_control)
return decision.make(ReuseDecision::reuseNot,
"authenticated and server reply missing Cache-Control");
- I tried adding override-expire in the cgi-bin refresh pattern, but that will override only for max-age in Cache-control but not relevant for auth flag.
Please find the logs below:
Please let me know if I am missing something.
Thanks,
Hari
Refresh pattern (URL:https://example.com/pcgi-bin/swdld/download.cgi?dwnld_code=xhM...):
refresh_pattern ^ftp: 1440 20% 10080
refresh_pattern ^gopher: 1440 0% 1440
refresh_pattern -i (/cgi-bin/|\?) 0 0% 0
refresh_pattern . 0 20% 4320
Logs:
2018/09/10 09:57:24.864| 11,2| http.cc(723) processReplyHeader: HTTP Server RESPONSE:
---------
HTTP/1.1 206 Partial Content
Date: Mon, 10 Sep 2018 09:57:20 GMT^M
Server: Apache/2.2^M
Content-Disposition: attachment; filename=somefile.iso;^M
Last-Modified: Thu, 22 Mar 2018 02:11:11 GMT^M
ETag: "4ad8a61-1193b000-567f6d2466dc0"^M
Accept-Ranges: bytes^M
Content-Length: 294342471^M
Content-Range: bytes 549049-294891519/294891520^M
Keep-Alive: timeout=5, max=100^M
Connection: Keep-Alive^M
Content-Type: application/unknown^M
^M
----------
2018/09/10 09:57:24.864| 11,5| Client.cc(134) setVirginReply: 0x5560bf163d78 setting virgin reply to 0x5560bf162c20
2018/09/10 09:57:24.864| ctx: exit level 0
2018/09/10 09:57:24.864| 83,3| AccessCheck.cc(42) Start: adaptation off, skipping
2018/09/10 09:57:24.864| 11,5| Client.cc(969) adaptOrFinalizeReply: adaptationAccessCheckPending=0
2018/09/10 09:57:24.864| 11,5| Client.cc(152) setFinalReply: 0x5560bf163d78 setting final reply to 0x5560bf162c20
2018/09/10 09:57:24.864| 20,3| store.cc(1807) replaceHttpReply: StoreEntry::replaceHttpReply: https://example.com/pcgi-bin/swdld/download.cgi?dwnld_code=xhMnkw8Z-oECuFusb12luTTCm0rP8jZiRFu8gsXRtc
2018/09/10 09:57:24.864| ctx: enter level 0: 'https://example.com/pcgi-bin/swdld/download.cgi?dwnld_code=xhMnkw8Z-oECuFusb12luTTCm0rP8jZiRFu8gsXRtcoacGcTu6dv-dkLcT4lqtgvM70n8-ucJsj09lRYt_a0t7_M5
2018/09/10 09:57:24.864| 11,3| http.cc(907) haveParsedReplyHeaders: HTTP CODE: 206
2018/09/10 09:57:24.864| 73,3| HttpRequest.cc(664) storeId: sent back effectiveRequestUrl: https://dl.cisco.com/pcgi-bin/swdld/download.cgi?dwnld_code=xhMnkw8Z-oECuFusb12luTTCm0rP8jZiRFu8gsXRtcoac
2018/09/10 09:57:24.864| 20,3| Controller.cc(386) peek: 76E544615E001DBF49EF0F94EE0A8F9A
2018/09/10 09:57:24.865| 20,4| Controller.cc(420) peek: cannot locate 76E544615E001DBF49EF0F94EE0A8F9A
2018/09/10 09:57:24.865| 20,3| store.cc(450) releaseRequest: 0 e:=p2IV/0x5560bf16ac60*3
2018/09/10 09:57:24.865| 20,3| store.cc(580) setPrivateKey: 01 e:=p2IV/0x5560bf16ac60*3
2018/09/10 09:57:24.865| 11,3| http.cc(982) haveParsedReplyHeaders: decided: do not cache and do not share because authenticated and server reply missing Cache-Control; HTTP status 206 e:=p2XIV/0x
2018/09/10 09:57:24.865| ctx: exit level 0
On Sat, Sep 8, 2018 at 8:13 PM Hariharan Sethuraman <srnhari@xxxxxxxxx> wrote:
But the partial data is been continuously sending back to client. Squid didn't wait for the complete file to downloadOn Sat, 8 Sep 2018, 20:08 Amos Jeffries, <squid3@xxxxxxxxxxxxx> wrote:On 8/09/18 7:44 PM, Hariharan Sethuraman wrote:
> And here is my HTTP request and reply:
> 1. HEAD request to read image information
> 2. GET request to download the image
>
Not quite. GET partial / Range request to fetch the content.
Squid converted it into a full request for the backend server due to
range_offset_limit -1. But that does mean Squid had to download ~240MB
of data before anything starts being sent to the client.
> (2)
> 2018/09/08 07:28:39.155| 11,2| client_side.cc(1278) parseHttpRequest:
> HTTP Client REQUEST:
> ---------
> GET /DcKbz9kqMQXK-zp95pv9LH11kjhTpxOJsJ-1FYEL4
> Host: example.com:3129 <http://example.com:3129>^M
> Range: bytes=242819145-^M
> User-Agent: curl/7.56.1^M
> Accept: */*^M
>
> 2018/09/08 07:28:39.938| 11,2| http.cc(2261) sendRequest: HTTP Server
> REQUEST:
> ---------
> GET /DcKbz9kqMQXK-zp95pv9LH11kjhTpxOJsJ-1FYEL4
> User-Agent: curl/7.56.1^M
> Accept: */*^M
> Host: exampletarget.com <http://exampletarget.com>^M
> Via: 1.1 jb7mgd (squid/4.2)^M
> Surrogate-Capability: jb7mgd="Surrogate/1.0"^M
> X-Forwarded-For: **.**.**.**^M
> Cache-Control: max-age=0^M
> Connection: keep-alive^M
>
> 2018/09/08 07:28:44.359| 11,2| http.cc(723) processReplyHeader: HTTP
> Server RESPONSE:
> ---------
> HTTP/1.1 200 OK
> Date: Sat, 08 Sep 2018 07:28:40 GMT^M
> Server: Apache/2.2^M
> Content-Disposition: attachment; filename=somefile.iso;^M
> Last-Modified: Fri, 20 Jul 2018 18:23:10 GMT^M
> ETag: "4a54c59-11653800-571726350bf80"^M
> Accept-Ranges: bytes^M
> Content-Length: 291846144^M
> Keep-Alive: timeout=5, max=100^M
> Connection: Keep-Alive^M
> Content-Type: application/unknown^M
>
> 2018/09/08 07:28:44.361| 11,2| Stream.cc(267) sendStartOfMessage: HTTP
> Client REPLY:
> ---------
> HTTP/1.1 206 Partial Content^M
> Date: Sat, 08 Sep 2018 07:28:40 GMT^M
> Server: Apache/2.2^M
> Content-Disposition: attachment; filename=somefile.iso;^M
> Last-Modified: Fri, 20 Jul 2018 18:23:10 GMT^M
> ETag: "4a54c59-11653800-571726350bf80"^M
> Accept-Ranges: bytes^M
> Content-Type: application/unknown^M
> X-Cache: MISS from jb7mgd^M
> X-Cache-Lookup: MISS from jb7mgd:3128^M
> Via: 1.1 jb7mgd (squid/4.2)^M
> Connection: keep-alive^M
> Content-Range: bytes 242819145-291846143/291846144^M
> Content-Length: 49026999^M
>
> Thanks,
> Hari
>
> On Sat, Sep 8, 2018 at 9:48 AM Hariharan Sethuraman wrote:
>
> Hi Amos,
>
> This is what I see when the download is in progress:
>
> KEY 44000000000000000902000000000000
> STORE_PENDING NOT_IN_MEMORY SWAPOUT_NONE PING_DONE
> RELEASE_REQUEST,DISPATCHED,PRIVATE,VALIDATED
So file stored in memory and scheduled for removal.
> LV:1536379799 LU:1536379801 LM:1532110990 EX:-1
> 4 locks, 1 clients, 1 refs
> Swap Dir -1, File 0XFFFFFFFF
> GET
> https://example.com/DhAskLOUpvRG2oeR_f_FxYTyLVHIN5esRF-LXOUKwkwyT0TOf6xO-AUm3KaM
> inmem_lo: 99225582
> inmem_hi: 99324372
> swapout: 0 bytes queued
>
Amos
_______________________________________________ squid-users mailing list squid-users@xxxxxxxxxxxxxxxxxxxxx http://lists.squid-cache.org/listinfo/squid-users
