Thanks for that Alex.
I do have another related question regarding Squid 206 handling.
I have a RESPMOD icap server that supports Allow:206 in the OPTIONS response.
I can see that for html webpages, squid sends Allow: 204, 206, trailers in RESPMOD requests, and my server can handle that fine.
But for files, specifically for zip files, squid sends Allow: trailers , showing no support for 206. Even if I reply with a 206, I get TCP reset and connection is closed.
Do you know why squid doesn't Allow 206 for files?
============================
Here is my configured OPTIONS response.
ICAP/1.0 200 OK
Methods: RESPMOD
Service: PyICAP Server 1.0
Preview: 0
Transfer-Complete: *
Max-Connections: 100
Options-TTL: 3600
Allow: 206
ISTag: "CmLYfciETxyLwZW3XYsPwJdCXvVehd"
Date: b'Mon', 16 b'Sep' 2019 20:50:52 GMT
Server: BaseICAP/1.0 Python/3.7.3
Methods: RESPMOD
Service: PyICAP Server 1.0
Preview: 0
Transfer-Complete: *
Max-Connections: 100
Options-TTL: 3600
Allow: 206
ISTag: "CmLYfciETxyLwZW3XYsPwJdCXvVehd"
Date: b'Mon', 16 b'Sep' 2019 20:50:52 GMT
Server: BaseICAP/1.0 Python/3.7.3
==================================================
And here is the RESPMOD request.
RESPMOD icap://127.0.0.1:13440/example ICAP/1.0
Host: 127.0.0.1:13440
Date: Mon, 16 Sep 2019 20:51:00 GMT
Encapsulated: req-hdr=0, res-hdr=337, res-body=579
Allow: trailers
X-Client-IP: 192.168.0.6
GET http://203.0.113.1/file.zip HTTP/1.1
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:69.0) Gecko/20100101 Firefox/69.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
DNT: 1
Upgrade-Insecure-Requests: 1
Host: 203.0.113.1
HTTP/1.1 200 OK
Date: Mon, 16 Sep 2019 20:51:00 GMT
Server: Apache/2.4.6 (CentOS)
Last-Modified: Thu, 29 Aug 2019 13:54:58 GMT
ETag: "989680-59141d9f0557a"
Accept-Ranges: bytes
Content-Length: 10000000
Content-Type: application/zip
Host: 127.0.0.1:13440
Date: Mon, 16 Sep 2019 20:51:00 GMT
Encapsulated: req-hdr=0, res-hdr=337, res-body=579
Allow: trailers
X-Client-IP: 192.168.0.6
GET http://203.0.113.1/file.zip HTTP/1.1
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:69.0) Gecko/20100101 Firefox/69.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
DNT: 1
Upgrade-Insecure-Requests: 1
Host: 203.0.113.1
HTTP/1.1 200 OK
Date: Mon, 16 Sep 2019 20:51:00 GMT
Server: Apache/2.4.6 (CentOS)
Last-Modified: Thu, 29 Aug 2019 13:54:58 GMT
ETag: "989680-59141d9f0557a"
Accept-Ranges: bytes
Content-Length: 10000000
Content-Type: application/zip
On Mon, Sep 16, 2019 at 3:54 PM Alex Rousskov <rousskov@xxxxxxxxxxxxxxxxxxxxxxx> wrote:
On 9/16/19 2:58 PM, Felipe Arturo Polanco wrote:
> In our case we don't need to modify the initial 10MB, just scan it for
> virus and if found, send a reset back to squid to not transmit the file.
Yes, my original response was based on that assumption.
In summary, you can ask Squid to own the 10MB prefix (i.e. do a huge
ICAP Preview) or you can ask the ICAP server to own the 10MB prefix
(i.e. extend ICAP 206). The choice balances various trade-offs,
including different overheads and different risks related to Squid
development associated with each option. Pick your poison :-).
Alex.
> On Mon, Sep 16, 2019 at 11:54 AM Alex Rousskov wrote:
>
> On 9/16/19 10:37 AM, Felipe Arturo Polanco wrote:
>
> > We would like to add some logic to our custom made ICAP server, one of
> > these logics is to analyze up to 10MB of data of a given file and
> if the
> > file is larger than that, squid should not keep sending it to icap,
> > basically, a 204 message should be returned.
>
> > squid has a cap of 64K of preview size that limit us in the aspect.
>
> True.
>
>
> > Is there a way to extend this limit? Modifying the source code
> perhaps?
>
> Yes, of course.
>
>
> > If so, what are the disadvantages of doing so?
>
> Simply increasing BodyPipe::MaxCapacity might open you up to
> difficult-to-find vulnerabilities where some old Squid code still
> assumes that the buffers are limited by 64KB and crashes/asserts when
> that assumption becomes false. FWIW, I have not audited all
> BodyPipe-related code and do not know whether such bad code exists
> today.
>
> Increasing BodyPipe::MaxCapacity will also increase Squid RAM
> requirements if your Squid receives requests (and/or ICAP
> REQMOD-generated HTTP responses) that exceed 64KB body limit. I am not
> sure about regular responses -- their accumulation is limited by
> other/independent restrictions.
>
>
> > Would it require more RAM per response
>
> [You should worry about "per message" accumulation -- requests will use
> the same limit, even if you do not have REQMOD services.]
>
> Yes, but only for messages that exceed the limit.
>
>
> > or is RAM dynamically allocated as the file is being received?
>
> Yes, the RAM in question is allocated dynamically on "as needed" basis.
>
> YMMV, but Squid may slow down a lot while dealing with the associated
> reallocations because of the too-small 64KB increment used for each such
> reallocation.
>
>
> An alternative solution to consider here is extending ICAP features to
> allow for splicing of the 10MB prefix (analyzed _and_ returned back by
> the ICAP server) and the remaining virgin response suffix (paused
> without excessive accumulation on the Squid side). This would be similar
> to the existing ICAP 206 use-original-body="10MB" feature[1], but would
> require additional negotiation and changes to disable storage of the
> 10MB prefix on the Squid side.
>
> This alternative can be implemented without worrying about hidden 64KB
> body buffer assumptions, but the implementation will not be trivial.
>
> [1]
> http://www.icap-forum.org/documents/specification/draft-icap-ext-partial-content-07.txt
>
>
> HTH,
>
> Alex.
>
_______________________________________________ squid-users mailing list squid-users@xxxxxxxxxxxxxxxxxxxxx http://lists.squid-cache.org/listinfo/squid-users
