On 3/6/21 5:38 PM, Niels Hofmans wrote: > Missed that in the RFC, I was hoping I could only send the headers across and modify those, without having to send over everything, including very large responses for performance reasons. There are ICAP extensions that allow header-only adaptation, but, IMO, it is best to get the basics working before adding support for experimental protocol extensions. Alex. > On 6 Mar 2021, at 23:22, Alex Rousskov <rousskov@xxxxxxxxxxxxxxxxxxxxxxx> wrote: > > On 3/6/21 3:33 PM, Niels Hofmans wrote: > >> I fixed a bug in the go-icap/icap library, seen here: > > FYI: Whatever you were trying to share in this paragraph, did not come > through the mailing list. However, we should not be discussing some ICAP > library bugs on this mailing list, so perhaps it is not important. > > >> But this made me wonder about the following: <PastedGraphic-1.tiff> > > In the future, please paste message _text_ and/or provide frame/stream > references to pieces of shared captures. Working with images is very > inefficient when it comes to protocol analysis. > > >> Here you can see squid issuing a RESPMOD to my ICAP server with a >> Preview of “0”, > > Yes. > > >> but it does contain a HTTP response. > > It contains an HTTP request header (shown in the PastedGraphic-1.tiff > image you have attached), probably an HTTP response header (not shown), > and probably no HTTP response body. The HTTP response does have a body, > but the ICAP client does not send it during the Preview:0 stage. This is > how ICAP Preview works. I see nothing wrong here. > > >> And since my ICAP service will not return a HTTP response since the >> Preview size is 0, my http client receives nothing from squid. > > AFAICT, your ICAP service is butchering HTTP responses by responding > with ICAP 200 responses that encapsulate HTTP messages without bodies. A > standard ICAP client will _not_ re-assemble the HTTP message by > combining your header with client-cached body parts. The ICAP standard > just does not work that way. When the ICAP service sends an ICAP 200 > response, the service must return the _entire_ HTTP message, headers and > body. > > As a consequence, an ICAP service cannot use standard ICAP to adapt HTTP > headers without receiving and returning the HTTP body as well. This > applies to REQMOD and RESPMOD transactions. You are just lucky that your > test REQMOD transactions have requests without bodies. > > >> I checked but I don’t think I see a 100 continuation from ICAP in the >> dump anymore. > > Yes, that is correct. However, the new trace shows another ICAP service > problem as detailed above. > > > HTH, > > Alex. > > >> On 6 Mar 2021, at 19:29, Alex Rousskov wrote: >> >> On 3/6/21 5:09 AM, Niels Hofmans wrote: >> >>> I’ve uploaded a tcpdump sample of where I’m requesting an image through >>> squid + icap and where you can see the image response being posted fully >>> to the ICAP service. >> >> AFAICT, all ICAP REQMOD and RESPMOD requests in your packet capture send >> zero-size Preview containing just the HTTP headers (e.g., see frame >> #44). In one case, Squid also sends the HTTP response body but only >> because your ICAP service explicitly requests that HTTP response body by >> responding to Squid Preview with ICAP 100 Continue control message >> (e.g., see frame #48)! >> >> If your ICAP service does not want to see an HTTP body, then it should >> not ask for it. It should respond (usually with ICAP 200 or ICAP 204) >> based on the Preview alone, without sending ICAP 100 Continue control >> message first. >> >> >> HTH, >> >> Alex. >> >> >>> On 5 Mar 2021, at 23:32, Alex Rousskov wrote: >>> On 3/5/21 5:21 PM, Niels Hofmans wrote: >>>> I receive that large payload right after an OPTIONS call to my ICAP >>>> endpoint. It is the preview. >>> The timing of an ICAP request does not determine whether that request >>> has a Preview stage and whether the "unexpected" body bytes were sent >>> during that Preview stage. I am asking these questions because I want to >>> determine whether Squid increases Preview size beyond what your server >>> has requested or does not send Preview at all. >>> Please share the ICAP headers of the problematic REQMOD request and the >>> last chunk metadata of that request body. You can get the latter from a >>> packet capture if your ICAP server does not report it in a convenient >>> form. In fact, sharing (a pointer to) the packet capture of the whole >>> problematic ICAP request is probably a good idea! >>> Alex. >>>> On 5 Mar 2021, at 17:21, Alex Rousskov wrote: >>>> On 3/5/21 2:55 AM, Niels Hofmans wrote: >>>>> One more: I believe ICAP is not respecting the Preview header for REQMOD >>>>> nor RESPMOD. >>>>> For the REQMOD OPTIONS requests, I respond with: >>>>> ICAP/1.0 200 OK >>>>> Allow: 200,204 >>>>> Connection: close >>>>> Date: Fri, 05 Mar 2021 07:34:56 GMT >>>>> Encapsulated: null-body=0 >>>>> Methods: REQMOD >>>>> Preview: 64000 >>>>> However in my ICAP service, I often receive a request body over 64000 >>>>> bytes, e.g. 1000000 request bytes. >>>> Receive when? During preview, instead of Preview, and/or after Preview? >>>>> I also noticed that a "Preview: 0” will not have any effect and the ICAP >>>>> service will still receive a complete HTTP REQMOD+REQRESP body. >>>> Receive when? During preview, instead of Preview, and/or after Preview? >>>> Alex. >>>>> icap_enable on >>>>> icap_connect_timeout 1 second >>>>> icap_service_failure_limit -1 >>>>> icap_send_client_ip on >>>>> icap_preview_size 0 >>>>> icap_persistent_connections on >>>>> icap_service service_req reqmod_precache bypass=0 >>>>> icap://172.17.0.2:1344/request >>>>> <icap://172.17.0.2:1344/request> <icap://172.17.0.2:1344/request >>>>> <icap://172.17.0.2:1344/request>> >>>>> <icap://172.17.0.2:1344/request >>>>> <icap://172.17.0.2:1344/request> <icap://172.17.0.2:1344/request >>>>> <icap://172.17.0.2:1344/request>>> >>>>> icap_service service_res respmod_precache bypass=0 >>>>> icap://172.17.0.2:1344/response >>>>> <icap://172.17.0.2:1344/response> <icap://172.17.0.2:1344/response >>>>> <icap://172.17.0.2:1344/response>> >>>>> <icap://172.17.0.2:1344/response >>>>> <icap://172.17.0.2:1344/response> <icap://172.17.0.2:1344/response >>>>> <icap://172.17.0.2:1344/response>>> >>>>> icap_preview_enable on >>>>> adaptation_access service_req allow all >>>>> adaptation_access service_res allow all _______________________________________________ squid-users mailing list squid-users@xxxxxxxxxxxxxxxxxxxxx http://lists.squid-cache.org/listinfo/squid-users
