Hi, thanks so much for the detailed response. I chose to test option 2 from your recommendations as I am new to squid and I do not understand how to set it up as a reverse proxy anyway. I made the change to my squid.conf :
#ssl_bump peek step1
ssl_bump bump step1
ssl_bump bump all
This made it work - which is great news. My curl requests now are satisfied by the cache when the pc is offline!
I do have 1 followup question which I think is unrelated, let me know if etiquette demands I create a new post for this. When I test using chromium browser, chromium sends OPTION requests - which I think is something to do with CORS. These always cause cache MISS from squid,.. I think because the return code is 204...?
1705669236.776 113 ::1 TCP_MISS/204 680 OPTIONS https://stuff.amazonaws.com/api/v1/stuff/stuff.json - HIER_DIRECT/3.135.146.17 application/json
I can prevent my chromium instance from making these (pointless?) OPTIONS calls using the following args, but I would rather not have to do this.
--disable-web-security --disable-features=IsolateOrigins,site-per-process
Any way I can get squid to cache these calls?
Thanks again and all the best,
Robin
On Thu, 18 Jan 2024 at 16:03, Alex Rousskov <rousskov@xxxxxxxxxxxxxxxxxxxxxxx> wrote:
On 2024-01-18 09:53, Robin Carlisle wrote:
> My expectation/hope is that squid would return the cached object on
> any network failure in between ubuntu-pc and the AWS endpoint - and
> continue to return this cached object forever. Is this something
> squid can do? It would seem that offline_mode should do this?
Yes and yes. The reason you are getting errors are not related to cache
hits or misses. Those errors happen _before_ Squid gets the requested
resource URL and looks up that resource in Squid cache.
> ssl_bump peek step1
> ssl_bump bump all
To get that URL (in your configuration), Squid must bump the connection.
To bump the connection at step2, Squid must contact the origin server.
When the cable is unplugged, Squid obviously cannot do that: The attempt
to open a Squid-AWS connection fails.
> .../200 0 CONNECT stuff.amazonaws.com:443 - HIER_DIRECT
> .../503 4087 GET https://stuff.amazonaws.com/api/... - HIER_NONE
Squid reports bumping errors to the client using HTTP responses. To do
that, Squid remembers the error response, bumps the client connection,
receives GET from the client on that bumped connection, and sends that
error response to the client. This is why you see both CONNECT/200 and
GET/503 access.log records. Note that Squid does not check whether the
received GET request would have been a cache hit in this case -- the
response to that request has been preordained by the earlier bumping
failure.
Solution candidates to consider include:
* Stop bumping: https_port 443 cert=/etc/squid/stuff.pem
Configure Squid as (a reverse HTTPS proxy for) the AWS service. Use
https_port. No SslBump rules/options! The client would think that it is
sending HTTPS requests directly to the service. Squid will forward
client requests to the service. If this works (and I do not have enough
information to know that this will work in your specific environment),
then you will get a much simpler setup.
* Bump at step1, before Squid contacts AWS: ssl_bump bump all
Bugs notwithstanding, there will be no Squid-AWS connection for cache
hits. The resulting certificate will not be based on AWS service info,
but it looks like your client is ignorant enough to ignore related
certificate problems.
HTH,
Alex.
> Hi, Hoping someone can help me with this issue that I have been
> struggling with for days now. I am setting up squid on an ubuntu PC to
> forward HTTPS requests to an API and an s3 bucket under my control on
> amazon AWS. The reason I am setting up the proxy is two-fold...
>
> 1) To reduce costs from AWS.
> 2) To provide content to the client on the ubuntu PC if there is a
> networking issue somewhere in between the ubuntu PC and AWS.
>
> Item 1 is going well so far. Item 2 is not going well. Setup details ...
>
> *# squid - setup cache folder*
> mkdir -p /var/cache/squid
> chown -R proxy:proxy /var/cache/squid
>
> *# ssl - generate key*
> apt --yes install squid-openssl libnss3-tools
> openssl req -new -newkey rsa:2048 -days 365 -nodes -x509 \
> -subj "/C=US/ST=Denial/L=Springfield/O=Dis/CN=www.example.com
> <http://www.example.com>" \
> -keyout /etc/squid/stuff.pem -out /etc/squid/stuff.pem
> chown root:proxy /etc/squid/stuff.pem
> chmod 644 /etc/squid/stuff.pem
>
> *# ssl - ssl DB*
> mkdir -p /var/lib/squid
> rm -rf /var/lib/squid/ssl_db
> /usr/lib/squid/security_file_certgen -c -s /var/lib/squid/ssl_db -M 4MB
> chown -R proxy:proxy /var/lib/squid/ssl_db
>
> *# /etc/squid/squid.conf :*
> acl to_aws dstdomain .amazonaws.com <http://amazonaws.com>
> acl from_local src localhost
> http_access allow to_aws
> http_access allow from_local
> cache allow all
> cache_dir ufs /var/cache/squid 1024 16 256
> offline_mode on
> http_port 3129 ssl-bump cert=/etc/squid/stuff.pem
> generate-host-certificates=on dynamic_cert_mem_cache_size=4MB
> sslcrtd_program /usr/lib/squid/security_file_certgen -s
> /var/lib/squid/ssl_db -M 4MB
> acl step1 at_step SslBump1
> ssl_bump peek step1
> ssl_bump bump all
> sslproxy_cert_error deny all
> cache_store_log stdio:/var/log/squid/store.log
> logfile_rotate 0
>
> *# /usr/bin/proxy-test :*
> #!/bin/bash
> curl --proxy http://localhost:3129 <http://localhost:3129> \
> --cacert /etc/squid/stuff.pem \
> -v "https://stuff.amazonaws.com/api/v1/stuff/stuff.json
> <https://stuff.amazonaws.com/api/v1/stuff/stuff.json>" \
> -H "Authorization: token MYTOKEN" \
> -H "Content-Type: application/json" \
> --output "/tmp/stuff.json"
>
>
>
> When network connectivity is GOOD, everything works well and I get cache
> HITS ...
>
> *# /var/log/squid/access.log*
> 1705587538.837 238 127.0.0.1 NONE_NONE/200 0 CONNECT
> stuff.amazonaws.com:443 <http://stuff.amazonaws.com:443> -
> HIER_DIRECT/3.136.246.238 <http://3.136.246.238> -
> 1705587538.838 0 127.0.0.1 TCP_MEM_HIT/200 32818 GET
> https://stuff.amazonaws.com/api/v1/stuff/stuff.json
> <https://stuff.amazonaws.com/api/v1/stuff/stuff.json> - HIER_NONE/-
> application/json
>
> *# extract from /usr/bin/proxy-test output*
> < HTTP/1.1 200 OK
> < Date: Thu, 18 Jan 2024 13:38:01 GMT
> < Content-Type: application/json
> < Content-Length: 32187
> < x-amzn-RequestId: 8afba80e-6df7-4d5b-a34b-a70bd9b54380
> < Last-Modified: 2024-01-03T11:23:19.000Z
> < Access-Control-Allow-Origin: *
> < x-amz-apigw-id: RvN1CF2_iYcEokA=
> < Cache-Control: max-age=2147483648,public,stale-if-error
> < ETag: "53896156c4e8e26933188a092c4e40f1"
> < X-Amzn-Trace-Id: Root=1-65a929b9-3bd3285934151c1a2495481a
> < Age: 2578
> < Warning: 110 squid/5.7 "Response is stale"
> < X-Cache: HIT from ubuntu-pc
> < X-Cache-Lookup: HIT from ubuntu-pc:3129
> < Via: 1.1 ubuntu-pc (squid/5.7)
> < Connection: keep-alive
>
>
> When network connectivity is BAD, I get errors and a cache MISS. In
> this test case I unplugged the ethernet cable from the back on the
> ubuntu-pc ...
>
> *# /var/log/squid/access.log*
> 1705588717.420 11 127.0.0.1 NONE_NONE/200 0 CONNECT
> stuff.amazonaws.com:443 <http://stuff.amazonaws.com:443> -
> HIER_DIRECT/3.135.162.228 <http://3.135.162.228> -
> 1705588717.420 0 127.0.0.1 NONE_NONE/503 4087 GET
> https://stuff.amazonaws.com/api/v1/stuff/stuff.json
> <https://stuff.amazonaws.com/api/v1/stuff/stuff.json> - HIER_NONE/-
> text/html
>
> *# extract from /usr/bin/proxy-test output*
> < HTTP/1.1 503 Service Unavailable
> < Server: squid/5.7
> < Mime-Version: 1.0
> < Date: Thu, 18 Jan 2024 14:38:37 GMT
> < Content-Type: text/html;charset=utf-8
> < Content-Length: 3692
> < X-Squid-Error: ERR_CONNECT_FAIL 101
> < Vary: Accept-Language
> < Content-Language: en
> < X-Cache: MISS from ubuntu-pc
> < X-Cache-Lookup: NONE from ubuntu-pc:3129
> < Via: 1.1 ubuntu-pc (squid/5.7)
> < Connection: close
>
> I have also seen it error in a different way with a 502 but with the
> same ultimate result.
>
> My expectation/hope is that squid would return the cached object on any
> network failure in between ubuntu-pc and the AWS endpoint - and continue
> to return this cached object forever. Is this something squid can do?
> It would seem that offline_mode should do this?
>
> Hope you can help,
>
> Robin
>
>
>
> _______________________________________________
> squid-users mailing list
> squid-users@xxxxxxxxxxxxxxxxxxxxx
> https://lists.squid-cache.org/listinfo/squid-users
_______________________________________________ squid-users mailing list squid-users@xxxxxxxxxxxxxxxxxxxxx https://lists.squid-cache.org/listinfo/squid-users
