On 2024-01-18 09:53, Robin Carlisle wrote:
My expectation/hope is that squid would return the cached object on
any network failure in between ubuntu-pc and the AWS endpoint - and
continue to return this cached object forever. Is this something
squid can do? It would seem that offline_mode should do this?
Yes and yes. The reason you are getting errors are not related to cache
hits or misses. Those errors happen _before_ Squid gets the requested
resource URL and looks up that resource in Squid cache.
ssl_bump peek step1
ssl_bump bump all
To get that URL (in your configuration), Squid must bump the connection.
To bump the connection at step2, Squid must contact the origin server.
When the cable is unplugged, Squid obviously cannot do that: The attempt
to open a Squid-AWS connection fails.
> .../200 0 CONNECT stuff.amazonaws.com:443 - HIER_DIRECT
> .../503 4087 GET https://stuff.amazonaws.com/api/... - HIER_NONE
Squid reports bumping errors to the client using HTTP responses. To do
that, Squid remembers the error response, bumps the client connection,
receives GET from the client on that bumped connection, and sends that
error response to the client. This is why you see both CONNECT/200 and
GET/503 access.log records. Note that Squid does not check whether the
received GET request would have been a cache hit in this case -- the
response to that request has been preordained by the earlier bumping
failure.
Solution candidates to consider include:
* Stop bumping: https_port 443 cert=/etc/squid/stuff.pem
Configure Squid as (a reverse HTTPS proxy for) the AWS service. Use
https_port. No SslBump rules/options! The client would think that it is
sending HTTPS requests directly to the service. Squid will forward
client requests to the service. If this works (and I do not have enough
information to know that this will work in your specific environment),
then you will get a much simpler setup.
* Bump at step1, before Squid contacts AWS: ssl_bump bump all
Bugs notwithstanding, there will be no Squid-AWS connection for cache
hits. The resulting certificate will not be based on AWS service info,
but it looks like your client is ignorant enough to ignore related
certificate problems.
HTH,
Alex.
Hi, Hoping someone can help me with this issue that I have been
struggling with for days now. I am setting up squid on an ubuntu PC to
forward HTTPS requests to an API and an s3 bucket under my control on
amazon AWS. The reason I am setting up the proxy is two-fold...
1) To reduce costs from AWS.
2) To provide content to the client on the ubuntu PC if there is a
networking issue somewhere in between the ubuntu PC and AWS.
Item 1 is going well so far. Item 2 is not going well. Setup details ...
*# squid - setup cache folder*
mkdir -p /var/cache/squid
chown -R proxy:proxy /var/cache/squid
*# ssl - generate key*
apt --yes install squid-openssl libnss3-tools
openssl req -new -newkey rsa:2048 -days 365 -nodes -x509 \
-subj "/C=US/ST=Denial/L=Springfield/O=Dis/CN=www.example.com
<http://www.example.com>" \
-keyout /etc/squid/stuff.pem -out /etc/squid/stuff.pem
chown root:proxy /etc/squid/stuff.pem
chmod 644 /etc/squid/stuff.pem
*# ssl - ssl DB*
mkdir -p /var/lib/squid
rm -rf /var/lib/squid/ssl_db
/usr/lib/squid/security_file_certgen -c -s /var/lib/squid/ssl_db -M 4MB
chown -R proxy:proxy /var/lib/squid/ssl_db
*# /etc/squid/squid.conf :*
acl to_aws dstdomain .amazonaws.com <http://amazonaws.com>
acl from_local src localhost
http_access allow to_aws
http_access allow from_local
cache allow all
cache_dir ufs /var/cache/squid 1024 16 256
offline_mode on
http_port 3129 ssl-bump cert=/etc/squid/stuff.pem
generate-host-certificates=on dynamic_cert_mem_cache_size=4MB
sslcrtd_program /usr/lib/squid/security_file_certgen -s
/var/lib/squid/ssl_db -M 4MB
acl step1 at_step SslBump1
ssl_bump peek step1
ssl_bump bump all
sslproxy_cert_error deny all
cache_store_log stdio:/var/log/squid/store.log
logfile_rotate 0
*# /usr/bin/proxy-test :*
#!/bin/bash
curl --proxy http://localhost:3129 <http://localhost:3129> \
--cacert /etc/squid/stuff.pem \
-v "https://stuff.amazonaws.com/api/v1/stuff/stuff.json
<https://stuff.amazonaws.com/api/v1/stuff/stuff.json>" \
-H "Authorization: token MYTOKEN" \
-H "Content-Type: application/json" \
--output "/tmp/stuff.json"
When network connectivity is GOOD, everything works well and I get cache
HITS ...
*# /var/log/squid/access.log*
1705587538.837 238 127.0.0.1 NONE_NONE/200 0 CONNECT
stuff.amazonaws.com:443 <http://stuff.amazonaws.com:443> -
HIER_DIRECT/3.136.246.238 <http://3.136.246.238> -
1705587538.838 0 127.0.0.1 TCP_MEM_HIT/200 32818 GET
https://stuff.amazonaws.com/api/v1/stuff/stuff.json
<https://stuff.amazonaws.com/api/v1/stuff/stuff.json> - HIER_NONE/-
application/json
*# extract from /usr/bin/proxy-test output*
< HTTP/1.1 200 OK
< Date: Thu, 18 Jan 2024 13:38:01 GMT
< Content-Type: application/json
< Content-Length: 32187
< x-amzn-RequestId: 8afba80e-6df7-4d5b-a34b-a70bd9b54380
< Last-Modified: 2024-01-03T11:23:19.000Z
< Access-Control-Allow-Origin: *
< x-amz-apigw-id: RvN1CF2_iYcEokA=
< Cache-Control: max-age=2147483648,public,stale-if-error
< ETag: "53896156c4e8e26933188a092c4e40f1"
< X-Amzn-Trace-Id: Root=1-65a929b9-3bd3285934151c1a2495481a
< Age: 2578
< Warning: 110 squid/5.7 "Response is stale"
< X-Cache: HIT from ubuntu-pc
< X-Cache-Lookup: HIT from ubuntu-pc:3129
< Via: 1.1 ubuntu-pc (squid/5.7)
< Connection: keep-alive
When network connectivity is BAD, I get errors and a cache MISS. In
this test case I unplugged the ethernet cable from the back on the
ubuntu-pc ...
*# /var/log/squid/access.log*
1705588717.420 11 127.0.0.1 NONE_NONE/200 0 CONNECT
stuff.amazonaws.com:443 <http://stuff.amazonaws.com:443> -
HIER_DIRECT/3.135.162.228 <http://3.135.162.228> -
1705588717.420 0 127.0.0.1 NONE_NONE/503 4087 GET
https://stuff.amazonaws.com/api/v1/stuff/stuff.json
<https://stuff.amazonaws.com/api/v1/stuff/stuff.json> - HIER_NONE/-
text/html
*# extract from /usr/bin/proxy-test output*
< HTTP/1.1 503 Service Unavailable
< Server: squid/5.7
< Mime-Version: 1.0
< Date: Thu, 18 Jan 2024 14:38:37 GMT
< Content-Type: text/html;charset=utf-8
< Content-Length: 3692
< X-Squid-Error: ERR_CONNECT_FAIL 101
< Vary: Accept-Language
< Content-Language: en
< X-Cache: MISS from ubuntu-pc
< X-Cache-Lookup: NONE from ubuntu-pc:3129
< Via: 1.1 ubuntu-pc (squid/5.7)
< Connection: close
I have also seen it error in a different way with a 502 but with the
same ultimate result.
My expectation/hope is that squid would return the cached object on any
network failure in between ubuntu-pc and the AWS endpoint - and continue
to return this cached object forever. Is this something squid can do?
It would seem that offline_mode should do this?
Hope you can help,
Robin
_______________________________________________
squid-users mailing list
squid-users@xxxxxxxxxxxxxxxxxxxxx
https://lists.squid-cache.org/listinfo/squid-users
_______________________________________________
squid-users mailing list
squid-users@xxxxxxxxxxxxxxxxxxxxx
https://lists.squid-cache.org/listinfo/squid-users
