Hi, Hoping someone can help me with this issue that I have been struggling with for days now. I am setting up squid on an ubuntu PC to forward HTTPS requests to an API and an s3 bucket under my control on amazon AWS. The reason I am setting up the proxy is two-fold...
1) To reduce costs from AWS.
2) To provide content to the client on the ubuntu PC if there is a networking issue somewhere in between the ubuntu PC and AWS.
Item 1 is going well so far. Item 2 is not going well. Setup details ...
# squid - setup cache folder
mkdir -p /var/cache/squid
chown -R proxy:proxy /var/cache/squid
# ssl - generate key
apt --yes install squid-openssl libnss3-tools
openssl req -new -newkey rsa:2048 -days 365 -nodes -x509 \
-subj "/C=US/ST=Denial/L=Springfield/O=Dis/CN=www.example.com" \
-keyout /etc/squid/stuff.pem -out /etc/squid/stuff.pem
chown root:proxy /etc/squid/stuff.pem
chmod 644 /etc/squid/stuff.pem
# ssl - ssl DB
mkdir -p /var/lib/squid
rm -rf /var/lib/squid/ssl_db
/usr/lib/squid/security_file_certgen -c -s /var/lib/squid/ssl_db -M 4MB
chown -R proxy:proxy /var/lib/squid/ssl_db
# /etc/squid/squid.conf :
acl to_aws dstdomain .amazonaws.com
acl from_local src localhost
http_access allow to_aws
http_access allow from_local
cache allow all
cache_dir ufs /var/cache/squid 1024 16 256
offline_mode on
http_port 3129 ssl-bump cert=/etc/squid/stuff.pem generate-host-certificates=on dynamic_cert_mem_cache_size=4MB
sslcrtd_program /usr/lib/squid/security_file_certgen -s /var/lib/squid/ssl_db -M 4MB
acl step1 at_step SslBump1
ssl_bump peek step1
ssl_bump bump all
sslproxy_cert_error deny all
cache_store_log stdio:/var/log/squid/store.log
logfile_rotate 0
# /usr/bin/proxy-test :
#!/bin/bash
curl --proxy http://localhost:3129 \
--cacert /etc/squid/stuff.pem \
-v "https://stuff.amazonaws.com/api/v1/stuff/stuff.json" \
-H "Authorization: token MYTOKEN" \
-H "Content-Type: application/json" \
--output "/tmp/stuff.json"
When network connectivity is GOOD, everything works well and I get cache HITS ...
# /var/log/squid/access.log
1705587538.837 238 127.0.0.1 NONE_NONE/200 0 CONNECT stuff.amazonaws.com:443 - HIER_DIRECT/3.136.246.238 -
1705587538.838 0 127.0.0.1 TCP_MEM_HIT/200 32818 GET https://stuff.amazonaws.com/api/v1/stuff/stuff.json - HIER_NONE/- application/json
# extract from /usr/bin/proxy-test output
< HTTP/1.1 200 OK
< Date: Thu, 18 Jan 2024 13:38:01 GMT
< Content-Type: application/json
< Content-Length: 32187
< x-amzn-RequestId: 8afba80e-6df7-4d5b-a34b-a70bd9b54380
< Last-Modified: 2024-01-03T11:23:19.000Z
< Access-Control-Allow-Origin: *
< x-amz-apigw-id: RvN1CF2_iYcEokA=
< Cache-Control: max-age=2147483648,public,stale-if-error
< ETag: "53896156c4e8e26933188a092c4e40f1"
< X-Amzn-Trace-Id: Root=1-65a929b9-3bd3285934151c1a2495481a
< Age: 2578
< Warning: 110 squid/5.7 "Response is stale"
< X-Cache: HIT from ubuntu-pc
< X-Cache-Lookup: HIT from ubuntu-pc:3129
< Via: 1.1 ubuntu-pc (squid/5.7)
< Connection: keep-alive
When network connectivity is BAD, I get errors and a cache MISS. In this test case I unplugged the ethernet cable from the back on the ubuntu-pc ...
# /var/log/squid/access.log
1705588717.420 11 127.0.0.1 NONE_NONE/200 0 CONNECT stuff.amazonaws.com:443 - HIER_DIRECT/3.135.162.228 -
1705588717.420 0 127.0.0.1 NONE_NONE/503 4087 GET https://stuff.amazonaws.com/api/v1/stuff/stuff.json - HIER_NONE/- text/html
# extract from /usr/bin/proxy-test output
< HTTP/1.1 503 Service Unavailable
< Server: squid/5.7
< Mime-Version: 1.0
< Date: Thu, 18 Jan 2024 14:38:37 GMT
< Content-Type: text/html;charset=utf-8
< Content-Length: 3692
< X-Squid-Error: ERR_CONNECT_FAIL 101
< Vary: Accept-Language
< Content-Language: en
< X-Cache: MISS from ubuntu-pc
< X-Cache-Lookup: NONE from ubuntu-pc:3129
< Via: 1.1 ubuntu-pc (squid/5.7)
< Connection: close
I have also seen it error in a different way with a 502 but with the same ultimate result.
My expectation/hope is that squid would return the cached object on any network failure in between ubuntu-pc and the AWS endpoint - and continue to return this cached object forever. Is this something squid can do? It would seem that offline_mode should do this?
Hope you can help,
Robin
1) To reduce costs from AWS.
2) To provide content to the client on the ubuntu PC if there is a networking issue somewhere in between the ubuntu PC and AWS.
Item 1 is going well so far. Item 2 is not going well. Setup details ...
# squid - setup cache folder
mkdir -p /var/cache/squid
chown -R proxy:proxy /var/cache/squid
# ssl - generate key
apt --yes install squid-openssl libnss3-tools
openssl req -new -newkey rsa:2048 -days 365 -nodes -x509 \
-subj "/C=US/ST=Denial/L=Springfield/O=Dis/CN=www.example.com" \
-keyout /etc/squid/stuff.pem -out /etc/squid/stuff.pem
chown root:proxy /etc/squid/stuff.pem
chmod 644 /etc/squid/stuff.pem
# ssl - ssl DB
mkdir -p /var/lib/squid
rm -rf /var/lib/squid/ssl_db
/usr/lib/squid/security_file_certgen -c -s /var/lib/squid/ssl_db -M 4MB
chown -R proxy:proxy /var/lib/squid/ssl_db
# /etc/squid/squid.conf :
acl to_aws dstdomain .amazonaws.com
acl from_local src localhost
http_access allow to_aws
http_access allow from_local
cache allow all
cache_dir ufs /var/cache/squid 1024 16 256
offline_mode on
http_port 3129 ssl-bump cert=/etc/squid/stuff.pem generate-host-certificates=on dynamic_cert_mem_cache_size=4MB
sslcrtd_program /usr/lib/squid/security_file_certgen -s /var/lib/squid/ssl_db -M 4MB
acl step1 at_step SslBump1
ssl_bump peek step1
ssl_bump bump all
sslproxy_cert_error deny all
cache_store_log stdio:/var/log/squid/store.log
logfile_rotate 0
# /usr/bin/proxy-test :
#!/bin/bash
curl --proxy http://localhost:3129 \
--cacert /etc/squid/stuff.pem \
-v "https://stuff.amazonaws.com/api/v1/stuff/stuff.json" \
-H "Authorization: token MYTOKEN" \
-H "Content-Type: application/json" \
--output "/tmp/stuff.json"
When network connectivity is GOOD, everything works well and I get cache HITS ...
# /var/log/squid/access.log
1705587538.837 238 127.0.0.1 NONE_NONE/200 0 CONNECT stuff.amazonaws.com:443 - HIER_DIRECT/3.136.246.238 -
1705587538.838 0 127.0.0.1 TCP_MEM_HIT/200 32818 GET https://stuff.amazonaws.com/api/v1/stuff/stuff.json - HIER_NONE/- application/json
# extract from /usr/bin/proxy-test output
< HTTP/1.1 200 OK
< Date: Thu, 18 Jan 2024 13:38:01 GMT
< Content-Type: application/json
< Content-Length: 32187
< x-amzn-RequestId: 8afba80e-6df7-4d5b-a34b-a70bd9b54380
< Last-Modified: 2024-01-03T11:23:19.000Z
< Access-Control-Allow-Origin: *
< x-amz-apigw-id: RvN1CF2_iYcEokA=
< Cache-Control: max-age=2147483648,public,stale-if-error
< ETag: "53896156c4e8e26933188a092c4e40f1"
< X-Amzn-Trace-Id: Root=1-65a929b9-3bd3285934151c1a2495481a
< Age: 2578
< Warning: 110 squid/5.7 "Response is stale"
< X-Cache: HIT from ubuntu-pc
< X-Cache-Lookup: HIT from ubuntu-pc:3129
< Via: 1.1 ubuntu-pc (squid/5.7)
< Connection: keep-alive
When network connectivity is BAD, I get errors and a cache MISS. In this test case I unplugged the ethernet cable from the back on the ubuntu-pc ...
# /var/log/squid/access.log
1705588717.420 11 127.0.0.1 NONE_NONE/200 0 CONNECT stuff.amazonaws.com:443 - HIER_DIRECT/3.135.162.228 -
1705588717.420 0 127.0.0.1 NONE_NONE/503 4087 GET https://stuff.amazonaws.com/api/v1/stuff/stuff.json - HIER_NONE/- text/html
# extract from /usr/bin/proxy-test output
< HTTP/1.1 503 Service Unavailable
< Server: squid/5.7
< Mime-Version: 1.0
< Date: Thu, 18 Jan 2024 14:38:37 GMT
< Content-Type: text/html;charset=utf-8
< Content-Length: 3692
< X-Squid-Error: ERR_CONNECT_FAIL 101
< Vary: Accept-Language
< Content-Language: en
< X-Cache: MISS from ubuntu-pc
< X-Cache-Lookup: NONE from ubuntu-pc:3129
< Via: 1.1 ubuntu-pc (squid/5.7)
< Connection: close
I have also seen it error in a different way with a 502 but with the same ultimate result.
My expectation/hope is that squid would return the cached object on any network failure in between ubuntu-pc and the AWS endpoint - and continue to return this cached object forever. Is this something squid can do? It would seem that offline_mode should do this?
Hope you can help,
Robin
_______________________________________________ squid-users mailing list squid-users@xxxxxxxxxxxxxxxxxxxxx https://lists.squid-cache.org/listinfo/squid-users
