Re: TCP_MISS only

Andy Armstrong <andy.armstrong@xxxxxxxxxx> · Fri, 21 Oct 2022 12:50:08 +0000

Hi Amos, All,

I continue to struggle with only receiving TCP_MISS when using squid-cache. I have now implemented a server which is responding to HTTP GETs. I have checked the endpoint with redbot.org
 and it states that :

1/The response allows all caches to store it
2/The response is fresh until 7 days from now
The reason may still be served by a cache once it becomes stale.

Therefore I believe the server side of this is now valid as a test for squid cache.

My client, is routing calls to squidcache, and when it arrives there, I see the following log entry:

1666355825.841   
106 10.1.1.70 TCP_MISS/200 60646 GET http://192.168.20.1:3004/largeapp/largeapp/007005/bongo? - HIER_DIRECT/192.168.20.1 application/json

Infact every single client call I make results in this TCP_MISS.

I therefore think this may be to do with my squid.conf.

Please see my squid conf below – paying particular attention to the refresh patterns I added specifically for this call (because it wasn’t working without these so I thought I would
 add these to see if it would work).

Squid.conf:

http_port 3128

acl localnet src 10.0.0.0/8     # RFC1918 possible internal network
acl localnet src 172.16.0.0/12  # RFC1918 possible internal network
acl localnet src 192.168.0.0/16 # RFC1918 possible internal network
acl localnet src fc00::/7       # RFC 4193 local private network range
acl localnet src fe80::/10      # RFC 4291 link-local (directly plugged) machines

acl SSL_ports port 443

acl Safe_ports port 80          # http
acl Safe_ports port 21          # ftp
acl Safe_ports port 443         # https
acl Safe_ports port 70          # gopher
acl Safe_ports port 210         # wais
acl Safe_ports port 280         # http-mgmt
acl Safe_ports port 488         # gss-http
acl Safe_ports port 591         # filemaker
acl Safe_ports port 777         # multiling http
acl Safe_ports port 1025-65535  # unregistered ports

acl CONNECT method CONNECT

http_access deny !Safe_ports
http_access deny CONNECT !SSL_ports
http_access allow localhost manager
http_access deny manager

#
# INSERT YOUR OWN RULE(S) HERE TO ALLOW ACCESS FROM YOUR CLIENTS
#

http_access allow localnet
http_access allow localhost
http_access deny all

coredump_dir /squid/var/cache/squid

cache_dir ufs /var/spool/squid 1024 16 256

refresh_pattern -i http:\/\/192.168.20.1:3001\/.* 10080 100% 43200  override-lastmod
refresh_pattern -i http:\/\/192.168.20.1:3004\/.* 10080 100% 43200  override-lastmod
refresh_pattern ^ftp:                      1440      20%        10080
refresh_pattern ^gopher:             1440      0%          1440
refresh_pattern -i (/cgi-bin/|\?) 0             0%          0

refresh_pattern .                              0              20%        4320

The complete output from the logs is :
admin@zcxsys01:~$ docker logs squid-container
2022/10/21 12:16:58| WARNING: BCP 177 violation. Detected non-functional IPv6 loopback.
2022/10/21 12:16:58| Set Current Directory to /etc/squid
2022/10/21 12:16:58| Creating missing swap directories
2022/10/21 12:16:58| /var/spool/squid exists
2022/10/21 12:16:58| /var/spool/squid/00 exists
2022/10/21 12:16:58| Making directories in /var/spool/squid/00
2022/10/21 12:16:58| /var/spool/squid/01 exists
2022/10/21 12:16:58| Making directories in /var/spool/squid/01
2022/10/21 12:16:58| /var/spool/squid/02 exists
2022/10/21 12:16:58| Making directories in /var/spool/squid/02
2022/10/21 12:16:58| /var/spool/squid/03 exists
2022/10/21 12:16:58| Making directories in /var/spool/squid/03
2022/10/21 12:16:58| /var/spool/squid/04 exists
2022/10/21 12:16:58| Making directories in /var/spool/squid/04
2022/10/21 12:16:59| /var/spool/squid/05 exists
2022/10/21 12:16:59| Making directories in /var/spool/squid/05
2022/10/21 12:16:59| /var/spool/squid/06 exists
2022/10/21 12:16:59| Making directories in /var/spool/squid/06
2022/10/21 12:16:59| /var/spool/squid/07 exists
2022/10/21 12:16:59| Making directories in /var/spool/squid/07
2022/10/21 12:16:59| /var/spool/squid/08 exists
2022/10/21 12:16:59| Making directories in /var/spool/squid/08
2022/10/21 12:16:59| /var/spool/squid/09 exists
2022/10/21 12:16:59| Making directories in /var/spool/squid/09
2022/10/21 12:16:59| /var/spool/squid/0A exists
2022/10/21 12:16:59| Making directories in /var/spool/squid/0A
2022/10/21 12:16:59| /var/spool/squid/0B exists
2022/10/21 12:16:59| Making directories in /var/spool/squid/0B
2022/10/21 12:16:59| /var/spool/squid/0C exists
2022/10/21 12:16:59| Making directories in /var/spool/squid/0C
2022/10/21 12:16:59| /var/spool/squid/0D exists
2022/10/21 12:16:59| Making directories in /var/spool/squid/0D
2022/10/21 12:16:59| /var/spool/squid/0E exists
2022/10/21 12:16:59| Making directories in /var/spool/squid/0E
2022/10/21 12:16:59| /var/spool/squid/0F exists
2022/10/21 12:16:59| Making directories in /var/spool/squid/0F
2022/10/21 12:16:59| Removing PID file (/run/squid.pid)
2022/10/21 12:16:59| WARNING: BCP 177 violation. Detected non-functional IPv6 loopback.
2022/10/21 12:16:58| Set Current Directory to /etc/squid
2022/10/21 12:16:58| Creating missing swap directories
2022/10/21 12:16:58| /var/spool/squid exists
2022/10/21 12:16:58| /var/spool/squid/00 exists
2022/10/21 12:16:58| Making directories in /var/spool/squid/00
2022/10/21 12:16:58| /var/spool/squid/01 exists
2022/10/21 12:16:58| Making directories in /var/spool/squid/01
2022/10/21 12:16:58| /var/spool/squid/02 exists
2022/10/21 12:16:58| Making directories in /var/spool/squid/02
2022/10/21 12:16:58| /var/spool/squid/03 exists
2022/10/21 12:16:58| Making directories in /var/spool/squid/03
2022/10/21 12:16:58| /var/spool/squid/04 exists
2022/10/21 12:16:58| Making directories in /var/spool/squid/04
2022/10/21 12:16:59| /var/spool/squid/05 exists
2022/10/21 12:16:59| Making directories in /var/spool/squid/05
2022/10/21 12:16:59| /var/spool/squid/06 exists
2022/10/21 12:16:59| Making directories in /var/spool/squid/06
2022/10/21 12:16:59| /var/spool/squid/07 exists
2022/10/21 12:16:59| Making directories in /var/spool/squid/07
2022/10/21 12:16:59| /var/spool/squid/08 exists
2022/10/21 12:16:59| Making directories in /var/spool/squid/08
2022/10/21 12:16:59| /var/spool/squid/09 exists
2022/10/21 12:16:59| Making directories in /var/spool/squid/09
2022/10/21 12:16:59| /var/spool/squid/0A exists
2022/10/21 12:16:59| Making directories in /var/spool/squid/0A
2022/10/21 12:16:59| /var/spool/squid/0B exists
2022/10/21 12:16:59| Making directories in /var/spool/squid/0B
2022/10/21 12:16:59| /var/spool/squid/0C exists
2022/10/21 12:16:59| Making directories in /var/spool/squid/0C
2022/10/21 12:16:59| /var/spool/squid/0D exists
2022/10/21 12:16:59| Making directories in /var/spool/squid/0D
2022/10/21 12:16:59| /var/spool/squid/0E exists
2022/10/21 12:16:59| Making directories in /var/spool/squid/0E
2022/10/21 12:16:59| /var/spool/squid/0F exists
2022/10/21 12:16:59| Making directories in /var/spool/squid/0F
2022/10/21 12:16:59| Removing PID file (/run/squid.pid)
2022/10/21 12:16:59| Set Current Directory to /etc/squid
2022/10/21 12:16:59| Starting Squid Cache version 5.2 for s390x-ibm-linux-gnu...
2022/10/21 12:16:59| Service Name: squid
2022/10/21 12:16:59| Process ID 11
2022/10/21 12:16:59| Process Roles: master worker
2022/10/21 12:16:59| With 1048576 file descriptors available
2022/10/21 12:16:59| Initializing IP Cache...
2022/10/21 12:16:59| DNS Socket created at 0.0.0.0, FD 8
2022/10/21 12:16:59| Adding nameserver 10.11.5.1 from /etc/resolv.conf
2022/10/21 12:16:59| Adding nameserver 10.11.5.2 from /etc/resolv.conf
2022/10/21 12:16:59| Adding domain test.sys.one from /etc/resolv.conf
2022/10/21 12:16:59| Logfile: opening log daemon:/var/log/squid/access.log
2022/10/21 12:16:59| Logfile Daemon: opening log /var/log/squid/access.log
2022/10/21 12:17:00| Unlinkd pipe opened on FD 14
2022/10/21 12:17:00| Local cache digest enabled; rebuild/rewrite every 3600/3600 sec
2022/10/21 12:17:00| Store logging disabled
2022/10/21 12:17:00| Swap maxSize 1048576 + 262144 KB, estimated 100824 objects
2022/10/21 12:17:00| Target number of buckets: 5041
2022/10/21 12:17:00| Using 8192 Store buckets
2022/10/21 12:17:00| Max Mem  size: 262144 KB
2022/10/21 12:17:00| Max Swap size: 1048576 KB
2022/10/21 12:17:00| Rebuilding storage in /var/spool/squid (dirty log)
2022/10/21 12:17:00| Using Least Load store dir selection
2022/10/21 12:17:00| Set Current Directory to /etc/squid
2022/10/21 12:17:00| Finished loading MIME types and icons.
2022/10/21 12:17:00| HTCP Disabled.
2022/10/21 12:17:00| Pinger socket opened on FD 19
2022/10/21 12:17:00| Squid plugin modules loaded: 0
2022/10/21 12:17:00| Adaptation support is off.
2022/10/21 12:17:00| Accepting HTTP Socket connections at conn2 local=0.0.0.0:3128 remote=[::] FD 17 flags=9
2022/10/21 12:17:00| Done reading /var/spool/squid swaplog (0 entries)
2022/10/21 12:17:00| Store rebuilding is 0.00% complete
2022/10/21 12:17:00| Finished rebuilding storage from disk.
2022/10/21 12:17:00|         0 Entries scanned
2022/10/21 12:17:00|         0 Invalid entries.
2022/10/21 12:17:00|         0 With invalid flags.
2022/10/21 12:17:00|         0 Objects loaded.
2022/10/21 12:17:00|         0 Objects expired.
2022/10/21 12:17:00|         0 Objects cancelled.
2022/10/21 12:17:00|         0 Duplicate URLs purged.
2022/10/21 12:17:00|         0 Swapfile clashes avoided.
2022/10/21 12:17:00|   Took 0.12 seconds (  0.00 objects/sec).
2022/10/21 12:17:00| Beginning Validation Procedure
2022/10/21 12:17:00|   Completed Validation Procedure
2022/10/21 12:17:00|   Validated 0 Entries
2022/10/21 12:17:00|   store_swap_size = 0.00 KB
2022/10/21 12:17:00| WARNING: BCP 177 violation. Detected non-functional IPv6 loopback.
2022/10/21 12:17:00| pinger: Initialising ICMP pinger ...
2022/10/21 12:17:00| pinger: ICMP socket opened.
2022/10/21 12:17:00| pinger: ICMPv6 socket opened
2022/10/21 12:17:01| storeLateRelease: released 0 objects
1666355216.966    213 10.1.1.70 TCP_MISS/200 60615 GET http://192.168.20.1:3004/largeapp/largeapp/007005/bongo? - HIER_DIRECT/192.168.20.1 application/json
1666355220.808     93 10.1.1.70 TCP_MISS/200 60615 GET http://192.168.20.1:3004/largeapp/largeapp/007005/bongo? - HIER_DIRECT/192.168.20.1 application/json
1666355221.369     66 10.1.1.70 TCP_MISS/200 60615 GET http://192.168.20.1:3004/largeapp/largeapp/007005/bongo? - HIER_DIRECT/192.168.20.1 application/json
1666355222.770    254 10.1.1.70 TCP_MISS/200 60615 GET http://192.168.20.1:3004/largeapp/largeapp/007005/bongo? - HIER_DIRECT/192.168.20.1 application/json
1666355224.514     82 10.1.1.70 TCP_MISS/200 60615 GET http://192.168.20.1:3004/largeapp/largeapp/007005/bongo? - HIER_DIRECT/192.168.20.1 application/json
1666355225.518     80 10.1.1.70 TCP_MISS/200 60615 GET http://192.168.20.1:3004/largeapp/largeapp/007005/bongo? - HIER_DIRECT/192.168.20.1 application/json
1666355225.963     65 10.1.1.70 TCP_MISS/200 60615 GET http://192.168.20.1:3004/largeapp/largeapp/007005/bongo? - HIER_DIRECT/192.168.20.1 application/json
1666355226.366     64 10.1.1.70 TCP_MISS/200 60615 GET http://192.168.20.1:3004/largeapp/largeapp/007005/bongo? - HIER_DIRECT/192.168.20.1 application/json
1666355226.715     69 10.1.1.70 TCP_MISS/200 60615 GET http://192.168.20.1:3004/largeapp/largeapp/007005/bongo? - HIER_DIRECT/192.168.20.1 application/json
1666355227.371    315 10.1.1.70 TCP_MISS/200 60615 GET http://192.168.20.1:3004/largeapp/largeapp/007005/bongo? - HIER_DIRECT/192.168.20.1 application/json
1666355227.700     56 10.1.1.70 TCP_MISS/200 60615 GET http://192.168.20.1:3004/largeapp/largeapp/007005/bongo? - HIER_DIRECT/192.168.20.1 application/json
1666355239.913    207 10.1.1.70 TCP_MISS/200 60615 GET http://192.168.20.1:3004/largeapp/largeapp/007005/bongo? - HIER_DIRECT/192.168.20.1 application/json
1666355244.729     96 10.1.1.70 TCP_MISS/200 60615 GET http://192.168.20.1:3004/largeapp/largeapp/007005/bongo? - HIER_DIRECT/192.168.20.1 application/json
1666355245.014     63 10.1.1.70 TCP_MISS/200 60615 GET http://192.168.20.1:3004/largeapp/largeapp/007005/bongo? - HIER_DIRECT/192.168.20.1 application/json
1666355408.390    208 10.1.1.70 TCP_MISS/200 60646 GET http://192.168.20.1:3004/largeapp/largeapp/007005/bongo? - HIER_DIRECT/192.168.20.1 application/json
1666355426.040    197 10.1.1.70 TCP_MISS/200 60646 GET http://192.168.20.1:3004/largeapp/largeapp/007005/bongo? - HIER_DIRECT/192.168.20.1 application/json
1666355426.650     63 10.1.1.70 TCP_MISS/200 60646 GET http://192.168.20.1:3004/largeapp/largeapp/007005/bongo? - HIER_DIRECT/192.168.20.1 application/json
1666355427.196     65 10.1.1.70 TCP_MISS/200 60646 GET http://192.168.20.1:3004/largeapp/largeapp/007005/bongo? - HIER_DIRECT/192.168.20.1 application/json
1666355427.565     60 10.1.1.70 TCP_MISS/200 60646 GET http://192.168.20.1:3004/largeapp/largeapp/007005/bongo? - HIER_DIRECT/192.168.20.1 application/json
1666355427.915     64 10.1.1.70 TCP_MISS/200 60646 GET http://192.168.20.1:3004/largeapp/largeapp/007005/bongo? - HIER_DIRECT/192.168.20.1 application/json
1666355824.295    209 10.1.1.70 TCP_MISS/200 60646 GET http://192.168.20.1:3004/largeapp/largeapp/007005/bongo? - HIER_DIRECT/192.168.20.1 application/json
1666355825.153    120 10.1.1.70 TCP_MISS/200 60646 GET http://192.168.20.1:3004/largeapp/largeapp/007005/bongo? - HIER_DIRECT/192.168.20.1 application/json
1666355825.841    106 10.1.1.70 TCP_MISS/200 60646 GET http://192.168.20.1:3004/largeapp/largeapp/007005/bongo? - HIER_DIRECT/192.168.20.1 application/json

I suspect something is wrong with my config – perhaps the rules are not working correctly. Please can you offer some advice as to why I can’t get this response to cache in squid-cache.

Kind regards,

Andy Armstrong 
安迪
阿姆斯特朗
Principal Specialist for Z Technologies
EMEA Squad Leader for Hybrid Cloud
Worldwide Community Leader for Hybrid Cloud
Member of the CTO Office Server & Storage EMEA
Distinguished Technical Specialist – The Open Group
IBM Master Inventor

Mobile: +447500103874

From:
squid-users <squid-users-bounces@xxxxxxxxxxxxxxxxxxxxx> on behalf of Andy Armstrong <andy.armstrong@xxxxxxxxxx>

Date: Thursday, 29 September 2022 at 15:00

To: Amos Jeffries <squid3@xxxxxxxxxxxxx>, squid-users@xxxxxxxxxxxxxxxxxxxxx <squid-users@xxxxxxxxxxxxxxxxxxxxx>

Subject: [EXTERNAL] Re: [squid-users] TCP_MISS only

Hi, Excellent I understand and agree with what you are saying. Is this behaviour documented within the Squid documentation anywhere, or is this more ‘how does the
 HTTP specification handle caching’? ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍

ZjQcmQRYFpfptBannerStart

This Message Is From an External Sender

This message came from outside your organization.

ZjQcmQRYFpfptBannerEnd

Hi,

Excellent I understand and agree with what you are saying. Is this behaviour documented within the Squid documentation anywhere, or is this more ‘how does the HTTP specification
 handle caching’?

I am moving forward with a HTTP GET to see if that works per my use case. I assume therefore that any other verb is simply not going to work out the box?

Kind regards,

Andy Armstrong 
安迪
阿姆斯特朗
Principal Specialist for Z Technologies
EMEA Squad Leader for Hybrid Cloud
Worldwide Community Leader for Hybrid Cloud
Member of the CTO Office Server & Storage EMEA
Distinguished Technical Specialist – The Open Group
IBM Master Inventor

Mobile: +447500103874

From:
squid-users <squid-users-bounces@xxxxxxxxxxxxxxxxxxxxx> on behalf of Amos Jeffries <squid3@xxxxxxxxxxxxx>

Date: Thursday, 29 September 2022 at 13:06

To: squid-users@xxxxxxxxxxxxxxxxxxxxx <squid-users@xxxxxxxxxxxxxxxxxxxxx>

Subject: [EXTERNAL] Re: [squid-users] TCP_MISS only

On 28/09/22 07:56, Andy Armstrong wrote:

> Okay – but what happens if you are communicating with a non REST 

> endpoint.

You are still communicating over HTTP. To interact with and benefit from 

HTTP agents like caches you need to comply to the HTTP semantics they use.

IMO, REST is just a useful tool to define (in abstract) an API's 

operation when considering what/how it needs to be implemented.

> Consider a Web services endpoint for example where a request 

> is only interacted with via POST but the operation for example may 

> frequently be a read based function akin to a HTTP GET?

That is by definition a broken implementation of HTTP. The agent is 

using a *delivery* API (POST) for retrieval (GET).

If you can separate the delivery and fetch operations HTTP becomes much 

easier to use.

> Is Squid just 

> simply not going to help cache those requests?

Not *by default*, no.

POST implies changing some arbitrary resource *other* than the URL 

presented. Based on data and logic which may not be provided in the 

request message URL+headers.

To use POST with caching both the client *and* the server have to 

explicitly tell the HTTP cache agent(s) what to do on every single HTTP 

message.

  - The client has to tell the cache whether a stored response is able 

to be produced as reply, what object-ID it is trying to retrieve, what 

object-ID's it already knows about (if any), and how old the stored 

object is allowed to be.

  - The server has to tell the cache whether the response can be stored, 

what to use for a unique-ID of the reply object, how old it already is, 

how long it can be stored for, how and when to update it when it becomes 

stale.

The Squid refresh_pattern can provide defaults for the storage times 

when they are omitted. But all the ID related things and whether to use 

cache at all can only come from the client/server.

As you can see by limiting yourself to POST-only you have imposed a huge 

amount of complexity. Using GET instead for fetches makes all the above 

*optional* where now it is mandatory.

> It is only helpful for 

> more strict alignment to REST principles?

> 

You lost me here. Squid implements HTTP.

REST is a very abstract simplification of basic HTTP/1.0 semantics. So 

the closer ones code aligns to REST the *easier* it is to implement HTTP 

properly. But HTTP/1.1+ are vastly more than REST.

HTH

Amos

_______________________________________________

squid-users mailing list

squid-users@xxxxxxxxxxxxxxxxxxxxx

http://lists.squid-cache.org/listinfo/squid-users 

Unless otherwise stated above:

IBM United Kingdom Limited

Registered in England and Wales with number 741598

Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU

Unless otherwise stated above:

IBM United Kingdom Limited

Registered in England and Wales with number 741598

Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU

_______________________________________________
squid-users mailing list
squid-users@xxxxxxxxxxxxxxxxxxxxx
http://lists.squid-cache.org/listinfo/squid-users