On 2021-06-22 8:00 a.m., squid-users-request@xxxxxxxxxxxxxxxxxxxxx wrote:
On 21/06/21 6:41 am, Darwin O'Connor wrote:
I run a transit prediction web app <https://www.transsee.ca/>. It
connects to a variety of web APIs to collect the real time transit data
it needs. The app's activities are split among many processes. They
currently uses libcurl to connect to squid for caching (often for as
little as 10-30 seconds) and benefits of connection sharing.
There is still cases where data isn't being cached no matter what I do.
It is https data, but I am able to cache other https pages like
https://cdn.mbta.com/realtime/Alerts.pb
The refresh_pattern:
refresh_pattern .?????????????? 60????? 99999%? 7200 override-expire
override-lastmod reload-into-ims ignore-reload ignore-no-cache
ignore-no-store ignore-private ignore-auth store-stale
Please be aware that several of those options may be causing you more
problems than they solve:
* ignore-no-cache - no longer exists.
* ignore-reload - contradicts reload-into-ims and that can cause
inconsistent behaviour between initial MISS and followup HIT responses.
* override-lastmod - replaces all Last-Modification (L-M) headers with
values indicating the object is almost brand new. That prevents caches
detecting that objects have stuck around long enough not to need
replacing.
Luckily the example response has no L-M header, so that is not the
problem here. But it may be affecting other traffic.
I have removed these options. My refresh_pattern is now
refresh_pattern . 60 99999% 7200 override-expire
ignore-no-store ignore-private ignore-auth store-stale
Here are two requests, one after another. The headers match the actual
client apps.
* Trying 127.0.0.1:3128...
* Connected to 127.0.0.1 (127.0.0.1) port 3128 (#0)
GET https://api.transport.nsw.gov.au/v1/gtfs/alerts/buses HTTP/1.1
Host: 127.0.0.1:3128
User-Agent: curl/7.77.0 (+https://www.transsee.ca/)
Accept: */*
Accept-Encoding: gzip
Authorization: apikey 4sc5KapPRVzX6OeQm74FPCx9O0bFsIB7lei3
Cache-Control: max-age=60
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Date: Tue, 22 Jun 2021 23:23:06 GMT
< Content-Type: application/protobuf
< Content-Length: 7317
< Set-Cookie:
AWSALB=uydcNa4qcva/6YErLw+Ztl55Xvjtkf1N6HQiEfXOrPv46pjWn6uHEnqcRlPMWjcHXu6vAsbYjB/1WdxrVIYu0AAbrtIcz00zyAIiimoWQhWUlQFb6PO9120gf46N;
Expires=Tue, 29 Jun 2021 23:23:06 GMT; Path=/
< Set-Cookie:
AWSALBCORS=uydcNa4qcva/6YErLw+Ztl55Xvjtkf1N6HQiEfXOrPv46pjWn6uHEnqcRlPMWjcHXu6vAsbYjB/1WdxrVIYu0AAbrtIcz00zyAIiimoWQhWUlQFb6PO9120gf46N;
Expires=Tue, 29 Jun 2021 23:23:06 GMT; Path=/; SameSite=None; Secure
< Server: Apache-Coyote/1.1
< X-Powered-By: Express
< Access-Control-Allow-Credentials: true
< ETag: W/"9b9e-32oicwXN7rJop21q+Y++RZhtnac"
< Vary: Accept-Encoding
< X-Cache: Hit from cloudfront
< X-Amz-Cf-Pop: SYD1-C
< X-Amz-Cf-Id: fQeOCQLj5CiuSv2JV06y7O0r5rM0LvNchlXKFSBDKkES0-tUrmaorQ==
< Age: 31
< Access-Control-Allow-Origin: *
< Content-Encoding: gzip
< X-Cache: MISS from transsee
< X-Cache-Lookup: MISS from transsee:3128
< Via: 1.1 e57fe70b9ed429fb51b4b2432cadc67b.cloudfront.net (CloudFront),
1.1 transsee (squid/4.15)
< Connection: keep-alive
* Trying 127.0.0.1:3128...
* Connected to 127.0.0.1 (127.0.0.1) port 3128 (#0)
GET https://api.transport.nsw.gov.au/v1/gtfs/alerts/buses HTTP/1.1
Host: 127.0.0.1:3128
User-Agent: curl/7.77.0 (+https://www.transsee.ca/)
Accept: */*
Accept-Encoding: gzip
Authorization: apikey 4sc5KapPRVzX6OeQm74FPCx9O0bFsIB7lei3
Cache-Control: max-age=60
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Date: Tue, 22 Jun 2021 23:23:13 GMT
< Content-Type: application/protobuf
< Content-Length: 7317
< Set-Cookie:
AWSALB=zjeR+TBhpLTKE+mr042891Cxkb7caSUqya7wvG82Ca7cK86Z7iQco0ahsj/9H9WtdgJuiX/D9++AOYiQf3+9yryXrZscG5eX+X0+vVjI+t2fI/86DjV27tXLyt3o;
Expires=Tue, 29 Jun 2021 23:23:13 GMT; Path=/
< Set-Cookie:
AWSALBCORS=zjeR+TBhpLTKE+mr042891Cxkb7caSUqya7wvG82Ca7cK86Z7iQco0ahsj/9H9WtdgJuiX/D9++AOYiQf3+9yryXrZscG5eX+X0+vVjI+t2fI/86DjV27tXLyt3o;
Expires=Tue, 29 Jun 2021 23:23:13 GMT; Path=/; SameSite=None; Secure
< Server: Apache-Coyote/1.1
< X-Powered-By: Express
< Access-Control-Allow-Credentials: true
< ETag: W/"9b9e-32oicwXN7rJop21q+Y++RZhtnac"
< Vary: Accept-Encoding
< X-Cache: Hit from cloudfront
< X-Amz-Cf-Pop: SYD1-C1
< X-Amz-Cf-Id: cL-w8TjYWTb6cjEOLsyfwyIvZvQYTI7Hn4R8Uz6cppHEJwVUWejO1w==
< Age: 38
< Access-Control-Allow-Origin: *
< Content-Encoding: gzip
< X-Cache: MISS from transsee
< X-Cache-Lookup: MISS from transsee:3128
< Via: 1.1 e57fe70b9ed429fb51b4b2432cadc67b.cloudfront.net (CloudFront),
1.1 transsee (squid/4.15)
< Connection: keep-alive
Here is the corresponding records from the access log. Before the
records where from different fetches.
1624404186.746 298 127.0.0.1 59540 TCP_MISS/200 8380 GET
https://api.transport.nsw.gov.au/v1/gtfs/alerts/buses -
HIER_DIRECT/3.105.153.49 application/protobuf
1624404193.862 269 127.0.0.1 59886 TCP_MISS/200 8380 GET
https://api.transport.nsw.gov.au/v1/gtfs/alerts/buses -
HIER_DIRECT/3.105.153.49 application/protobuf
_______________________________________________
squid-users mailing list
squid-users@xxxxxxxxxxxxxxxxxxxxx
http://lists.squid-cache.org/listinfo/squid-users
