|
Hello, I’m having some difficulties understanding some errors I’m getting with squid while trying to act as a cache of a yum repository. Squid works “mostly” fine and cache “most” of the files going through… but some won’t get into the cache and this is quite problematic for us. It doesn’t seem to depend on the size of the file or its name… I’m moving a problematic file and it will get cached just fine. And no matter how many times I “wget” the problematic file through the proxy, the file won’t get into the cache. Config wise, I’ve tried multiple things. SMP / non SMP. Rock / aufs / in memory only… The machine its running on is quite beefy (200GB of ram, 48 cores) and does nothing else than running squid. And so far my config looks like this: debug_options ALL,1 11,3 31,3 65,3 16,5 20,5 cache_mem 1 GB maximum_object_size_in_memory 10 MB maximum_object_size 512 MB workers 3 access_log daemon:/var/log/squid/access.log
squid refresh_pattern . 3600 90% 43200 override-expire ignore-no-cache ignore-no-store ignore-private ignore-must-revalidate ignore-reload acl authorized_domains dstdomain .hpc.edf.fr http_access allow authorized_domains http_access allow localhost http_port
10.10.10.10:3142 http_port 127.0.0.1:3142 cache_dir rock /data 3000 max-size=500MB And without further ado, the errors I’m getting in the logs with some of the files… ---------- HTTP/1.1 200 OK^M Date: Wed, 18 Nov 2020 14:25:56 GMT^M Server: Apache/2.4.10 (Debian)^M Last-Modified: Thu, 08 Oct 2020 14:54:47 GMT^M ETag: "e6c-5b12a0365d88b"^M Accept-Ranges: bytes^M Content-Length: 3692^M Vary: Accept-Encoding^M Content-Type: application/xml^M X-Cache: MISS from vipproxy1.mydomain^M X-Cache-Lookup: MISS from vipproxy1.mydomain:3142^M Via: 1.1 vipproxy1.mydomain (squid/5.0.1-VCS)^M Connection: keep-alive^M ^M ---------- 2020/11/18 15:25:56.616 kid3| 20,5| store.cc(814) write: storeWrite: writing 1145 bytes for '3BD6C6FADCADCACA042F71576702C21D' 2020/11/18 15:25:56.616 kid3| 20,3| store_swapout.cc(374) mayStartSwapOut: already allowed 2020/11/18 15:25:56.616 kid3| 11,3| http.cc(1087) persistentConnStatus: conn21 local=123.456.789.15:49420 remote=123.456.789.107:80 HIER_DIRECT FD 41 flags=1 eof=0 2020/11/18 15:25:56.616 kid3| 20,5| store.cc(814) write: storeWrite: writing 2547 bytes for '3BD6C6FADCADCACA042F71576702C21D' 2020/11/18 15:25:56.616 kid3| 20,3| store_swapout.cc(374) mayStartSwapOut: already allowed 2020/11/18 15:25:56.616 kid3| 11,3| http.cc(1087) persistentConnStatus: conn21 local=123.456.789.15:49420 remote=123.456.789.107:80 HIER_DIRECT FD 41 flags=1 eof=0 2020/11/18 15:25:56.616 kid3| 20,3| store.cc(1065) complete: storeComplete: '3BD6C6FADCADCACA042F71576702C21D' 2020/11/18 15:25:56.616 kid3| 20,3| store.cc(1258) validLength: storeEntryValidLength: Checking '3BD6C6FADCADCACA042F71576702C21D' 2020/11/18 15:25:56.616 kid3| 20,5| store.cc(1260) validLength: storeEntryValidLength: object_len = 4007 2020/11/18 15:25:56.616 kid3| 20,5| store.cc(1261) validLength: storeEntryValidLength: hdr_sz = 315 2020/11/18 15:25:56.616 kid3| 20,5| store.cc(1262) validLength: storeEntryValidLength: content_length = 3692 2020/11/18 15:25:56.616 kid3| 20,3| store_swapout.cc(374) mayStartSwapOut: already allowed 2020/11/18 15:25:56.616 kid3| 20,5| store_swapout.cc(49) storeSwapOutStart: storeSwapOutStart: Begin SwapOut 'http://repos.mydomain/repos/rpm-hpc/el8/repodata/repomd.xml' to dirno -1, fileno FFFFFFFF 2020/11/18 15:25:56.616 kid3| 20,3| store_swapmeta.cc(52) storeSwapMetaBuild: storeSwapMetaBuild URL:
http://repos.mydomain/repos/rpm-hpc/el8/repodata/repomd.xml 2020/11/18 15:25:56.616 kid3| 20,2| store_io.cc(38) storeCreate: storeCreate: no swapdirs for e:tw1343=sp2V/0x559eb98619f0*4 2020/11/18 15:25:56.616 kid3| 20,5| Transients.cc(295) evictCached: e:tw1343=sp2V/0x559eb98619f0*4 2020/11/18 15:25:56.616 kid3| 20,3| store.cc(466) unlock: Client unlocking key 3BD6C6FADCADCACA042F71576702C21D e:tw1343=sp2V/0x559eb98619f0*4 2020/11/18 15:25:56.616 kid3| 20,3| store.cc(466) unlock: FwdState unlocking key 3BD6C6FADCADCACA042F71576702C21D e:tw1343=sp2V/0x559eb98619f0*3 2020/11/18 15:25:56.616 kid3| 20,3| store.cc(442) lock: store_client::copy locked key 3BD6C6FADCADCACA042F71576702C21D e:tw1343=sp2V/0x559eb98619f0*3 2020/11/18 15:25:56.616 kid3| 20,3| store.cc(466) unlock: store_client::copy unlocking key 3BD6C6FADCADCACA042F71576702C21D e:tw1343=sp2V/0x559eb98619f0*3 2020/11/18 15:25:56.616 kid3| 20,3| store_swapout.cc(347) mayStartSwapOut: already rejected 2020/11/18 15:25:56.616 kid3| 20,3| store.cc(442) lock: storeUnregister locked key 3BD6C6FADCADCACA042F71576702C21D e:tw1343=sp2V/0x559eb98619f0*3 2020/11/18 15:25:56.616 kid3| 20,3| store.cc(466) unlock: storeUnregister unlocking key 3BD6C6FADCADCACA042F71576702C21D e:tw1343=sp2V/0x559eb98619f0*3 2020/11/18 15:25:56.616 kid3| 20,3| store.cc(466) unlock: clientReplyContext::removeStoreReference unlocking key 3BD6C6FADCADCACA042F71576702C21D e:tw1343=sp2V/0x559eb98619f0*2 2020/11/18 15:25:56.616 kid3| 20,3| store.cc(466) unlock: ClientHttpRequest::loggingEntry unlocking key 3BD6C6FADCADCACA042F71576702C21D e:tw1343=sp2V/0x559eb98619f0*1 2020/11/18 15:25:56.616 kid3| 20,5| store.cc(482) doAbandon: e:tw1343=sp2V/0x559eb98619f0*0 via ClientHttpRequest::loggingEntry 2020/11/18 15:25:56.616 kid3| 20,5| Controller.cc(654) handleIdleEntry: destroying unlocked entry: 0x559eb98619f0 e:tw1343=sp2V/0x559eb98619f0*0 2020/11/18 15:25:56.616 kid3| 20,3| store.cc(398) destroyStoreEntry: destroyStoreEntry: destroying 0x559eb98619f8 2020/11/18 15:25:56.616 kid3| 20,3| store.cc(380) destroyMemObject: 0x559eb9861a70 in e:tw1343=sp2V/0x559eb98619f0*0 2020/11/18 15:25:56.616 kid3| 20,5| Transients.cc(320) disconnect: e:tw1343=sp2V/0x559eb98619f0*0 2020/11/18 15:25:56.616 kid3| 20,3| MemObject.cc(108) ~MemObject: MemObject destructed, this=0x559eb9861a70 2020/11/18 15:25:56.616 kid3| 20,5| store.cc(353) ~StoreEntry: StoreEntry destructed, this=0x559eb98619f0 I’m running a rather recent version of squid that I recompiled from git with the “enable-http-violations” use flag. Squid Cache: Version 5.0.1-VCS Service Name: squid This binary uses OpenSSL 1.1.1c FIPS 28 May 2019. For legal restrictions on distribution see
https://www.openssl.org/source/license.html configure options: '--build=x86_64-redhat-linux-gnu' '--host=x86_64-redhat-linux-gnu' '--program-prefix=' '--prefix=/usr' '--exec-prefix=/usr' '--bindir=/usr/bin' '--sbindir=/usr/sbin' '--sysconfdir=/etc' '--datadir=/usr/share'
'--includedir=/usr/include' '--libdir=/usr/lib64' '--libexecdir=/usr/libexec' '--localstatedir=/var' '--sharedstatedir=/var/lib' '--mandir=/usr/share/man' '--infodir=/usr/share/info' '--libexecdir=/usr/lib64/squid' '--datadir=/usr/share/squid' '--sysconfdir=/etc/squid'
'--with-logdir=/var/log/squid' '--with-pidfile=/var/run/squid.pid' '--disable-dependency-tracking' '--enable-eui' '--enable-follow-x-forwarded-for' '--enable-auth' '--enable-auth-basic=DB,fake,getpwnam,LDAP,NCSA,PAM,POP3,RADIUS,SASL,SMB,SMB_LM' '--enable-auth-ntlm=SMB_LM,fake'
'--enable-auth-digest=file,LDAP' '--enable-auth-negotiate=kerberos' '--enable-storeid-rewrite-helpers=file' '--enable-cache-digests' '--enable-cachemgr-hostname=localhost' '--enable-delay-pools' '--enable-epoll' '--enable-icap-client' '--enable-ident-lookups'
'--enable-linux-netfilter' '--enable-removal-policies=heap,lru' '--enable-snmp' '--enable-ssl' '--enable-ssl-crtd' '--enable-storeio=aufs,diskd,ufs,rock' '--enable-diskio' '--enable-wccpv2' '--enable-esi' '--enable-ecap' '--with-aio' '--with-default-user=squid'
'--with-dl' '--with-openssl' '--with-pthreads' '--disable-arch-native' '--disable-security-cert-validators' '--with-swapdir=/var/spool/squid' 'build_alias=x86_64-redhat-linux-gnu' 'host_alias=x86_64-redhat-linux-gnu' 'CFLAGS=-O2 -g -pipe -Wall -Werror=format-security
-Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection
-fcf-protection' 'LDFLAGS=-Wl,-z,relro -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld' 'CXXFLAGS=-O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches
-specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection' 'PKG_CONFIG_PATH=:/usr/lib64/pkgconfig:/usr/share/pkgconfig' '--enable-http-violations' We have resorted to use apt-cacher-ng instead of squid for the moment as we are heavily impacted “performance wise” by this lack of caching on some of the files. If anyone has hints or suggestions of what might be the problem ? Thx, Guillaume. |
Ce message et toutes les pièces jointes (ci-après le 'Message') sont établis à l'intention exclusive des destinataires et les informations qui y figurent sont strictement confidentielles. Toute utilisation de ce Message non conforme à sa destination, toute diffusion ou toute publication totale ou partielle, est interdite sauf autorisation expresse.
Si vous n'êtes pas le destinataire de ce Message, il vous est interdit de le copier, de le faire suivre, de le divulguer ou d'en utiliser tout ou partie. Si vous avez reçu ce Message par erreur, merci de le supprimer de votre système, ainsi que toutes ses copies, et de n'en garder aucune trace sur quelque support que ce soit. Nous vous remercions également d'en avertir immédiatement l'expéditeur par retour du message.
Il est impossible de garantir que les communications par messagerie électronique arrivent en temps utile, sont sécurisées ou dénuées de toute erreur ou virus.
____________________________________________________
This message and any attachments (the 'Message') are intended solely for the addressees. The information contained in this Message is confidential. Any use of information contained in this Message not in accord with its purpose, any dissemination or disclosure, either whole or partial, is prohibited except formal approval.
If you are not the addressee, you may not copy, forward, disclose or use any part of it. If you have received this message in error, please delete it and all copies from your system and notify the sender immediately by return message.
E-mail communication cannot be guaranteed to be timely secure, error or virus-free.
_______________________________________________ squid-users mailing list squid-users@xxxxxxxxxxxxxxxxxxxxx http://lists.squid-cache.org/listinfo/squid-users
