Thank you, Alex and Eli,
Just wanted to update that I was able to solve the problem in case someone runs into it in the future,
based on Eliezers suggestion, I disabled logrotate (although it can be tweaked to work), and just wrote
a custom cron job to run daily and rotate the logs for each squid instance using something like this:
Just wanted to update that I was able to solve the problem in case someone runs into it in the future,
based on Eliezers suggestion, I disabled logrotate (although it can be tweaked to work), and just wrote
a custom cron job to run daily and rotate the logs for each squid instance using something like this:
squid -f /etc/squid/squid12.conf -k rotate......
Now everything seems to be working properly, logfile_rotate is set to 1, so the files move over once
and then they get deleted, just as I wanted.
and then they get deleted, just as I wanted.
On 7 Sep 2022, 19:27 +0300, NgTech LTD <ngtech1ltd@xxxxxxxxx>, wrote:
_______________________________________________Good one, Alex.For this specific use case you need a special rotate script which will know the confs file and will loop over them.Later on I will try to see if yave one of these on my servers.Basically you will need an array of config files and loop on them.The pid shouldn't be relevevant for a rotate operation but it depends on the nature of the system.(on a 24/7 system you should know about a service that is down way before the logrotate happpens)If you have a set of config files you can generate a set of postrotate commands compared to a special script.Let me know if this solution might fit for your use case.Eliezer
בתאריך יום ד׳, 7 בספט׳ 2022, 3:53, מאת Alex Rousskov <rousskov@xxxxxxxxxxxxxxxxxxxxxxx>:> pid_filename /var/run/squid2.pid
> postrotate
> test ! -e /var/run/squid.pid || ... /usr/sbin/squid -k rotate
> endscript
I spotted one more (potentially critical) problem: Your Squid
configuration sets pid_filename to /var/run/squid2.pid but your
logrotate configuration assumes Squid uses /var/run/squid.pid.
IMHO, in general, it is best not to guess where Squid has its PID if you
are using "squid -k ...". If you want to test whether Squid is currently
running, try using "squid -k check" instead.
HTH,
Alex.
On 9/6/22 20:45, Alex Rousskov wrote:
> On 9/6/22 18:02, roee klinger wrote:
>> it seems that the logs has filled over 100GB of log data, since I made
>> a configuration mistake (I think?) by setting this:
>>
>> logfile_rotate 0
>
> This is correct setting when using an external log rotation tool like
> the logrotate daemon. More on that below.
>
>
>> If I remember and read correctly, this means that the rotation of the
>> files is disabled and they will just keeping increasing
>> in size if left unchecked.
>
> To be more precise, this means that you are relying on an external tool
> to rename the log files. With this setting, Squid rotate command closes
> the access log and opens a new one (under the same name). While that
> might sound useless, it is the right (and necessary) thing for Squid to
> do when combined with the correct external log rotation setup.
>
>
>> I have now gone ahead and changed all the configuration file to this
>> setting:
>>
>> logfile_rotate 1
>>
>> So now it should rotate once daily, and on the next rotation it should
>> be deleted, and this is all handled by logrotate on Debian-based
>> machines?
>
> AFAIK, if you are using an external (to Squid) tool like logrotate, you
> should be setting logfile_rotate to zero.
>
>
>> This is my / cat /etc/logrotate.d/squid:
>> ➜ / cat /etc/logrotate.d/squid
>> #
>> # Logrotate fragment for squid.
>> #
>> /var/log/squid/*.log {
>> daily
>> compress
>> delaycompress
>> rotate 2
>> missingok
>> nocreate
>> sharedscripts
>> prerotate
>> test ! -x /usr/sbin/sarg-reports || /usr/sbin/sarg-reports daily
>> endscript
>> postrotate
>> test ! -e /var/run/squid.pid || test ! -x /usr/sbin/squid ||
>> /usr/sbin/squid -k rotate
>> endscript
>> }
>
> This is not my area of expertise, but the above configuration does not
> look 100% correct to me: sarg-reports execution failures should have no
> effect on log rotation but does (AFAICT). There may be other problems
> (e.g., I do not know whether your /usr/sbin/squid finds the right Squid
> configuration file). I hope sysadmin experts on this mailing list will
> help you polish this.
>
> You should be able to test whether the above is working (e.g., by asking
> logrotate to rotate). Testing is critical even if you do end up getting
> expert log rotation help on this list (this email is not it!).
>
>
> HTH,
>
> Alex.
>
>
>> Is there a way for me to set it so it just get deleted every 24 or 12
>> hours without the archive first?
>>
>> Thanks,
>> Roee
>> On 6 Sep 2022, 16:28 +0300, Alex Rousskov
>> <rousskov@xxxxxxxxxxxxxxxxxxxxxxx>, wrote:
>>> On 9/6/22 07:41, roee klinger wrote:
>>>
>>>> It is also important to know that I am running multiple Squid instances
>>>> on the same machine, they are all getting the error at the same time
>>>
>>> What external event(s) happen at that time? Something is probably
>>> sending a signal to the logging daemon process. It would be good to know
>>> what that something (and that signal) is. Your syslog or cache.log might
>>> contain more info. Analyzing the timing/schedule of these problems may
>>> also be helpful in identifying the trigger.
>>>
>>>
>>>> Is a possible workaround that might be just replacing the line with
>>>> this?
>>>
>>>> access_log /var/log/squid/access2.log
>>>
>>> As you know, this configuration (in this deprecated spelling or with and
>>> explicit "stdio:" prefix) will result in Squid workers writing to the
>>> log file directly instead of asking the logging daemon. This will,
>>> naturally, get rid of the pipe between workers and their daemons, and
>>> the associated broken pipe error.
>>>
>>>> or will this cause a problem?
>>>
>>> Impossible to say for sure without knowing whether your workers benefit
>>> from the anticipated performance advantages of avoiding blocking file
>>> I/O _and_ whether those advantages are real (in your environment). Too
>>> many variables and too many unknowns. I would treat this as an important
>>> (and potentially disruptive) configuration change and carefully test the
>>> outcome.
>>>
>>>
>>> HTH,
>>>
>>> Alex.
>>>
>>>
>>>> INFO -
>>>> Versions:
>>>>
>>>> Squid Cache: Version 4.10
>>>> Ubuntu 20.04.4 LTS
>>>>
>>>>
>>>> Example squid.conf:
>>>>
>>>> visible_hostname squid2
>>>>
>>>> access_log daemon:/var/log/squid/access2.log squid
>>>>
>>>> cache_log /var/log/squid/cache2.log
>>>>
>>>> pid_filename /var/run/squid2.pid
>>>>
>>>>
>>>> acl localnet src 0.0.0.1-0.255.255.255# RFC 1122 "this" network (LAN)
>>>>
>>>> acl localnet src 10.0.0.0/8 <http://10.0.0.0/8> # RFC 1918 local
>>>> private network (LAN)
>>>>
>>>> acl localnet src 100.64.0.0/10 <http://100.64.0.0/10># RFC 6598
>>>> shared address space (CGN)
>>>>
>>>> acl localnet src 169.254.0.0/16 <http://169.254.0.0/16> # RFC 3927
>>>> link-local (directly plugged) machines
>>>>
>>>> acl localnet src 172.16.0.0/12 <http://172.16.0.0/12># RFC 1918
>>>> local private network (LAN)
>>>>
>>>> acl localnet src 192.168.0.0/16 <http://192.168.0.0/16> # RFC 1918
>>>> local private network (LAN)
>>>>
>>>> acl localnet src fc00::/7 # RFC 4193 local private network range
>>>>
>>>> acl localnet src fe80::/10# RFC 4291 link-local (directly plugged)
>>>> machines
>>>>
>>>> acl SSL_ports port 443
>>>>
>>>> acl Safe_ports port 80# http
>>>>
>>>> acl Safe_ports port 21# ftp
>>>>
>>>> acl Safe_ports port 443 # https
>>>>
>>>> acl Safe_ports port 70# gopher
>>>>
>>>> acl Safe_ports port 210 # wais
>>>>
>>>> acl Safe_ports port 1025-65535# unregistered ports
>>>>
>>>> acl Safe_ports port 280 # http-mgmt
>>>>
>>>> acl Safe_ports port 488 # gss-http
>>>>
>>>> acl Safe_ports port 591 # filemaker
>>>>
>>>> acl Safe_ports port 777 # multiling http
>>>>
>>>> acl CONNECT method CONNECT
>>>>
>>>> http_access deny !Safe_ports
>>>>
>>>> http_access deny CONNECT !SSL_ports
>>>>
>>>> http_access allow localhost manager
>>>>
>>>> http_access deny manager
>>>>
>>>> # include /etc/squid/conf.d/*
>>>>
>>>> http_access allow localhost
>>>>
>>>> acl aws src *censored*
>>>>
>>>> http_access allow aws
>>>>
>>>> # http_access deny all
>>>>
>>>> tcp_outgoing_address *censored*
>>>>
>>>> http_port 10002
>>>>
>>>> coredump_dir /var/spool/squid
>>>>
>>>> refresh_pattern ^ftp: 144020% 10080
>>>>
>>>> refresh_pattern ^gopher:14400%1440
>>>>
>>>> refresh_pattern -i (/cgi-bin/|\?) 0 0%0
>>>>
>>>> refresh_pattern \/(Packages|Sources)(|\.bz2|\.gz|\.xz)$ 0 0% 0
>>>> refresh-ims
>>>>
>>>> refresh_pattern \/Release(|\.gpg)$ 0 0% 0 refresh-ims
>>>>
>>>> refresh_pattern \/InRelease$ 0 0% 0 refresh-ims
>>>>
>>>> refresh_pattern \/(Translation-.*)(|\.bz2|\.gz|\.xz)$ 0 0% 0
>>>> refresh-ims
>>>>
>>>> refresh_pattern . 0 20% 4320
>>>>
>>>>
>>>> shutdown_lifetime 1 seconds
>>>>
>>>> logfile_rotate 0
>>>>
>>>> max_filedescriptors 16384
>>>>
>>>> dns_nameservers 8.8.8.8 8.8.4.4 1.1.1.1
>>>>
>>>> cache deny all
>>>>
>>>> cache_dir null /tmp
>>>>
>>>> via off
>>>>
>>>> forwarded_for off
>>>>
>>>> request_header_access From deny all
>>>>
>>>> request_header_access Server deny all
>>>>
>>>> request_header_access WWW-Authenticate deny all
>>>>
>>>> request_header_access Link deny all
>>>>
>>>> request_header_access Cache-Control deny all
>>>>
>>>> request_header_access Proxy-Connection deny all
>>>>
>>>> request_header_access X-Cache deny all
>>>>
>>>> request_header_access X-Cache-Lookup deny all
>>>>
>>>> request_header_access Via deny all
>>>>
>>>> request_header_access X-Forwarded-For deny all
>>>>
>>>> request_header_access Pragma deny all
>>>>
>>>> request_header_access Keep-Alive deny all
>>>>
>>>> dns_v4_first on
>>>>
>>>>
>>>> Example service file:
>>>>
>>>> ## Copyright (C) 1996-2020 The Squid Software Foundation and
>>>> contributors
>>>>
>>>> ##
>>>>
>>>> ## Squid software is distributed under GPLv2+ license and includes
>>>>
>>>> ## contributions from numerous individuals and organizations.
>>>>
>>>> ## Please see the COPYING and CONTRIBUTORS files for details.
>>>>
>>>> ##
>>>>
>>>>
>>>> [Unit]
>>>>
>>>> Description=Squid Web Proxy Server
>>>>
>>>> Documentation=man:squid(8)
>>>>
>>>> After=network.target network-online.target nss-lookup.target
>>>>
>>>>
>>>> [Service]
>>>>
>>>> Type=forking
>>>>
>>>> PIDFile=/var/run/squid2.pid
>>>>
>>>> ExecStartPre=/usr/sbin/squid --foreground -z -f /etc/squid/squid2.conf
>>>>
>>>> ExecStart=/usr/sbin/squid -sYC -f /etc/squid/squid2.conf
>>>>
>>>> ExecReload=/bin/kill -HUP $MAINPID
>>>>
>>>> KillMode=mixed
>>>>
>>>>
>>>> [Install]
>>>>
>>>> WantedBy=multi-user.target
>>>>
>>>>
>>>>
>>>> Permissions:
>>>>
>>>> ➜ ls -alt /etc/squid/
>>>> total 128
>>>> drwxr-xr-x 2 root root 4096 Sep 6 11:33 .
>>>> -rw-r--r-- 1 root root 2831 Sep 6 11:33 squid7.conf
>>>> drwxr-xr-x 116 root root 4096 Sep 6 11:33 ..
>>>> -rw-r--r-- 1 root root 2830 Sep 6 11:33 squid2.conf
>>>> -rw-r--r-- 1 root root 2836 Sep 6 11:33 squid13.conf
>>>> -rw-r--r-- 1 root root 2836 Sep 6 11:32 squid23.conf
>>>> -rw-r--r-- 1 root root 2836 Sep 6 11:32 squid19.conf
>>>> -rw-r--r-- 1 root root 2832 Sep 6 11:32 squid1.conf
>>>> -rw-r--r-- 1 root root 2836 Sep 6 11:32 squid17.conf
>>>> -rw-r--r-- 1 root root 2832 Sep 6 11:31 squid4.conf
>>>> -rw-r--r-- 1 root root 2834 Sep 6 11:31 squid21.conf
>>>> -rw-r--r-- 1 root root 2833 Sep 6 11:31 squid25.conf
>>>> -rw-r--r-- 1 root root 2834 Sep 6 11:31 squid12.conf
>>>> -rw-r--r-- 1 root root 2832 Sep 6 11:31 squid3.conf
>>>> -rw-r--r-- 1 root root 2836 Sep 6 11:30 squid10.conf
>>>> -rw-r--r-- 1 root root 2835 Sep 6 11:30 squid11.conf
>>>> -rw-r--r-- 1 root root 2833 Sep 6 11:30 squid18.conf
>>>> -rw-r--r-- 1 root root 2830 Sep 6 11:30 squid8.conf
>>>> -rw-r--r-- 1 root root 2830 Sep 6 11:30 squid6.conf
>>>> -rw-r--r-- 1 root root 2833 Sep 6 11:30 squid28.conf
>>>> -rw-r--r-- 1 root root 2830 Sep 6 11:25 squid9.conf
>>>> -rw-r--r-- 1 root root 2836 Sep 6 11:25 squid24.conf
>>>> -rw-r--r-- 1 root root 2835 Sep 6 11:25 squid22.conf
>>>> -rw-r--r-- 1 root root 2837 Sep 6 11:25 squid20.conf
>>>> -rw-r--r-- 1 root root 2836 Sep 6 11:25 squid16.conf
>>>> -rw-r--r-- 1 root root 2835 Sep 6 11:25 squid15.conf
>>>> -rw-r--r-- 1 root root 2836 Sep 6 11:25 squid14.conf
>>>> -rw-r--r-- 1 root root 2831 Sep 6 11:25 squid5.conf
>>>> -rw-r--r-- 1 root root 2833 Sep 6 11:25 squid27.conf
>>>> -rw-r--r-- 1 root root 2835 Sep 6 11:25 squid26.conf
>>>> -rw-r--r-- 1 root root 2835 Sep 6 11:25 squid30.conf
>>>> -rw-r--r-- 1 root root 2835 Sep 6 11:25 squid29.conf
>>>>
>>>>
>>>> _______________________________________________
>>>> squid-users mailing list
>>>> squid-users@xxxxxxxxxxxxxxxxxxxxx
>>>> http://lists.squid-cache.org/listinfo/squid-users
>>>
>>> _______________________________________________
>>> squid-users mailing list
>>> squid-users@xxxxxxxxxxxxxxxxxxxxx
>>> http://lists.squid-cache.org/listinfo/squid-users
>
> _______________________________________________
> squid-users mailing list
> squid-users@xxxxxxxxxxxxxxxxxxxxx
> http://lists.squid-cache.org/listinfo/squid-users
_______________________________________________
squid-users mailing list
squid-users@xxxxxxxxxxxxxxxxxxxxx
http://lists.squid-cache.org/listinfo/squid-users
squid-users mailing list
squid-users@xxxxxxxxxxxxxxxxxxxxx
http://lists.squid-cache.org/listinfo/squid-users
_______________________________________________ squid-users mailing list squid-users@xxxxxxxxxxxxxxxxxxxxx http://lists.squid-cache.org/listinfo/squid-users
