I appreciate very much your looking at my question.
>Date: Thu, 30 May 2024 22:15:27 +1200
>From: Amos Jeffries <squid3@xxxxxxxxxxxxx>
>To: squid-users@xxxxxxxxxxxxxxxxxxxxx
>Subject: Re: can't explain 403 denied for authenticated
> user
>Message-ID: <96746157-da51-47db-8d52-d65239f2717e@xxxxxxxxxxxxx>
>Content-Type: text/plain; charset=UTF-8; format=flowed
>
>On 25/05/24 07:28, Kevin wrote:
>> Hi,
>>
>> We have 2 external ACLs that take a request's data (IP, authenticated
>> username, URL, user-agent, etc) and uses that information to determine
>> whether a user or host should be permitted to access that URL.?? It
>> almost always works well, but we have a recurring occasional issue that
>> I can't figure out.
>>
>> When it occurs, it's always around 4AM.?? This particular request occurs
>> often - averages about once a second throughout the day.
>>
>> What we see is a "403 forbidden" for a (should be) permitted site from
>> an authenticated user from the same IP/user and to the same site that
>> gets a "202 connection established" every other time.
>
>Is maybe 4am the time when your auth system refreshes all nonce?
> - thus making any currently in-use by the clients invalid until they
>re-auth. You might see a mix of 403/401/407 in a bunch at such times.
>From: Amos Jeffries <squid3@xxxxxxxxxxxxx>
>To: squid-users@xxxxxxxxxxxxxxxxxxxxx
>Subject: Re: can't explain 403 denied for authenticated
> user
>Message-ID: <96746157-da51-47db-8d52-d65239f2717e@xxxxxxxxxxxxx>
>Content-Type: text/plain; charset=UTF-8; format=flowed
>
>On 25/05/24 07:28, Kevin wrote:
>> Hi,
>>
>> We have 2 external ACLs that take a request's data (IP, authenticated
>> username, URL, user-agent, etc) and uses that information to determine
>> whether a user or host should be permitted to access that URL.?? It
>> almost always works well, but we have a recurring occasional issue that
>> I can't figure out.
>>
>> When it occurs, it's always around 4AM.?? This particular request occurs
>> often - averages about once a second throughout the day.
>>
>> What we see is a "403 forbidden" for a (should be) permitted site from
>> an authenticated user from the same IP/user and to the same site that
>> gets a "202 connection established" every other time.
>
>Is maybe 4am the time when your auth system refreshes all nonce?
> - thus making any currently in-use by the clients invalid until they
>re-auth. You might see a mix of 403/401/407 in a bunch at such times.
My auth system is just squid with one flat file each for basic and digest authentication (we allow basic only if an application doesn't support digest).
>Or maybe in a similar style one/some of the clients is broken and fails
>to update its nonce before it expires at 4am?
The client is a Java-based application. I can find out more from the team that wrote it.
> - looking at which client agent and IP were getting the 403 and/or the
>nonce which received 403 will give you hints about this possibility.
>nonce which received 403 will give you hints about this possibility.
Oh: it's only this one application from this one particular host.
>
>Or your network router(s) do garbage collection and terminate
>long-running connections to free up TCP resources?
> - thus forcing a lot of client re-connects at 4am, which may:
> a) overload the auth system/helper, or
> b) break a transaction that included nonce update for clients -
>resulting in their next request being invalid nonce.
I'm not aware of a 4AM connection-terminating task but will confirm. As mentioned (in answer to a), the auth system is just squid. Re: b - I hadn't thought of that.
>
>
>Or maybe you have log processing software that does "squid -k restart"
>instead of the proper "squid -k rotate" to get access to the log files?
>
>Or maybe you have log processing software that does "squid -k restart"
>instead of the proper "squid -k rotate" to get access to the log files?
I hadn't checked that! But logrotate runs at 00:00 and the rotate script does use "squid -k rotate".
>Or maybe your auth system has a limit on how large nonce-count can become?
> - I notice that the working request has 0x2F uses and the forbidden
>has 0x35 (suspiciously close to 50 in decimal)
I don't know what that is. I didn't find a lot of helpful detail on squid's digest auth implementation.
>
>>
>> The difference I see in the logs:? though all the digest auth info looks
>> okay, the %un field in the log for the usual (successful) request is the
>> authenticated username, while in the failed request, it's "-".?? So
>> though there hasn't been an authentication error or "authentication
>> required" in the log - and the username is in the authentication details
>> in the log entry -? it seems like squid isn't recognizing that username
>> as %un.
>
>
>Be aware that a properly behaving client will *not* send credentials
>until they are requested, after which it should *always* send
>credentials on that same connection (even if they are not requested
>explicitly).
>>
>> The difference I see in the logs:? though all the digest auth info looks
>> okay, the %un field in the log for the usual (successful) request is the
>> authenticated username, while in the failed request, it's "-".?? So
>> though there hasn't been an authentication error or "authentication
>> required" in the log - and the username is in the authentication details
>> in the log entry -? it seems like squid isn't recognizing that username
>> as %un.
>
>
>Be aware that a properly behaving client will *not* send credentials
>until they are requested, after which it should *always* send
>credentials on that same connection (even if they are not requested
>explicitly).
>That means some requests on a multiplex/pipeline/keep-alive connection
>MAY arrive with credentials and be accepted(2xx)/denied(403) without
>authentication having occured. Entirely due to your *_access directives
>sequence. In these cases the log will show auth headers but no value for
>%un and/or %ul.
>
>
>>
>> My squid.conf first tests a request to see if an unauthenticated request
>> from a particular host is permitted.? That external ACL doesn't take a
>> username as an argument.?? If that external ACL passes, the request is
>> allowed.
>>
>
>Please *show* the config lines rather than describing what you *think*
>they do. Their exact ordering matters *a lot*.
> Obfuscation of sensitive details is okay/expected so long as you makes
>it easy for us to tell that value A and value B are different.
>it easy for us to tell that value A and value B are different.
>
>FWIW; if your config file is *actually* containing only what you
>described it is missing a number of things necessary to have a safe and
>secure proxy. A look at your full config (without comments or empty
>lines) will help us point out any unnoticed issues for you to consider
>fixing.
>FWIW; if your config file is *actually* containing only what you
>described it is missing a number of things necessary to have a safe and
>secure proxy. A look at your full config (without comments or empty
>lines) will help us point out any unnoticed issues for you to consider
>fixing.
Understood. Here it is:
acl localnet src 10.0.0.0/8 # RFC1918 possible internal network
acl localnet src 172.16.0.0/12 # RFC1918 possible internal network
acl localnet src 192.168.0.0/16 # RFC1918 possible internal network
acl localnet src fc00::/7 # RFC 4193 local private network range
acl localnet src fe80::/10 # RFC 4291 link-local (directly plugged) machines
acl windows_net src 172.18.114.0/24
acl sample_host src 172.18.115.1/32
acl rsync port 873
acl SSL_ports port 443
acl SSL_ports port 873 #873 is rsync
acl Safe_ports port 80 # http
acl Safe_ports port 21 # ftp
acl Safe_ports port 443 # https
acl Safe_ports port 70 # gopher
acl Safe_ports port 210 # wais
acl Safe_ports port 1025-65535 # unregistered ports
acl Safe_ports port 280 # http-mgmt
acl Safe_ports port 488 # gss-http
acl Safe_ports port 591 # filemaker
acl Safe_ports port 777 # multiling http
acl Safe_ports port 873
acl CONNECT method CONNECT
acl PURGE method PURGE
acl localhost src 127.0.0.1
http_access allow PURGE localhost
http_access deny PURGE
acl URN proto URN
http_access deny URN
http_access deny manager
acl API_FIREFOX dstdomain api.profiler.firefox.com
http_access deny API_FIREFOX
acl ff_browser browser ^Mozilla/5\.0
acl rma_ua browser ^RMA/1\.0.*compatible;.RealMedia
uri_whitespace encode
acl trellix_phone_cloud dstdomain amcore-ens.rest.gti.trellix.com
http_access deny trellix_phone_cloud
external_acl_type host_based_filter children-max=15 ttl=0 %ACL %DATA %SRC %>rd %>rP /PATH/TO/FILTER/SCRIPT.py
acl HostBasedRules external host_based_filter
http_access allow HostBasedRules
auth_param digest program /usr/lib/squid/digest_file_auth -c /etc/squid/passwd
auth_param digest realm squid
auth_param digest children 2
auth_param basic program /usr/lib/squid/basic_ncsa_auth /etc/squid/basic_passwd
auth_param basic children 2
auth_param basic realm squidb
auth_param basic credentialsttl 2 hours
acl auth_users proxy_auth REQUIRED
external_acl_type custom_acl_db children-max=15 ttl=0 %ACL %DATA %ul %SRC %>rd %>rP %credentials /PATH/TO/FILTER/SCRIPT.py
acl CustomAclDB external custom_acl_db
http_access allow CustomAclDB
acl CRLs url_regex "/etc/squid/conf.d/CRL_urls.txt"
http_access allow CRLs
deny_info 303:https://abc.def.com/
http_access deny all
acl apache rep_header Server ^Apache
icp_access allow localnet
icp_access deny all
http_access deny !Safe_ports
http_access deny CONNECT !SSL_ports
http_access allow localhost manager
http_access deny manager
http_access allow localhost
http_port 3128
coredump_dir /var/cache/squid
refresh_pattern ^ftp: 1440 20% 10080
refresh_pattern ^gopher: 1440 0% 1440
refresh_pattern -i (/cgi-bin/|\?) 0 0% 0
refresh_pattern . 0 20% 4320
logformat squid %ts.%03tu %6tr %>a %Ss/%>Hs %<st %rm %ru %un %Sh/%<A %mt
logformat squidmime %ul %ts.%03tu %6tr %>a %Ss/%>Hs %<st %rm %ru %un %Sh/%<A %mt [%>h] [%<h]
logformat common %>a %ui %un [%tl] "%rm %ru HTTP/%rv" %>Hs %<st %Ss:%Sh
logformat combined %>a %ui %un [%tl] "%rm %ru HTTP/%rv" %>Hs %<st "%{Referer}>h" "%{User-Agent}>h" %Ss:%Sh
logformat ip_port %ts %tu %>a %>p %<lp %<a %<p %dt %tr %un %>ru %Ss
access_log daemon:/var/log/squid/access.log combined
access_log daemon:/var/log/squid/network.log ip_port
cache_log /var/log/squid/cache.log
cache_store_log /var/log/squid/store.log
access_log daemon:/var/log/squid/useragent.log useragent
visible_hostname proxy.abc.com
cache deny all >
>> The next line in squid.conf is
>>
>> acl auth_users proxy_auth REQUIRED
>>
>
>FYI the above just means that Squid is using authentication. It says
>nothing about when the authentication will be (or not be) performed.
>
>
>> ... and after that, the external ACL that takes the username as well as
>> the other info.
>>
>
>
>HTH
>Amos
From: squid-users-request@xxxxxxxxxxxxxxxxxxxxx
To: "squid-users" <squid-users@xxxxxxxxxxxxxxxxxxxxx>
Sent: Thursday, May 30, 2024 8:00:01 AM
Subject: squid-users Digest, Vol 117, Issue 31
To: "squid-users" <squid-users@xxxxxxxxxxxxxxxxxxxxx>
Sent: Thursday, May 30, 2024 8:00:01 AM
Subject: squid-users Digest, Vol 117, Issue 31
I appreciate very much your looking at my question.
>Date: Thu, 30 May 2024 22:15:27 +1200
>From: Amos Jeffries <squid3@xxxxxxxxxxxxx>
>To: squid-users@xxxxxxxxxxxxxxxxxxxxx
>Subject: Re: can't explain 403 denied for authenticated
> user
>Message-ID: <96746157-da51-47db-8d52-d65239f2717e@xxxxxxxxxxxxx>
>Content-Type: text/plain; charset=UTF-8; format=flowed
>
>On 25/05/24 07:28, Kevin wrote:
>> Hi,
>>
>> We have 2 external ACLs that take a request's data (IP, authenticated
>> username, URL, user-agent, etc) and uses that information to determine
>> whether a user or host should be permitted to access that URL.?? It
>> almost always works well, but we have a recurring occasional issue that
>> I can't figure out.
>>
>> When it occurs, it's always around 4AM.?? This particular request occurs
>> often - averages about once a second throughout the day.
>>
>> What we see is a "403 forbidden" for a (should be) permitted site from
>> an authenticated user from the same IP/user and to the same site that
>> gets a "202 connection established" every other time.
>
>Is maybe 4am the time when your auth system refreshes all nonce?
> - thus making any currently in-use by the clients invalid until they
>re-auth. You might see a mix of 403/401/407 in a bunch at such times.
>From: Amos Jeffries <squid3@xxxxxxxxxxxxx>
>To: squid-users@xxxxxxxxxxxxxxxxxxxxx
>Subject: Re: can't explain 403 denied for authenticated
> user
>Message-ID: <96746157-da51-47db-8d52-d65239f2717e@xxxxxxxxxxxxx>
>Content-Type: text/plain; charset=UTF-8; format=flowed
>
>On 25/05/24 07:28, Kevin wrote:
>> Hi,
>>
>> We have 2 external ACLs that take a request's data (IP, authenticated
>> username, URL, user-agent, etc) and uses that information to determine
>> whether a user or host should be permitted to access that URL.?? It
>> almost always works well, but we have a recurring occasional issue that
>> I can't figure out.
>>
>> When it occurs, it's always around 4AM.?? This particular request occurs
>> often - averages about once a second throughout the day.
>>
>> What we see is a "403 forbidden" for a (should be) permitted site from
>> an authenticated user from the same IP/user and to the same site that
>> gets a "202 connection established" every other time.
>
>Is maybe 4am the time when your auth system refreshes all nonce?
> - thus making any currently in-use by the clients invalid until they
>re-auth. You might see a mix of 403/401/407 in a bunch at such times.
My auth system is just squid with one flat file each for basic and digest authentication (we allow basic only if an application doesn't support digest).
>Or maybe in a similar style one/some of the clients is broken and fails
>to update its nonce before it expires at 4am?
The client is a Java-based application. I can find out more from the team that wrote it.
> - looking at which client agent and IP were getting the 403 and/or the
>nonce which received 403 will give you hints about this possibility.
>nonce which received 403 will give you hints about this possibility.
Oh: it's only this one application from this one particular host.
>
>Or your network router(s) do garbage collection and terminate
>long-running connections to free up TCP resources?
> - thus forcing a lot of client re-connects at 4am, which may:
> a) overload the auth system/helper, or
> b) break a transaction that included nonce update for clients -
>resulting in their next request being invalid nonce.
I'm not aware of a 4AM connection-terminating task but will confirm. As mentioned (in answer to a), the auth system is just squid. Re: b - I hadn't thought of that.
>
>
>Or maybe you have log processing software that does "squid -k restart"
>instead of the proper "squid -k rotate" to get access to the log files?
>
>Or maybe you have log processing software that does "squid -k restart"
>instead of the proper "squid -k rotate" to get access to the log files?
I hadn't checked that! But logrotate runs at 00:00 and the rotate script does use "squid -k rotate".
>Or maybe your auth system has a limit on how large nonce-count can become?
> - I notice that the working request has 0x2F uses and the forbidden
>has 0x35 (suspiciously close to 50 in decimal)
I don't know what that is. I didn't find a lot of helpful detail on squid's digest auth implementation.
>
>>
>> The difference I see in the logs:? though all the digest auth info looks
>> okay, the %un field in the log for the usual (successful) request is the
>> authenticated username, while in the failed request, it's "-".?? So
>> though there hasn't been an authentication error or "authentication
>> required" in the log - and the username is in the authentication details
>> in the log entry -? it seems like squid isn't recognizing that username
>> as %un.
>
>
>Be aware that a properly behaving client will *not* send credentials
>until they are requested, after which it should *always* send
>credentials on that same connection (even if they are not requested
>explicitly).
>>
>> The difference I see in the logs:? though all the digest auth info looks
>> okay, the %un field in the log for the usual (successful) request is the
>> authenticated username, while in the failed request, it's "-".?? So
>> though there hasn't been an authentication error or "authentication
>> required" in the log - and the username is in the authentication details
>> in the log entry -? it seems like squid isn't recognizing that username
>> as %un.
>
>
>Be aware that a properly behaving client will *not* send credentials
>until they are requested, after which it should *always* send
>credentials on that same connection (even if they are not requested
>explicitly).
>That means some requests on a multiplex/pipeline/keep-alive connection
>MAY arrive with credentials and be accepted(2xx)/denied(403) without
>authentication having occured. Entirely due to your *_access directives
>sequence. In these cases the log will show auth headers but no value for
>%un and/or %ul.
>
>
>>
>> My squid.conf first tests a request to see if an unauthenticated request
>> from a particular host is permitted.? That external ACL doesn't take a
>> username as an argument.?? If that external ACL passes, the request is
>> allowed.
>>
>
>Please *show* the config lines rather than describing what you *think*
>they do. Their exact ordering matters *a lot*.
> Obfuscation of sensitive details is okay/expected so long as you makes
>it easy for us to tell that value A and value B are different.
>it easy for us to tell that value A and value B are different.
>
>FWIW; if your config file is *actually* containing only what you
>described it is missing a number of things necessary to have a safe and
>secure proxy. A look at your full config (without comments or empty
>lines) will help us point out any unnoticed issues for you to consider
>fixing.
>FWIW; if your config file is *actually* containing only what you
>described it is missing a number of things necessary to have a safe and
>secure proxy. A look at your full config (without comments or empty
>lines) will help us point out any unnoticed issues for you to consider
>fixing.
Understood. Here it is:
acl localnet src 10.0.0.0/8 # RFC1918 possible internal network
acl localnet src 172.16.0.0/12 # RFC1918 possible internal network
acl localnet src 192.168.0.0/16 # RFC1918 possible internal network
acl localnet src fc00::/7 # RFC 4193 local private network range
acl localnet src fe80::/10 # RFC 4291 link-local (directly plugged) machines
acl windows_net src 172.18.114.0/24
acl sample_host src 172.18.115.1/32
acl rsync port 873
acl SSL_ports port 443
acl SSL_ports port 873 #873 is rsync
acl Safe_ports port 80 # http
acl Safe_ports port 21 # ftp
acl Safe_ports port 443 # https
acl Safe_ports port 70 # gopher
acl Safe_ports port 210 # wais
acl Safe_ports port 1025-65535 # unregistered ports
acl Safe_ports port 280 # http-mgmt
acl Safe_ports port 488 # gss-http
acl Safe_ports port 591 # filemaker
acl Safe_ports port 777 # multiling http
acl Safe_ports port 873
acl CONNECT method CONNECT
acl PURGE method PURGE
acl localhost src 127.0.0.1
http_access allow PURGE localhost
http_access deny PURGE
acl URN proto URN
http_access deny URN
http_access deny manager
acl API_FIREFOX dstdomain api.profiler.firefox.com
http_access deny API_FIREFOX
acl ff_browser browser ^Mozilla/5\.0
acl rma_ua browser ^RMA/1\.0.*compatible;.RealMedia
uri_whitespace encode
acl trellix_phone_cloud dstdomain amcore-ens.rest.gti.trellix.com
http_access deny trellix_phone_cloud
external_acl_type host_based_filter children-max=15 ttl=0 %ACL %DATA %SRC %>rd %>rP /PATH/TO/FILTER/SCRIPT.py
acl HostBasedRules external host_based_filter
http_access allow HostBasedRules
auth_param digest program /usr/lib/squid/digest_file_auth -c /etc/squid/passwd
auth_param digest realm squid
auth_param digest children 2
auth_param basic program /usr/lib/squid/basic_ncsa_auth /etc/squid/basic_passwd
auth_param basic children 2
auth_param basic realm squidb
auth_param basic credentialsttl 2 hours
acl auth_users proxy_auth REQUIRED
external_acl_type custom_acl_db children-max=15 ttl=0 %ACL %DATA %ul %SRC %>rd %>rP %credentials /PATH/TO/FILTER/SCRIPT.py
acl CustomAclDB external custom_acl_db
http_access allow CustomAclDB
acl CRLs url_regex "/etc/squid/conf.d/CRL_urls.txt"
http_access allow CRLs
deny_info 303:https://abc.def.com/
http_access deny all
acl apache rep_header Server ^Apache
icp_access allow localnet
icp_access deny all
http_access deny !Safe_ports
http_access deny CONNECT !SSL_ports
http_access allow localhost manager
http_access deny manager
http_access allow localhost
http_port 3128
coredump_dir /var/cache/squid
refresh_pattern ^ftp: 1440 20% 10080
refresh_pattern ^gopher: 1440 0% 1440
refresh_pattern -i (/cgi-bin/|\?) 0 0% 0
refresh_pattern . 0 20% 4320
logformat squid %ts.%03tu %6tr %>a %Ss/%>Hs %<st %rm %ru %un %Sh/%<A %mt
logformat squidmime %ul %ts.%03tu %6tr %>a %Ss/%>Hs %<st %rm %ru %un %Sh/%<A %mt [%>h] [%<h]
logformat common %>a %ui %un [%tl] "%rm %ru HTTP/%rv" %>Hs %<st %Ss:%Sh
logformat combined %>a %ui %un [%tl] "%rm %ru HTTP/%rv" %>Hs %<st "%{Referer}>h" "%{User-Agent}>h" %Ss:%Sh
logformat ip_port %ts %tu %>a %>p %<lp %<a %<p %dt %tr %un %>ru %Ss
access_log daemon:/var/log/squid/access.log combined
access_log daemon:/var/log/squid/network.log ip_port
cache_log /var/log/squid/cache.log
cache_store_log /var/log/squid/store.log
access_log daemon:/var/log/squid/useragent.log useragent
visible_hostname proxy.abc.com
cache deny all >
>> The next line in squid.conf is
>>
>> acl auth_users proxy_auth REQUIRED
>>
>
>FYI the above just means that Squid is using authentication. It says
>nothing about when the authentication will be (or not be) performed.
>
>
>> ... and after that, the external ACL that takes the username as well as
>> the other info.
>>
>
>
>HTH
>Amos
_______________________________________________ squid-users mailing list squid-users@xxxxxxxxxxxxxxxxxxxxx https://lists.squid-cache.org/listinfo/squid-users
