Re: Squid TCP_TUNNEL_ABORTED/200

[Jonathan Lee <jonathanlee571@xxxxxxxxx>]

Have you attempted to enable debugging ??

Researching debug_options I found you can control detailed messages in the cache.log

Sent from my iPhone

On May 3, 2024, at 10:37, Emre Oksum <emreoksum@xxxxxxxxx> wrote:



Hi Amos, thank you for your reply.

>What your "for example,..." describes is Transparent Proxy (TPROXY).
>However, what you have in the config below is very different. The IP the
>client is connected **to** (not "from") is being pinned on outgoing
>connections.

Sorry for the misunderstanding. Maybe I wasn't clear with my wording. I only need to create a proxy instance where the IPv6 address that client uses to connect to Squid, is used by Squid to connect to remote locations. In this setup, server running Squid has around 50k IPv6 addresses assigned to it and client is expected to connect to Squid proxy with 50k different IPv6 addresses of the Squid and Squid should always use the IP address client connects to it as outgoing address. I'm not sure if I explained that well.

So if client connects to Squid proxy by the address of Squid let's say is feef:1234::1, Squid should use that IP for outgoing connections. That's not transparent proxy TPROXY because client and proxy is on different networks in this setup. Just like ordinary HTTP proxies.

>The FLOW_CONTROL_ERROR is not something produced by Squid. Likely it
>comes from the TCP stack and/or OS routing system.

Client connects to Squid by a script written in Golang. Thats where they get that error. On the Squid's access.log, I can see that error as TCP_TUNNEL_ABORTED/200

>Some improvements highlighted inline below.
>Nothing stands out to me as being related to your issues.

Thank you, I'll fix them however I don't think this issue is any related to the config.

>Any particular reason not to use the registered port 3128 ?
>(Not important, just wondering.)

My client wants to prevent proxies from being detected by bots so we picked a different port number but it's not the one I shared here. I edited numbers and addresses from the config before sharing it here.

>I/we will need to see the PCAP trace along with a cache.log generated
>using "debug_options ALL,6" to confirm a bug or identify other breakage
>though.

Interestingly, debug_options ALL does not log anything related to this issue to cache.log. That left me very confused about this problem.

I'm currently sending you the PCAP file. It's being uploaded. I would be appreciated if you can take a look at it.

Thanks

Amos Jeffries <squid3@xxxxxxxxxxxxx>, 3 May 2024 Cum, 19:31 tarihinde şunu yazdı:

On 4/05/24 02:29, Emre Oksum wrote:
> Hi everyone,
>
> I'm having a issue with Squid Cache 4.10 which I cannot fix for weeks
> now and kinda lost at the moment. I will be appreciated if someone can
> guide me through the issue I'm having.
> I need to create a IPv6 HTTP proxy which should match the entry address
> to outgoing TCP address. For example, if user is connecting from
> fe80:abcd::1 it should exit the HTTP proxy from the same address. We got
> like 50k addresses like this at the moment.

What your "for example,..." describes is Transparent Proxy (TPROXY).


However, what you have in the config below is very different. The IP the
client is connected **to** (not "from") is being pinned on outgoing
connections.


> The issue is, client connecting to the proxy is receiving "EOF" or
> "FLOW_CONTROL_ERROR" on their side.

The FLOW_CONTROL_ERROR is not something produced by Squid. Likely it
comes from the TCP stack and/or OS routing system.

The EOF may be coming from either Squid or the OS. It also may be
perfectly normal for the circumstances, or a side effect of an error
elsewhere.


To solve will require identifying exactly what is sending those signals,
and why. Since they are signals going to the client, focus on the
client->Squid connections (not the Squid->server ones you talk about
testing below).



> When I test connection by connecting
> to whatismyip.com <http://whatismyip.com> everything works fine and
> entry IP always matches with outgoing IP for each of the 50k addresses.
> Client tells me this problem occurs both at GET and POST requests with
> around 10 MB of data.

Well, you are trying to manually force certain flow patterns that
prohibit or break some major HTTP performance features. Some problems
are to be expected.

The issues which I expect to occur in your proxy would not show up in a
trivial outgoing-IP or connectivity test.


> I initially thought that could be related to server resources being
> drained but upon inspecting server resource usage, Squid isn't even
> topping at 100% CPU or RAM anytime so not that.
>

IMO, "FLOW_CONTROL_ERROR" is likely related to quantity of traffic
flooding through the proxy to specific origin servers.

The concept you are implementing of the outgoing TCP connection having
the same IP as the incoming connection reduces the available TCP sockets
by 25%. Prohibiting the OS from allocating ports on otherwise unused
outgoing addresses when



> My Squid.conf is like this at the moment:

Some improvements highlighted inline below.
Nothing stands out to me as being related to your issues.

>
> auth_param basic program /usr/lib/squid/basic_ncsa_auth /etc/squid/passwd
> acl auth_users proxy_auth REQUIRED
> http_access allow auth_users
> http_access deny !auth_users

Above two lines are backwards. Deny first, then allow.


> cache deny all
> dns_nameservers <nameservers here>
> dns_v4_first off
> via off
> forwarded_for delete
> follow_x_forwarded_for deny all
> server_persistent_connections off

*If* the issue turns out to be congestion on Squid->server connections
enabling this might be worthwhile. Otherwise it should be fine.


> max_filedesc 1048576

You can remove that line. "max_filedesc" was a RedHat hack from 20+
years ago when the feature was experimental.

Any value you set on the line above, will be erased and replaced by the
line below:


> max_filedescriptors 1048576
> workers 8
> http_port [::0]:1182

Above is just a complicated way to write:

  http_port 1182


Any particular reason not to use the registered port 3128 ?
(Not important, just wondering.)


> acl binding1 myip fe80:abcd::1
> tcp_outgoing_address fe80:abcd::1 binding1
> acl binding2 myip fe80:abcd::2
> tcp_outgoing_address fe80:abcd::2 binding2
> acl binding3 myip fe80:abcd::3
> tcp_outgoing_address fe80:abcd::3 binding3
> ...
> ...
> ...
> access_log /var/log/squid/access.log squid

> cache_store_log none

You can erase this line.
This is default setting. No need to manually set it.


> cache deny all

You can erase this line.
This "cache deny all" exists earlier in the config.


>
> I've tried to get a PCAP file and realized when client tries to connect
> with a new IPv6 address, Squid is not trying to open a new connection
> instead tries to resume a previously opened one on a different outgoing
> IPv6 address.

Can you provide the trace demonstrating that issue?

Although, as noted earlier your problems are apparently on the client
connections. This is about server connections behaviour.


> I set server_persistent_connections off which should have
> disabled this behavior but it's still the same.

Nod. Yes that should forbid re-use of connections.

I/we will need to see the PCAP trace along with a cache.log generated
using "debug_options ALL,6" to confirm a bug or identify other breakage
though.



> I tried using a newer
> version of Squid but it behaved differently and did not follow my
> outgoing address specifications and kept connecting on IPv4.

That would seem to indicate that your IPv4 connectivity is better than
your IPv6 connectivity. Later Squid use various "Happy Eyeballs"
implementations for the server selection.

You can usually work around this by configuring the DNS server specified
by dns_nameservers to only deliver IPv6 results when a mixed set are
available.


HTH
Amos
_______________________________________________
squid-users mailing list
squid-users@xxxxxxxxxxxxxxxxxxxxx
https://lists.squid-cache.org/listinfo/squid-users

_______________________________________________
squid-users mailing list
squid-users@xxxxxxxxxxxxxxxxxxxxx
https://lists.squid-cache.org/listinfo/squid-users

_______________________________________________
squid-users mailing list
squid-users@xxxxxxxxxxxxxxxxxxxxx
https://lists.squid-cache.org/listinfo/squid-users