On Thu, Feb 18, 2010 at 12:27 AM, Henrik Nordstrom <henrik@xxxxxxxxxxxxxxxxxxx> wrote: > ons 2010-02-17 klockan 21:40 -0800 skrev Tory M Blue: > >> And sorry "sleeping" was just my way of citing the box shows no load, >> almost no IO 4-5 when I'm hitting it hard. I do not see this issue >> with lesser threads, it's only when I turn up the juice. But with >> turning up the connections per second I would expect to see some type >> of load and I see none. > > Anything in /var/log/messages? > > The above problem description is almost an exact match for Linux > iptables connectiontracking table limit being hit. > > Regards > Henrik Thanks Henrik, nothing in /var/logs/messages or even dmesg and iptables Not running. No rules in place, service shutdown. Thats not the culprit, with less than 12 children at the beginning of my run 2010/02/17 10:29:51| Completed Validation Procedure 2010/02/17 10:29:51| Validated 948594 Entries 2010/02/17 10:29:51| store_swap_size = 3794376k 2010/02/17 10:29:51| storeLateRelease: released 0 objects 2010/02/18 09:53:08| squidaio_queue_request: WARNING - Queue congestion 2010/02/18 09:53:12| squidaio_queue_request: WARNING - Queue congestion 2010/02/18 09:53:17| squidaio_queue_request: WARNING - Queue congestion Even dropped my thread count and as soon as my load test starts (get maybe 10 children launched), I get the error 2010/02/18 09:56:18| squidaio_queue_request: WARNING - Queue congestion 2010/02/18 09:56:28| squidaio_queue_request: WARNING - Queue congestion Okay I've found some issues that I had not seen before, Feb 18 18:37:06 kvm0 kernel: nf_conntrack: table full, dropping packet. I would like to kick the netfilter team and fedora team in the shins. The issue was my squid boxes are virtual and the errors were being logged on the domain box (not domain as in MS). So now I'm trying to go through the system and remove all this garbage. This server does not need to track the connections and or log them. There does not seem to be a simple way to disable, just a lot of sysctl options and I'm unclear if these will do it entirely. net.ipv4.conf.all.rp_filter=0 net.ipv4.conf.all.arp_filter=0 net.ipv4.conf.default.rp_filter=0 net.ipv4.conf.default.arp_filter=0 net.ipv4.conf.lo.rp_filter=0 net.ipv4.conf.lo.arp_filter=0 net.ipv4.conf.eth0.rp_filter=0 net.ipv4.conf.eth0.arp_filter=0 net.ipv4.conf.eth1.rp_filter=0 net.ipv4.conf.eth1.arp_filter=0 net.ipv4.conf.br0.rp_filter=0 net.ipv4.conf.br0.arp_filter=0 net.ipv4.conf.br1.rp_filter=0 net.ipv4.conf.br1.arp_filter=0 net.ipv4.conf.vnet0.rp_filter=0 net.ipv4.conf.vnet0.arp_filter=0 net.ipv4.conf.vnet1.rp_filter=0 net.ipv4.conf.vnet1.arp_filter=0 But I'll be quiet here for a few, until I need assistance from the squid community. I'm still seeing the queue congestion, but if it actually doubles the thresholds each time, I may get to a good place, or can be okay to ignore the messages. Obviously the queue congestion was not causing the 500's, the dropping of packets by netfilter was Thanks Tory
