Re: Squid marks alive siblings as dead.

[Tony Dodd <tony@xxxxxxx>]

Thanks for the suggestion, Mike.  I tried adding:

icp_query_timeout 5000
dead_peer_timeout 10 seconds

To my configs, but this just resulted in it taking longer for the 
instance to mark the sibling as dead.  I'm assuming, however, that 
icp_query_timeout does absolutely nothing for htcp queries?

As a side-note, if I run the same neighbours as ICP, they work 
correctly; unfortunately, this isn't really a valid option for me, as 
pretty much everything we cache varies on headers.

Thanks!
Tony

leongmzlist wrote:
> Try increasing the ICP/HTCP response time.  The auto timeout detection 
> doesn't work too well for me, and my peers are on the same switch.
>
> mike
>
> At 06:01 PM 10/11/2007, Tony Dodd wrote:
> > Hey All,
> >
> > Been working on rolling out HTCP cache_peer relationships within my
> > squid cluster, but I'm running into a 'small' issue.  After starting
> > squid with the cache_peers configured to use htcp, squid sends quite a
> > few UDP packets to the siblings on the htcp port, however, it then marks
> > them as dead.
> >
> > Ideas/help would be appreciated!
> >
> > Below are tcpdumps, and the config I'm utilizing.
> >
> > Cache8 tcpdump from squid startup on cache1 to cache8 being marked as 
> > dead:
> >
> > 00:52:41.512445 IP cache1.int.last.fm.4827 > cache8.int.last.fm.4827:
> > UDP, length 921
> > 00:52:41.807481 IP cache1.int.last.fm.4827 > cache8.int.last.fm.4827:
> > UDP, length 789
> > 00:52:41.807995 IP cache1.int.last.fm.4827 > cache8.int.last.fm.4827:
> > UDP, length 346
> > 00:52:41.808106 IP cache1.int.last.fm.4827 > cache8.int.last.fm.4827:
> > UDP, length 354
> > 00:52:41.808227 IP cache1.int.last.fm.4827 > cache8.int.last.fm.4827:
> > UDP, length 362
> > 00:52:41.968961 IP cache1.int.last.fm.4827 > cache8.int.last.fm.4827:
> > UDP, length 322
> > 00:52:42.023668 IP cache1.int.last.fm.4827 > cache8.int.last.fm.4827:
> > UDP, length 789
> > 00:52:42.173640 IP cache1.int.last.fm.4827 > cache8.int.last.fm.4827:
> > UDP, length 299
> > 00:52:42.277635 IP cache1.int.last.fm.4827 > cache8.int.last.fm.4827:
> > UDP, length 789
> > 00:52:42.641855 IP cache1.int.last.fm.4827 > cache8.int.last.fm.4827:
> > UDP, length 485
> > 00:52:42.790272 IP cache1.int.last.fm.4827 > cache8.int.last.fm.4827:
> > UDP, length 366
> > 00:52:42.795551 IP cache1.int.last.fm.4827 > cache8.int.last.fm.4827:
> > UDP, length 350
> > 00:52:42.898732 IP cache1.int.last.fm.4827 > cache8.int.last.fm.4827:
> > UDP, length 292
> > 00:52:43.293162 IP cache1.int.last.fm.4827 > cache8.int.last.fm.4827:
> > UDP, length 296
> > 00:52:43.627891 IP cache1.int.last.fm.4827 > cache8.int.last.fm.4827:
> > UDP, length 284
> > 00:52:44.394020 IP cache1.int.last.fm.4827 > cache8.int.last.fm.4827:
> > UDP, length 366
> > 00:52:44.675878 IP cache1.int.last.fm.4827 > cache8.int.last.fm.4827:
> > UDP, length 321
> > 00:52:45.241068 IP cache1.int.last.fm.4827 > cache8.int.last.fm.4827:
> > UDP, length 940
> > 00:52:47.834801 IP cache1.int.last.fm.4827 > cache8.int.last.fm.4827:
> > UDP, length 349
> > 00:52:48.947938 IP cache1.int.last.fm.4827 > cache8.int.last.fm.4827:
> > UDP, length 302
> > 00:52:49.048647 IP cache1.int.last.fm.4827 > cache8.int.last.fm.4827:
> > UDP, length 321
> > 00:52:49.120675 IP cache1.int.last.fm.4827 > cache8.int.last.fm.4827:
> > UDP, length 567
> > 00:52:49.324587 IP cache1.int.last.fm.4827 > cache8.int.last.fm.4827:
> > UDP, length 363
> > 00:52:50.250438 IP cache1.int.last.fm.4827 > cache8.int.last.fm.4827:
> > UDP, length 370
> > 00:52:51.396922 IP cache1.int.last.fm.4827 > cache8.int.last.fm.4827:
> > UDP, length 318
> > 00:52:51.512461 IP cache1.int.last.fm.4827 > cache8.int.last.fm.4827:
> > UDP, length 935
> > 00:52:51.927925 IP cache1.int.last.fm.4827 > cache8.int.last.fm.4827:
> > UDP, length 349
> > 00:53:02.172733 IP cache1.int.last.fm.4827 > cache8.int.last.fm.4827:
> > UDP, length 771
> >
> > Cache1 tcpdump from squid startup on cache1 to cache8 being marked as 
> > dead:
> >
> > 00:52:41.506998 IP cache1.int.last.fm.4827 > cache8.int.last.fm.4827:
> > UDP, length 921
> > 00:52:41.802035 IP cache1.int.last.fm.4827 > cache8.int.last.fm.4827:
> > UDP, length 789
> > 00:52:41.802574 IP cache1.int.last.fm.4827 > cache8.int.last.fm.4827:
> > UDP, length 346
> > 00:52:41.802686 IP cache1.int.last.fm.4827 > cache8.int.last.fm.4827:
> > UDP, length 354
> > 00:52:41.802807 IP cache1.int.last.fm.4827 > cache8.int.last.fm.4827:
> > UDP, length 362
> > 00:52:41.963543 IP cache1.int.last.fm.4827 > cache8.int.last.fm.4827:
> > UDP, length 322
> > 00:52:42.018232 IP cache1.int.last.fm.4827 > cache8.int.last.fm.4827:
> > UDP, length 789
> > 00:52:42.168224 IP cache1.int.last.fm.4827 > cache8.int.last.fm.4827:
> > UDP, length 299
> > 00:52:42.272199 IP cache1.int.last.fm.4827 > cache8.int.last.fm.4827:
> > UDP, length 789
> > 00:52:42.636431 IP cache1.int.last.fm.4827 > cache8.int.last.fm.4827:
> > UDP, length 485
> > 00:52:42.784852 IP cache1.int.last.fm.4827 > cache8.int.last.fm.4827:
> > UDP, length 366
> > 00:52:42.790131 IP cache1.int.last.fm.4827 > cache8.int.last.fm.4827:
> > UDP, length 350
> > 00:52:42.893315 IP cache1.int.last.fm.4827 > cache8.int.last.fm.4827:
> > UDP, length 292
> > 00:52:43.287743 IP cache1.int.last.fm.4827 > cache8.int.last.fm.4827:
> > UDP, length 296
> > 00:52:43.622474 IP cache1.int.last.fm.4827 > cache8.int.last.fm.4827:
> > UDP, length 284
> > 00:52:44.388599 IP cache1.int.last.fm.4827 > cache8.int.last.fm.4827:
> > UDP, length 366
> > 00:52:44.670459 IP cache1.int.last.fm.4827 > cache8.int.last.fm.4827:
> > UDP, length 321
> > 00:52:45.235623 IP cache1.int.last.fm.4827 > cache8.int.last.fm.4827:
> > UDP, length 940
> > 00:52:47.829382 IP cache1.int.last.fm.4827 > cache8.int.last.fm.4827:
> > UDP, length 349
> > 00:52:48.942519 IP cache1.int.last.fm.4827 > cache8.int.last.fm.4827:
> > UDP, length 302
> > 00:52:49.043228 IP cache1.int.last.fm.4827 > cache8.int.last.fm.4827:
> > UDP, length 321
> > 00:52:49.115246 IP cache1.int.last.fm.4827 > cache8.int.last.fm.4827:
> > UDP, length 567
> > 00:52:49.319166 IP cache1.int.last.fm.4827 > cache8.int.last.fm.4827:
> > UDP, length 363
> > 00:52:50.245010 IP cache1.int.last.fm.4827 > cache8.int.last.fm.4827:
> > UDP, length 370
> > 00:52:51.391500 IP cache1.int.last.fm.4827 > cache8.int.last.fm.4827:
> > UDP, length 318
> > 00:52:51.507016 IP cache1.int.last.fm.4827 > cache8.int.last.fm.4827:
> > UDP, length 935
> > 00:52:51.922504 IP cache1.int.last.fm.4827 > cache8.int.last.fm.4827:
> > UDP, length 349
> > 00:53:02.167290 IP cache1.int.last.fm.4827 > cache8.int.last.fm.4827:
> > UDP, length 771
> >
> > Cache1 cache.log from restart to cache8 being marked dead:
> >
> > 2007/10/12 00:52:41| Reconfiguring Squid Cache (version 2.6.STABLE14)...
> > 2007/10/12 00:52:41| FD 339 Closing HTTP connection
> > 2007/10/12 00:52:41| Closing Pinger socket on FD 344
> > 2007/10/12 00:52:41| FD 340 Closing ICP connection
> > 2007/10/12 00:52:41| FD 341 Closing HTCP socket
> > 2007/10/12 00:52:41| FD 343 Closing SNMP socket
> > 2007/10/12 00:52:41| Cache dir '/var/spool/squid-int' size remains
> > unchanged at 104857600 KB
> > 2007/10/12 00:52:41| User-Agent logging is disabled.
> > 2007/10/12 00:52:41| DNS Socket created at 0.0.0.0, port 32825, FD 219
> > 2007/10/12 00:52:41| Adding nameserver 10.0.6.12 from squid.conf
> > 2007/10/12 00:52:41| Adding nameserver 10.0.0.12 from squid.conf
> > 2007/10/12 00:52:41| Accepting accelerated HTTP connections at 0.0.0.0,
> > port 8081, FD 225.
> > 2007/10/12 00:52:41| Accepting ICP messages at 0.0.0.0, port 3131, FD 
> > 228.
> > 2007/10/12 00:52:41| Accepting HTCP messages on port 4827, FD 233.
> > 2007/10/12 00:52:41| Accepting SNMP messages on port 3402, FD 244.
> > 2007/10/12 00:52:41| WCCP Disabled.
> > 2007/10/12 00:52:41| Pinger socket opened on FD 288
> > 2007/10/12 00:52:41| Configuring Parent 10.0.0.15/80/0
> > 2007/10/12 00:52:41| Configuring Parent 10.0.20.1/81/0
> > 2007/10/12 00:52:41| Configuring Parent 10.0.20.1/82/0
> > 2007/10/12 00:52:41| Configuring Parent 10.0.20.1/83/0
> > 2007/10/12 00:52:41| Configuring Parent 10.0.20.1/84/0
> > 2007/10/12 00:52:41| Configuring Parent 10.0.20.2/81/0
> > 2007/10/12 00:52:41| Configuring Parent 10.0.20.2/82/0
> > 2007/10/12 00:52:41| Configuring Parent 10.0.20.2/83/0
> > 2007/10/12 00:52:41| Configuring Parent 10.0.20.2/84/0
> > 2007/10/12 00:52:41| Configuring Parent 10.0.0.35/80/0
> > 2007/10/12 00:52:41| Configuring Parent 10.0.0.35/81/0
> > 2007/10/12 00:52:41| Configuring Parent 10.0.0.114/80/0
> > 2007/10/12 00:52:41| Configuring Parent 10.0.0.114/81/0
> > 2007/10/12 00:52:41| Configuring Parent 10.0.0.114/82/0
> > 2007/10/12 00:52:41| Configuring Parent 10.0.0.114/83/0
> > 2007/10/12 00:52:41| Configuring Sibling 10.0.12.8/8081/4827
> > 2007/10/12 00:52:41| Loaded Icons.
> > 2007/10/12 00:52:41| Ready to serve requests.
> > 2007/10/12 00:53:02| Detected DEAD Sibling: 10.0.12.8
> >
> >
> >
> > Global squid configuration (on deployment, things that aren't valid for
> > a particular box - i.e. having itself in the squid.conf as a cache_peer
> > - are removed):
> >
> > http_port 8081 accel vhost
> > hosts_file /etc/hosts
> > dns_nameservers 10.0.6.12 10.0.0.12
> > #mime_table /etc/squid-int/mime.conf
> > cache_effective_user proxy
> > cache_replacement_policy heap GDSF
> > cache_swap_low 90
> > cache_swap_high 95
> > coredump_dir /var/spool/squid-int
> > cache_dir aufs /var/spool/squid-int 102400 16 256
> > cache_mem 200 MB
> > maximum_object_size_in_memory 50 KB
> > access_log /var/log/squid-int/access.log
> > cache_log /var/log/squid-int/cache.log
> > cache_store_log /var/log/squid-int/store.log
> > #visible_hostname cache2.int.last.fm
> > pid_filename /var/run/squid-int.pid
> > logfile_rotate 10
> > memory_pools off
> > maximum_object_size 50 MB
> > quick_abort_min 0 KB
> > quick_abort_max 0 KB
> > log_icp_queries off
> > client_db off
> > half_closed_clients off
> > query_icmp on
> > pinger_program /usr/bin/pinger
> > netdb_ping_period 120 second
> > test_reachability on
> > digest_generation on
> > digest_bits_per_entry 4
> > digest_rebuild_period 2 hours
> > digest_rewrite_period 2 hours
> > digest_swapout_chunk_size 16 KB
> > digest_rebuild_chunk_percentage 5
> > prefer_direct off
> >
> > acl all src 0.0.0.0/0.0.0.0
> > acl squiddies src 10.0.0.0/255.0.0.0
> > acl static dstdomain static.last.fm
> > acl userserve dstdomain userserve.last.fm
> > acl ws dstdomain ws.audioscrobbler.com
> > acl mainsite dstdom_regex www.last.*fm.* cn.last.fm lastfm.spiegel.de
> > ws.audioscrobbler.com ext.last.fm
> > http_access allow static
> > http_access allow ws
> > http_access allow userserve
> > http_access allow mainsite
> > http_access allow squiddies
> > http_access deny all
> >
> > miss_access allow all
> > miss_access deny squiddies
> >
> > icp_access allow squiddies
> > icp_port 3131
> >
> > htcp_port 4827
> >
> > ###Static
> > cache_peer 10.0.0.15 parent 80 0 no-query originserver no-digest
> > no-netdb-exchange name=static
> > cache_peer_domain static static.last.fm
> > ###Static ends
> >
> > ###Userserve
> > cache_peer 10.0.20.1 parent 81 0 no-query originserver no-digest
> > no-netdb-exchange name=userserve1-81 round-robin
> > cache_peer_domain userserve1-81 userserve.last.fm
> >
> > cache_peer 10.0.20.1 parent 82 0 no-query originserver no-digest
> > no-netdb-exchange name=userserve1-82 round-robin
> > cache_peer_domain userserve1-82 userserve.last.fm
> >
> > cache_peer 10.0.20.1 parent 83 0 no-query originserver no-digest
> > no-netdb-exchange name=userserve1-83 round-robin
> > cache_peer_domain userserve1-83 userserve.last.fm
> >
> > cache_peer 10.0.20.1 parent 84 0 no-query originserver no-digest
> > no-netdb-exchange name=userserve1-84 round-robin
> > cache_peer_domain userserve1-84 userserve.last.fm
> >
> > cache_peer 10.0.20.2 parent 81 0 no-query originserver no-digest
> > no-netdb-exchange name=userserve2-81 round-robin
> > cache_peer_domain userserve2-81 userserve.last.fm
> >
> > cache_peer 10.0.20.2 parent 82 0 no-query originserver no-digest
> > no-netdb-exchange name=userserve2-82 round-robin
> > cache_peer_domain userserve2-82 userserve.last.fm
> >
> > cache_peer 10.0.20.2 parent 83 0 no-query originserver no-digest
> > no-netdb-exchange name=userserve2-83 round-robin
> > cache_peer_domain userserve2-83 userserve.last.fm
> >
> > cache_peer 10.0.20.2 parent 84 0 no-query originserver no-digest
> > no-netdb-exchange name=userserve2-84 round-robin
> > cache_peer_domain userserve2-84 userserve.last.fm
> > ###Userserve Ends
> >
> >
> > ###ws.audioscrobbler.com & mainsite
> > cache_peer 10.0.0.35 parent 80 0 no-query originserver no-digest
> > no-netdb-exchange name=perlbal1-80 round-robin
> > cache_peer_access perlbal1-80 allow mainsite
> >
> > cache_peer 10.0.0.35 parent 81 0 no-query originserver no-digest
> > no-netdb-exchange name=perlbal1-81 round-robin
> > cache_peer_access perlbal1-81 allow mainsite
> >
> > cache_peer 10.0.0.114 parent 80 0 no-query originserver no-digest
> > no-netdb-exchange name=perlbal2-80 round-robin
> > cache_peer_access perlbal2-80 allow mainsite
> >
> > cache_peer 10.0.0.114 parent 81 0 no-query originserver no-digest
> > no-netdb-exchange name=perlbal2-81 round-robin
> > cache_peer_access perlbal2-81 allow mainsite
> >
> > cache_peer 10.0.0.114 parent 82 0 no-query originserver no-digest
> > no-netdb-exchange name=perlbal2-82 round-robin
> > cache_peer_access perlbal2-82 allow mainsite
> >
> > cache_peer 10.0.0.114 parent 83 0 no-query originserver no-digest
> > no-netdb-exchange name=perlbal2-83 round-robin
> > cache_peer_access perlbal2-83 allow mainsite
> > ###ws.audioscrobbler.com & mainsite Ends
> >
> > snmp_port 3402
> > acl snmppublic snmp_community public
> > snmp_access allow snmppublic
> > snmp_incoming_address 0.0.0.0
> > snmp_outgoing_address 255.255.255.255
> >
> > cache_peer 10.0.12.1 neighbour 8081 4827 round-robin htcp
> > cache_peer 10.0.12.2 neighbour 8081 4827 round-robin htcp
> > cache_peer 10.0.12.3 neighbour 8081 4827 round-robin htcp
> > cache_peer 10.0.12.4 neighbour 8081 4827 round-robin htcp
> > cache_peer 10.0.12.5 neighbour 8081 4827 round-robin htcp
> > cache_peer 10.0.12.6 neighbour 8081 4827 round-robin htcp
> > cache_peer 10.0.12.7 neighbour 8081 4827 round-robin htcp
> > cache_peer 10.0.12.8 neighbour 8081 4827 round-robin htcp
> >
> >
> > Thanks guys
> > --
> > Tony Dodd, Systems Administrator
> >
> > Last.fm | http://www.last.fm
> > Karen House 1-11 Baches Street
> > London N1 6DL
> >
> > check out my music taste at:
> > http://www.last.fm/user/hawkeviper

--
Tony Dodd, Systems Administrator

Last.fm | http://www.last.fm
Karen House 1-11 Baches Street
London N1 6DL

check out my music taste at:
http://www.last.fm/user/hawkeviper