Some assumptions: 1) You are using a Cisco Router to redirect traffic to the squid box via WCCP 2) 12.4(15)T8 or higher IOS on the router 3) In my setups, the squid box is always Layer 2 adjacent to the Cisco router, either through a dedicated interface, or a sub-interface. 4) The ability to compile and install a Linux kernel. Please note that in these steps, I am NOT using a redhat kernel, nor am I using the RedHat method of building a kernel. 5) Some steps outlined here can be achieved through several different means, follow the steps exactly before emailing me or the list, as I have tested other methods, and they don't always work (case in point: GRE tunnel interface creation.) 6) This setup assumes a separate WCCP service group for each direction of the HTTP connection, this is not needed, but makes the setup more scalable. If you choose to do it a different way, then YMMV. In the kernel build specific steps, I actually include possibly to much information, as well as tell you to enable things that are not always needed for TPROXY related functionality, or never related to TPROXY functionality. I included them because they fit more environments, and thus less time wasted by people asking me questions, not that I mind but I don't have enough time to answer all the emails I get. I tried to prepare this information out without errors, if the steps don't work, email me with the details of where you had problems so that I can adjust the steps below. At the end the steps below are some common things to watch for in the steps that can cause the setup not to work. Steps 1) Install CentOS 5.3, make sure you install nothing other than the base packages, and trim even those down. I tend to install specific packages from the distro later. Note: I suggest that you make separate partition(s) for where squid will actually store its caches. Later mount these partitions with specific options (like "noatime") that will help increase performance. 2) In the initial ncurses-based setup screen, turn off services that you don't need, and turn off selinux compeletely. 3) After install and initial bootup and configuration, run "yum update" to update the system for fixes, etc. Then reboot. 4) After step 2, issue this yum command: yum install libcap libcap-devel gcc gcc-c++ bison flex yacc autoconf automake ncurses ncurses-devel rpm-devel libpcap tcpdump Note: let it install other dependency packages. The command above installs compiles, utilities, etc. 5) Download iptables-1.4.3.2 from netfilter.org 6) Download kernel 2.6.30 from kernel.org 7) Download squid-3.1.0.8 from squid.org 8) Decompress the kernel source, I decompress it to /usr/src/, although I have read all over the place that this is a bad thing to do. The location really does not have to be /usr/src/ 9) Go into the kernel source directory, issue the following command: cp /boot/config-2.6.18-128.1.14.el5 ./RH-config-boxed.config 10) Issue this command: make menuconfig 11) When the ncurses-based kernel config screen loads, select the "Load an Alternate Configuration File" and type in the full path to the RH-config-boxed.config. This will load the current kernel config, and there may be some errors, all of which can be ignored. 12) Configure the kernel as you normally would, but be sure to enable the following: In "Networking support -> Networking options" Enable (not as modules): Packet socket Packet socket: mmapped IO TCP/IP networking IP: advanced router IP: policy routing Enable (as modules): IP: tunneling Enable (not as modules): IP: GRE tunnels over IP IP: broadcast GRE over IP Network packet filtering framework (Netfilter) In "Networking support -> Networking options -> Network packet filtering framework (Netfilter)" Enable (not as modules): Advanced netfilter configuration In "Networking support -> Networking options -> Network packet filtering framework (Netfilter) -> Core Netfilter Configuration" Enable (as modules): Netfilter connection tracking support Enable (not as modules): Connection tracking security mark support Connection tracking events Enable (as modules): Connection tracking netlink interface Transparent proxying support (EXPERIMENTAL) Netfilter Xtables support (required for ip_tables) "CONNMARK" target support "MARK" target support "TPROXY" target support (EXPERIMENTAL) "connmark" connection mark match support "conntrack" connection tracking match support "mark" match support "socket" match support (EXPERIMENTAL) "state" match support In "Networking support -> Networking options -> Network packet filtering framework (Netfilter) -> IP: Netfilter Configuration" Enable (as modules): IPv4 connection tracking support (required for NAT) IP tables support (required for filtering/masq/NAT) Full NAT MASQUERADE target support REDIRECT target support Packet mangling 13) After setting the above options, and any other items you want, exit out of the kernel config, saving your changes. It will save the kernel compile config to RH-config-boxed.config so issue the following command to put the new config in the right place: cp RH-config-boxed.config config-centos Then do the make, make_install_modules, make install, if no errors, adjust grub.conf to boot to the new kernel. The reboot to the new kernel. 14) Assuming the kernel compiled, installed and booted properly, it is time to update iptables. Decompress the iptables source, and use the following configure command: ./configure --enable-devel --enable-libipq --bindir=/bin --sbindir=/sbin --sysconfdir=/etc --with-kernel=<path to new kernel source dir> --with-kbuild=<path to new kernel source dir> --with-ksource=<path to new kernel source dir> Then do the make, make install 15) Edit ld.so.conf to add a library path: vim /etc/ld.so.conf add a line to the end of the file: /usr/local/lib 16) I suggest rebooting, just to make sure that iptables upgrade is working. Do a reboot, then do a "service iptables status" to make sure that iptables is running fine. If it is not running ok, it will show either a status failed, or an empty rule set. 17) Assuming that the iptables upgrade is working fine, the next steps are to add the rules and interfaces needed for WCCP and TPROXY functionality. First, issue a "insmod ip_gre", then do a "dmesg | tail" and make sure that you see "GRE over IPv4 tunneling driver" if you don't see anything, or you get an error from insmod, you either compiled GRE not as a kernel module, or not at all. 18) Create the GRE tunnel interface, issue "ifconfig gre0 <ip address of squid server> netmask <netmask of squid server ip> up" 19) Issue an "ifconfig" and make sure you see a gre0 interface. 20) Next we need to add iptables rules to allow traffic to the gre interface itself, gre protocol traffic across the Ethernet interface, and WCCP traffic from the router. We will do this by editing the iptables save file directly. Do the following: a.) service iptables save b.) vim /etc/sysconfig/iptables c.) just after the line that says ":RH-Firewall-1-INPUT - [0:0]" add the following: -A INPUT -i gre0 -j ACCEPT -A INPUT -p gre -j ACCEPT -A INPUT -i eth0 -p gre -j ACCEPT d.) Somewhere lower in the file where the "-A RH-Firewall-1-INPUT" rules are, add the following: -A RH-Firewall-1-INPUT -s <address of cisco router>/32 -p udp -m udp --dport 2048 -j ACCEPT NOTE: The "<address of cisco router>" in this instance is the ip address of the router that the squid box itself uses. Port 2048 is the port that WCCP traffic uses. e.) save the file and exit vim. Do a "service iptables restart" to make sure you don't get any errors. 21) Next we add the TPROXY related iptables rules, from the command line prompt issue the following commands: iptables -t mangle -N DIVERT iptables -t mangle -A DIVERT -j MARK --set-mark 1 iptables -t mangle -A DIVERT -j ACCEPT iptables -t mangle -A PREROUTING -p tcp -m socket -j DIVERT iptables -t mangle -A PREROUTING -p tcp --dport 80 -j TPROXY --tproxy-mark 0x1/0x1 --on-port <port squid listens to> --on-ip <squid server ip> 22) Assuming the commands in step 21 didn't give errors, do a "service iptables save" 23) We need to edit the iptables rule order, to make sure it is in the correct order: "vim /etc/sysconfig/iptables" and make sure the section at the bottom of the file, from the ":DIVERT - [0:0]" onward looks something like: -A DIVERT -j MARK --set-xmark 0x1/0xffffffff -A DIVERT -j ACCEPT -A PREROUTING -p tcp -m socket -j DIVERT -A PREROUTING -p tcp -m tcp --dport 80 -j TPROXY --on-port <squid port> --on-ip <squid server ip> --tproxy-mark 0x1/0x1 When doing the cli commands in step 21, iptables puts the rules in the wrong order for them to work for TPROXY. Save the file and exit vim. 24) do a "service iptables restart" 26) Issue the following commands, per the TPROXY wiki article: ip rule add fwmark 1 lookup 100 ip route add local 0.0.0.0/0 dev lo table 100 If no errors from the two commands, add them to the end of /etc/rc.d/rc.local 27) I'm not sure this is needed anymore, but it doesn't seem to break anything, so add the following line to the end of /etc/rc.d/rc.local: echo 1 > /proc/sys/net/ipv4/ip_nonlocal_bind 28) In the beginning of the rc.local file, add the following two lines: modprobe ip_gre <the ifconfig gre0 command line from step 18 above> 29) Edit /etc/sysctl.conf, make sure there is a line somewhere that says: # Controls IP packet forwarding net.ipv4.ip_forward = 1 30) Reboot the server 31) Next, build you squid 3.1.x.x, make sure that pass "--enable-linux-netfilter" to the configure directive, build and install squid. 32) You will need to edit the squid config to make sure that it has what it needs to use TPROXY and WCCP. For tproxy, you need one item, the "http_port" directive needs tproxy appended to it, I do the following: http_port <ip of squid server>:<port to bind to, "squid port" from step 23 above> tproxy disable-pmtu-discovery=always 33) Configure the WCCP specific related items in the squid config file, specifically: a) wccp2_router <router address used in the steps above> b) wccp_version 2 c) wccp2_rebuild_wait on d) wccp2_fowarding_method 1 e) wccp2_return_method 1 f) wccp2_assigment_method 1 g) wccp2_service dynamic 80 h) wccp2_service dynamic 90 i) wccp2_service_info 80 protocol=tcp flags=src_ip_hash priority=240 ports=80 j) wccp2_service_info 90 protocol=tcp flags=dst_ip_hash,ports_source priority=240 ports=80 34) There are other squid configuration items that are need which are not included here because they are not specific to WCCP/TPROXY functionality and the specifics vary by environment. 35) In the router configuration, you need to do at least the following: a.) enable wccp globally with (this might actually be optional): ip wccp web-cache b.) enable the specific services: 1.) ip wccp 80 2.) ip wccp 90 Note: I use a redirect list ACL with the two commands above so that the router doesn't WCCP redirect specific web sites. The command would look like "ip wccp 80 redirect-list 122", and access-list 122 would be a list of denies for sites to not be redirected, and a permit any any at the end of the access-list to allow all other websites to be wccp redirected. c.) On the interface that is adjacent to the squid box, do a "ip wccp redirect exclude in" this command makes it so that the router does not redirect the squid traffic as well as other client traffic. Some things to check along the way: 1) Make sure that the GRE interface on the squid box is seeing packets coming in. You should never see any packets going out the GRE interface, and you will only see packets coming in the GRE interface after the WCCP process on the router redirects them to the squid box. 2) Keep in mind that you should separate the idea in your mind between WCCP traffic and tunneled HTTP traffic. The router and the squid process talk to each other for status and service information with WCCP. HTTP traffic received from the client by the router for redirect is encapsulated in GRE (to preserve it) and then forwarded to the GRE interface. 3) The global command "show ip wccp" is a useful router command. In the output of this command, you should see two "Service Identifier" sections (one for 80, one for 90 if you use my setup steps.) Within each "Service Identifier" group, the number of service group clients and routers should each be 1. If they are not, them some facet of WCCP conversation between the router and the squid server is not working. Check the iptables port 2048 setup step 20 part d above. 4) wccp event debug commands are useful. Use "debug ip wccp events" and set you router logging to debug level to see "Hello" and "Here_I_am" packets. You need to make sure you see both to insure that the router and squid box are WCCP talking. 5) Surf the Web from a client and see if you get to a web site. If it doesn't work, check the items above, recheck the steps. It if does work, go to a site that tells you your IP to make sure TPROXY is working and using the client IP and not the squid IP. It is entirely possible that the whole setup will work, but not the client IP spoofing. I also suggest that you burn the setup in with web surfing to make sure it doesn't break. I am interested to here people's feedback so that I can improve the steps above, as well as share optimizations. Nick
