Amos did this already....although the wiki article needs some corrections because Amos merged the older with the newer. I need to get that information to him. The steps, if followed in the wiki article may not work quite right. Nick -----Original Message----- From: Henrik Nordstrom [mailto:henrik@xxxxxxxxxxxxxxxxxxx] Sent: Monday, June 29, 2009 4:01 PM To: Ritter, Nicholas Cc: squid-users Subject: Re: Updated CentOS/Squid/Tproxy Transparency steps. Would be great if you could dump this in a wiki article to make it easier to find (and update if needed). Regards Henrik tor 2009-06-25 klockan 12:30 -0500 skrev Ritter, Nicholas: > Some assumptions: > > 1) You are using a Cisco Router to redirect traffic to the squid box via > WCCP > > 2) 12.4(15)T8 or higher IOS on the router > > 3) In my setups, the squid box is always Layer 2 adjacent to the Cisco > router, either through a dedicated interface, or a sub-interface. > > 4) The ability to compile and install a Linux kernel. Please note that > in these steps, I am NOT using a redhat kernel, nor am I using the > RedHat method of building a kernel. > > 5) Some steps outlined here can be achieved through several different > means, follow the steps exactly before emailing me or the list, as I > have tested other methods, and they don't always work (case in point: > GRE tunnel interface creation.) > > 6) This setup assumes a separate WCCP service group for each direction > of the HTTP connection, this is not needed, but makes the setup more > scalable. If you choose to do it a different way, then YMMV. > > In the kernel build specific steps, I actually include possibly to much > information, as well as tell you to enable things that are not always > needed for TPROXY related functionality, or never related to TPROXY > functionality. I included them because they fit more environments, and > thus less time wasted by people asking me questions, not that I mind but > I don't have enough time to answer all the emails I get. I tried to > prepare this information out without errors, if the steps don't work, > email me with the details of where you had problems so that I can adjust > the steps below. At the end the steps below are some common things to > watch for in the steps that can cause the setup not to work. > > > Steps > > 1) Install CentOS 5.3, make sure you install nothing other than the base > packages, and trim even those down. I tend to install specific packages > from the distro later. > > Note: I suggest that you make separate partition(s) for where squid will > actually store its caches. Later mount these partitions with specific > options (like "noatime") that will help increase performance. > > 2) In the initial ncurses-based setup screen, turn off services that you > don't need, and turn off selinux compeletely. > > 3) After install and initial bootup and configuration, run "yum update" > to update the system for fixes, etc. Then reboot. > > 4) After step 2, issue this yum command: > > yum install libcap libcap-devel gcc gcc-c++ bison flex yacc autoconf > automake ncurses ncurses-devel rpm-devel libpcap tcpdump > > Note: let it install other dependency packages. The command above > installs compiles, utilities, etc. > > > 5) Download iptables-1.4.3.2 from netfilter.org > > 6) Download kernel 2.6.30 from kernel.org > > 7) Download squid-3.1.0.8 from squid.org > > 8) Decompress the kernel source, I decompress it to /usr/src/, although > I have read all over the place that this is a bad thing to do. The > location really does not have to be /usr/src/ > > 9) Go into the kernel source directory, issue the following command: > cp /boot/config-2.6.18-128.1.14.el5 ./RH-config-boxed.config > > 10) Issue this command: make menuconfig > > 11) When the ncurses-based kernel config screen loads, select the "Load > an Alternate Configuration File" and type in the full path to the > RH-config-boxed.config. This will load the current kernel config, and > there may be some errors, all of which can be ignored. > > > 12) Configure the kernel as you normally would, but be sure to enable > the following: > > In "Networking support -> Networking options" > > Enable (not as modules): > Packet socket > Packet socket: mmapped IO > TCP/IP networking > IP: advanced router > IP: policy routing > > Enable (as modules): > IP: tunneling > > Enable (not as modules): > IP: GRE tunnels over IP > IP: broadcast GRE over IP > Network packet filtering framework (Netfilter) > > > In "Networking support -> Networking options -> Network packet filtering > framework (Netfilter)" > > Enable (not as modules): > Advanced netfilter configuration > > > In "Networking support -> Networking options -> Network packet filtering > framework (Netfilter) -> Core Netfilter > > Configuration" > > Enable (as modules): > Netfilter connection tracking support > > Enable (not as modules): > Connection tracking security mark support > Connection tracking events > > Enable (as modules): > Connection tracking netlink interface > Transparent proxying support (EXPERIMENTAL) > Netfilter Xtables support (required for ip_tables) > "CONNMARK" target support > "MARK" target support > "TPROXY" target support (EXPERIMENTAL) > "connmark" connection mark match support > "conntrack" connection tracking match support > "mark" match support > "socket" match support (EXPERIMENTAL) > "state" match support > > > In "Networking support -> Networking options -> Network packet filtering > framework (Netfilter) -> IP: Netfilter Configuration" > > Enable (as modules): > IPv4 connection tracking support (required for NAT) > IP tables support (required for filtering/masq/NAT) > Full NAT > MASQUERADE target support > REDIRECT target support > Packet mangling > > > 13) After setting the above options, and any other items you want, exit > out of the kernel config, saving your changes. It will save the kernel > compile config to RH-config-boxed.config so issue the following command > to put the new config in the right > > place: > > cp RH-config-boxed.config config-centos > > Then do the make, make_install_modules, make install, if no errors, > adjust grub.conf to boot to the new kernel. The reboot to > > the new kernel. > > 14) Assuming the kernel compiled, installed and booted properly, it is > time to update iptables. Decompress the iptables > > source, and use the following configure command: > > ./configure --enable-devel --enable-libipq --bindir=/bin --sbindir=/sbin > --sysconfdir=/etc --with-kernel=<path to new kernel source dir> > --with-kbuild=<path to new kernel source dir> --with-ksource=<path to > new kernel source dir> > > > Then do the make, make install > > > > 15) Edit ld.so.conf to add a library path: > > vim /etc/ld.so.conf > > add a line to the end of the file: /usr/local/lib > > > 16) I suggest rebooting, just to make sure that iptables upgrade is > working. Do a reboot, then do a "service iptables status" to make sure > that iptables is running fine. If it is not running ok, it will show > either a status failed, or an empty rule set. > > 17) Assuming that the iptables upgrade is working fine, the next steps > are to add the rules and interfaces needed for WCCP and TPROXY > functionality. First, issue a "insmod ip_gre", then do a "dmesg | tail" > and make sure that you see "GRE over IPv4 tunneling driver" if you don't > see anything, or you get an error from insmod, you either compiled GRE > not as a kernel module, or not at all. > > 18) Create the GRE tunnel interface, issue "ifconfig gre0 <ip address of > squid server> netmask <netmask of squid server ip> up" > > 19) Issue an "ifconfig" and make sure you see a gre0 interface. > > 20) Next we need to add iptables rules to allow traffic to the gre > interface itself, gre protocol traffic across the Ethernet interface, > and WCCP traffic from the router. We will do this by editing the > iptables save file directly. Do the following: > > a.) service iptables save > b.) vim /etc/sysconfig/iptables > c.) just after the line that says ":RH-Firewall-1-INPUT - [0:0]" add the > following: > -A INPUT -i gre0 -j ACCEPT > -A INPUT -p gre -j ACCEPT > -A INPUT -i eth0 -p gre -j ACCEPT > d.) Somewhere lower in the file where the "-A RH-Firewall-1-INPUT" rules > are, add the following: > -A RH-Firewall-1-INPUT -s <address of cisco router>/32 -p udp -m > udp --dport 2048 -j ACCEPT > > NOTE: The "<address of cisco router>" in this instance is the ip > address of the router that the squid box itself uses. > > Port 2048 is the port that WCCP traffic uses. > > e.) save the file and exit vim. Do a "service iptables restart" to make > sure you don't get any errors. > > > 21) Next we add the TPROXY related iptables rules, from the command line > prompt issue the following commands: > > iptables -t mangle -N DIVERT > iptables -t mangle -A DIVERT -j MARK --set-mark 1 > iptables -t mangle -A DIVERT -j ACCEPT > iptables -t mangle -A PREROUTING -p tcp -m socket -j DIVERT > iptables -t mangle -A PREROUTING -p tcp --dport 80 -j TPROXY > --tproxy-mark 0x1/0x1 --on-port <port squid listens to> --on-ip > > <squid server ip> > > 22) Assuming the commands in step 21 didn't give errors, do a "service > iptables save" > > 23) We need to edit the iptables rule order, to make sure it is in the > correct order: "vim /etc/sysconfig/iptables" and make sure the section > at the bottom of the file, from the ":DIVERT - [0:0]" onward looks > something like: > > -A DIVERT -j MARK --set-xmark 0x1/0xffffffff > -A DIVERT -j ACCEPT > -A PREROUTING -p tcp -m socket -j DIVERT > -A PREROUTING -p tcp -m tcp --dport 80 -j TPROXY --on-port <squid port> > --on-ip <squid server ip> --tproxy-mark 0x1/0x1 > > When doing the cli commands in step 21, iptables puts the rules in the > wrong order for them to work for TPROXY. > > Save the file and exit vim. > > > 24) do a "service iptables restart" > > 26) Issue the following commands, per the TPROXY wiki article: > > ip rule add fwmark 1 lookup 100 > ip route add local 0.0.0.0/0 dev lo table 100 > > If no errors from the two commands, add them to the end of > /etc/rc.d/rc.local > > > 27) I'm not sure this is needed anymore, but it doesn't seem to break > anything, so add the following line to the end of /etc/rc.d/rc.local: > echo 1 > /proc/sys/net/ipv4/ip_nonlocal_bind > > 28) In the beginning of the rc.local file, add the following two lines: > > modprobe ip_gre > <the ifconfig gre0 command line from step 18 above> > > > 29) Edit /etc/sysctl.conf, make sure there is a line somewhere that > says: > > # Controls IP packet forwarding > net.ipv4.ip_forward = 1 > > 30) Reboot the server > > 31) Next, build you squid 3.1.x.x, make sure that pass > "--enable-linux-netfilter" to the configure directive, build and install > squid. > > 32) You will need to edit the squid config to make sure that it has what > it needs to use TPROXY and WCCP. For tproxy, you need one item, the > "http_port" directive needs tproxy appended to it, I do the following: > http_port <ip of squid server>:<port to bind to, "squid port" from step > 23 above> tproxy disable-pmtu-discovery=always > > 33) Configure the WCCP specific related items in the squid config file, > specifically: > > a) wccp2_router <router address used in the steps above> > b) wccp_version 2 > c) wccp2_rebuild_wait on > d) wccp2_fowarding_method 1 > e) wccp2_return_method 1 > f) wccp2_assigment_method 1 > g) wccp2_service dynamic 80 > h) wccp2_service dynamic 90 > i) wccp2_service_info 80 protocol=tcp flags=src_ip_hash > priority=240 ports=80 > j) wccp2_service_info 90 protocol=tcp > flags=dst_ip_hash,ports_source priority=240 ports=80 > > 34) There are other squid configuration items that are need which are > not included here because they are not specific to WCCP/TPROXY > functionality and the specifics vary by environment. > > 35) In the router configuration, you need to do at least the following: > a.) enable wccp globally with (this might actually be optional): > ip wccp web-cache > b.) enable the specific services: > 1.) ip wccp 80 > 2.) ip wccp 90 > Note: I use a redirect list ACL with the two commands > above so that the router doesn't WCCP redirect specific web sites. The > command would look like "ip wccp 80 redirect-list 122", and access-list > 122 would be a list of denies for sites to not be redirected, and a > permit any any at the end of the access-list to allow all other websites > to be wccp redirected. > c.) On the interface that is adjacent to the squid box, do a "ip > wccp redirect exclude in" this command makes it so that the router does > not redirect the squid traffic as well as other client traffic. > > > > Some things to check along the way: > > 1) Make sure that the GRE interface on the squid box is seeing packets > coming in. You should never see any packets going out the GRE interface, > and you will only see packets coming in the GRE interface after the WCCP > process on the router redirects them to the squid box. > > 2) Keep in mind that you should separate the idea in your mind between > WCCP traffic and tunneled HTTP traffic. The router and the squid process > talk to each other for status and service information with WCCP. HTTP > traffic received from the client by the router for redirect is > encapsulated in GRE (to preserve it) and then forwarded to the GRE > interface. > > 3) The global command "show ip wccp" is a useful router command. In the > output of this command, you should see two "Service Identifier" sections > (one for 80, one for 90 if you use my setup steps.) Within each "Service > Identifier" group, the number of service group clients and routers > should each be 1. If they are not, them some facet of WCCP conversation > between the router and the squid server is not working. Check the > iptables port 2048 setup step 20 part d above. > > 4) wccp event debug commands are useful. Use "debug ip wccp events" and > set you router logging to debug level to see "Hello" and "Here_I_am" > packets. You need to make sure you see both to insure that the router > and squid box are WCCP talking. > > 5) Surf the Web from a client and see if you get to a web site. If it > doesn't work, check the items above, recheck the steps. > > It if does work, go to a site that tells you your IP to make sure TPROXY > is working and using the client IP and not the squid IP. It is entirely > possible that the whole setup will work, but not the client IP spoofing. > I also suggest that you burn the setup in with web surfing to make sure > it doesn't break. > > I am interested to here people's feedback so that I can improve the > steps above, as well as share optimizations. > > Nick > > > >
